Working with Electron apps

Automate Electron apps like Slack, Discord, and VS Code using keyboard shortcuts and screenshots

Electron apps include Slack, Discord, VS Code, Notion, Figma, and many other popular desktop applications.

Why Electron apps are different

Electron apps expose a minimal Accessibility tree compared to native macOS apps. Inner UI elements often appear as opaque AXGroup elements with no labels or structure, making traditional AX-based automation challenging.

When you run snapshot -i on an Electron app, you may see very few interactive elements:

agent-native snapshot Slack -i

Snapshot: Slack (pid 1234) -- 3 elements
---------------------------------------------
AXWindow "Slack" [ref=n1]
  AXGroup [ref=n2]
    AXGroup [ref=n3]

This minimal tree structure means you need alternative automation strategies.

Automation strategies

Use keyboard shortcuts

Most Electron apps have rich keyboard support. This is the primary method for automating Electron apps.

1. Identify available shortcuts

   Check the app’s menu bar or documentation for keyboard shortcuts. Common patterns:

   • Cmd+K: Quick switcher (Slack, Discord, VS Code)
   • Cmd+N: New window/message
   • Cmd+T: New tab
   • Cmd+W: Close window/tab
   • Cmd+F: Search
2. Send keystrokes with agent-native

   Use the key command to send keyboard shortcuts:

   agent-native key Slack cmd+k                    # Open quick switcher
   agent-native key Slack "#general" return        # Navigate to channel
   agent-native key Slack "Hello team!" return     # Send message
   

   See KeyCommand.swift:36-44 for implementation details.

3. Chain commands for complex workflows

   Combine multiple keystrokes to accomplish tasks:

   agent-native key Slack cmd+k                    # Open switcher
   agent-native key Slack "@john" return           # Find user
   agent-native key Slack "Meeting at 3pm" return  # Send DM
   

Use screenshots for visual context

When the AX tree doesn’t provide enough information, capture screenshots to understand what’s on screen:

agent-native screenshot Slack /tmp/slack.png

The screenshot command captures the app’s frontmost window (see ScreenshotCommand.swift:29-42):

# With custom path
agent-native screenshot Slack ~/Desktop/slack-state.png

# Default auto-generated path
agent-native screenshot Slack
# Output: /tmp/agent-native-abc123.png

# JSON output for parsing
agent-native screenshot Slack --json
# {"path": "/tmp/...", "width": 1920, "height": 1080}

Use screenshots to verify navigation state before and after keyboard actions.

Check window titles

The window title often reflects the current navigation state:

agent-native get title Slack

Chad Donohue (DM) - Ryan Florence Fan Club - Slack

This is useful for:

• Confirming successful navigation
• Determining which channel or conversation is active
• Verifying modal states

# Navigate and verify
agent-native key Slack cmd+k
agent-native key Slack "#engineering" return
title=$(agent-native get title Slack)
if [[ "$title" == *"#engineering"* ]]; then
  echo "Successfully navigated to #engineering"
fi

Paste files

For file uploads, use the paste command to copy a file to the clipboard and paste it:

agent-native paste Slack /path/to/screenshot.png

This is equivalent to:

1. Copying the file to the clipboard
2. Sending Cmd+V to the app

See SKILL.md:60-67 for the paste command details.

Example: Automating Slack

Here’s a complete workflow for sending a message with an image to a Slack channel:

1. Open and activate Slack

   agent-native open Slack
   
2. Navigate to the channel

   agent-native key Slack cmd+k              # Quick switcher
   agent-native key Slack "#general" return  # Select channel
   
3. Verify navigation

   title=$(agent-native get title Slack)
   echo "Current view: $title"
   
4. Type message

   agent-native key Slack "Check out this screenshot:"
   
5. Paste image

   agent-native paste Slack /tmp/report.png
   
6. Send message

   agent-native key Slack return
   

Common keyboard shortcuts

Slack

Shortcut	Action
cmd+k	Quick switcher
cmd+u	Upload file
cmd+n	New message
cmd+f	Search
cmd+/	Show shortcuts
cmd+shift+k	Direct messages
cmd+shift+a	All unreads

VS Code

Shortcut	Action
cmd+p	Quick open
cmd+shift+p	Command palette
cmd+b	Toggle sidebar
cmd+j	Toggle terminal
cmd+k cmd+s	Keyboard shortcuts

Discord

Shortcut	Action
cmd+k	Quick switcher
cmd+n	Create/join server
cmd+shift+t	Create thread
cmd+i	Toggle inbox

Best practices

Always re-snapshot or verify state after navigation actions. Electron app states change frequently, and keyboard shortcuts may have unexpected effects.

Use delays between actions

Add small delays between rapid keystrokes to ensure the app processes each action:

agent-native key Slack cmd+k
sleep 0.5
agent-native key Slack "#general" return

Verify with screenshots

Capture screenshots before and after actions to debug automation failures:

agent-native screenshot Slack /tmp/before.png
agent-native key Slack cmd+k
agent-native screenshot Slack /tmp/after.png

Check window titles

Use window titles to confirm navigation:

agent-native get title Slack

Test shortcuts manually

Verify keyboard shortcuts work manually before automating them.

Troubleshooting

Keyboard shortcuts don't work

Possible causes:

• App is not focused/activated
• Shortcut requires different modifiers on your system
• App has custom keybindings

Solutions:

1. Ensure the app is activated:
   agent-native open Slack  # Activates the app
   
2. Check the app’s keyboard shortcuts in Preferences
3. Try alternative shortcuts (e.g., cmd+shift+k instead of cmd+k)

Text not typing correctly

Possible causes:

• Special characters require escaping
• App processes input asynchronously

Solutions:

1. Add delays between keystrokes:
   agent-native key Slack "Hello"
   sleep 0.2
   agent-native key Slack "world"
   
2. Use quotes for text with spaces or special characters:
   agent-native key Slack "Message with spaces"
   

Can't verify navigation state

Solutions:

1. Use window titles:
   agent-native get title Slack
   
2. Use screenshots:
   agent-native screenshot Slack /tmp/verify.png
   
3. Add delays to allow UI to update:
   agent-native key Slack cmd+k
   sleep 0.5
   agent-native get title Slack
   

Next steps

System Settings

Automate System Settings and native macOS apps

Safari automation

Interact with web content in Safari