Refs and snapshots

Understanding the ref system that makes element interactions possible

Refs are how you reference UI elements across commands. A snapshot assigns refs like @n1, @n2, @n3 to elements in the Accessibility tree. You then use these refs in subsequent commands to click, type, or inspect specific elements.

What are refs?

A ref (reference) is a short identifier like @n1, @n2, etc. that points to a specific UI element. Think of refs as temporary “handles” for elements discovered during a snapshot.

# Take a snapshot
$ agent-native snapshot Safari --interactive
Snapshot: Safari (pid 1234) -- 12 elements
---------------------------------------------
  AXWindow "Safari" [ref=n1]
    AXButton "Back" [AXPress] [ref=n2]
    AXButton "Forward" [AXPress] [ref=n3]
    AXTextField [ref=n4]
    AXButton "Go" [AXPress] [ref=n5]

# Click using a ref
$ agent-native click @n2
OK Clicked: AXButton title="Back"

Refs are numbered sequentially starting from n1 in each snapshot. The @ prefix distinguishes refs from app names.

How snapshots work

The snapshot command walks the Accessibility tree and assigns refs to elements. From SnapshotCommand.swift:5-133:

1. Walk the tree

   Traverse the app’s Accessibility tree to the specified depth:

   let element = AXEngine.appElement(pid: pid)
   let tree = AXEngine.walkTree(element, maxDepth: depth)
   
2. Filter elements

   Optionally filter to only interactive elements with --interactive, or use --compact to skip empty structural nodes:

   // SnapshotCommand.swift:26-33
   static let interactiveRoles: Set<String> = [
       "AXButton", "AXTextField", "AXCheckBox",
       "AXRadioButton", "AXPopUpButton", "AXLink",
       "AXMenuItem", "AXTab", "AXSearchField", ...
   ]
   
3. Assign refs

   Number each element sequentially:

   var refCounter = 0
   for (node, depth) in tree {
       refCounter += 1
       let ref = "n\(refCounter)"
       // Store for later resolution
   }
   
4. Save to RefStore

   Persist ref → element mappings to a temp JSON file:

   RefStore.save(refEntries)
   

The RefStore

Refs are persisted to disk at /tmp/agent-native-refs.json so they survive between CLI invocations. Each command runs as a separate process, so refs must be saved externally.

From RefStore.swift:4-35:

struct RefEntry: Codable {
    let ref: String           // "n1", "n2", etc.
    let pid: Int32            // Process ID of the app
    let role: String          // AXButton, AXTextField, etc.
    let title: String?        // Button text, window title
    let label: String?        // Accessibility label
    let identifier: String?   // Programmatic identifier
    let pathHint: String      // XPath-like path in the tree
}

private static let storePath: URL = {
    let tmp = FileManager.default.temporaryDirectory
    return tmp.appendingPathComponent("agent-native-refs.json")
}()

static func save(_ entries: [RefEntry]) { ... }
static func load() -> [RefEntry] { ... }

The RefStore contains only the metadata needed to re-find elements, not live AXUIElement pointers (which can’t be serialized).

Ref resolution

When you use a ref like @n5, agent-native must resolve it back to a live AXUIElement. This happens by re-searching the tree using the stored attributes.

From RefStore.swift:37-74:

static func resolve(ref: String) throws -> (element: AXUIElement, node: AXNode) {
    let entries = load()
    guard let entry = entries.first(where: { $0.ref == cleanRef }) else {
        throw AXError.elementNotFound("Unknown ref: @\(cleanRef). Run `snapshot` first.")
    }
    
    let appElement = AXEngine.appElement(pid: entry.pid)
    let results = AXEngine.findElements(
        root: appElement,
        role: entry.role,
        title: entry.title,
        label: entry.label,
        identifier: entry.identifier,
        maxDepth: 15
    )
    
    // Return first exact match
    if let match = results.first(where: { /* exact attribute match */ }) {
        return match
    }
    
    throw AXError.elementNotFound(
        "Could not re-resolve @\(cleanRef). The UI may have changed -- run `snapshot` again.")
}

1. Load ref metadata

   Read the entry from agent-native-refs.json.

2. Search the live tree

   Use AXEngine.findElements() to search for elements matching the stored role, title, label, and identifier.

3. Match exactly

   Return the element that exactly matches all stored attributes.

4. Error if not found

   If the element no longer exists or has changed, throw elementNotFound.

Ref lifecycle and invalidation

Refs become invalid when:

1. The UI changes - If a button is removed or its text changes, the ref can’t be resolved
2. The app restarts - PIDs change, invalidating all refs for that app
3. A new snapshot is taken - Each snapshot command overwrites the entire RefStore

Refs are ephemeral. They’re valid for the current UI state only. If your interaction changes the UI significantly (e.g., navigating to a new screen), take a fresh snapshot.

When refs fail

$ agent-native click @n8
Error: Could not re-resolve @n8. The UI may have changed -- run `snapshot` again.

This means the element with ref @n8 no longer exists or has different attributes. Solution: take a new snapshot.

Interactive vs full snapshots

You can control what gets refs:

Interactive snapshots

Use --interactive to only assign refs to elements you can interact with:

$ agent-native snapshot Safari --interactive

This filters to interactiveRoles (buttons, text fields, checkboxes, links, etc.) and skips structural elements like groups and static text.

Use --interactive for agent workflows. It reduces noise and makes snapshots easier to parse.

Compact snapshots

Use --compact to remove empty structural elements:

$ agent-native snapshot Safari --compact

From SnapshotCommand.swift:69-74, this skips elements with:

• No title, label, or value
• No actions
• Has children (i.e., is just a container)

Full snapshots

By default, every element gets a ref. This is useful for debugging or when you need to inspect structural elements:

$ agent-native snapshot Safari

Example workflow

1. Take initial snapshot

   $ agent-native snapshot Safari --interactive
   Snapshot: Safari (pid 1234) -- 8 elements
   AXWindow "Safari" [ref=n1]
     AXTextField [ref=n2]
     AXButton "Go" [AXPress] [ref=n3]
   
2. Interact with refs

   $ agent-native click @n2
   OK Clicked: AXTextField
   
   $ agent-native type @n2 "example.com"
   OK Typed 11 chars into AXTextField
   
   $ agent-native click @n3
   OK Clicked: AXButton title="Go"
   
3. Re-snapshot after UI changes

   After the page loads, the UI is different. Take a new snapshot:

   $ agent-native snapshot Safari --interactive
   Snapshot: Safari (pid 1234) -- 15 elements
   # Now @n1, @n2, etc. refer to the NEW UI state
   

See also

Accessibility tree

Understanding the tree that refs point into

Workflow

The snapshot → interact → re-snapshot pattern