UI & scene introspection

What the AI sees

Every platform dumps its own native structure: the full GameObject hierarchy of a Unity scene, the Android view tree with Compose semantics, the iOS/macOS view tree with SwiftUI controls flattened in as addressable nodes, the Flutter element tree, the Godot scene tree, and the DOM — shadow DOM included. Nodes that render text carry their live value, so the AI reads labels and input contents straight from the tree.

Dumps are capped at 3000 nodes and default to a token-efficient flat list, so a whole screen fits in one MCP reply.

How it works

The on-device SDK walks the platform's own hierarchy and serializes each node with what the AI needs to act on it: name or class, a /-separated path, component or type names, visibility, and current text. That path becomes the address for every follow-up task — click it, type into it, inspect it in depth. Stale paths come back as honest errors, so the AI re-traverses and self-corrects instead of acting on old structure.

Example: traversing a Unity scene

One run_task call returns the live hierarchy under a node — path selects the starting node (empty means all loaded scenes), depth limits how many levels are included:

run_task("scene_traverse", { "path": "Canvas/MainPanel", "depth": 2 })

→ {
    "path": "Canvas/MainPanel",
    "depth": 2,
    "scene": "MainMenu",
    "node": {
      "name": "MainPanel", "active": true,
      "components": ["RectTransform", "CanvasRenderer", "Image"],
      "childCount": 2,
      "children": [
        { "name": "TitleText", "text": "Settings",
          "components": ["RectTransform", "Text"], "childCount": 0 },
        { "name": "StartButton", "active": true,
          "components": ["RectTransform", "Button"], "childCount": 1 }
      ]
    },
    "nodeCount": 3,
    "truncated": false
  }

Every node reports its name, active state, component type names and child count; text-rendering nodes (UGUI Text, TextMeshPro, InputField) also report their current value. Any path here feeds straight into ui_click or view_component.

The traversal task on each platform

Free during the public beta.