Accessibility Tree
A semantic tree structure automatically constructed by browsers from the DOM and ARIA attributes, exposing key page information to assistive technologies such as screen readers. Each node contains three types of properties: role, name, and state. Decorative nodes are excluded. The accessibility tree serves as an important signal for AI agents when interpreting webpage structure.
Core Facts
Computer use capability allows a model to observe the screen state via screenshots or by reading the interface accessibility tree, and then output action instructions such as click coordinates and keyboard input
The accessibility tree is a semantic tree structure built by the browser based on the DOM; it strips away visual style information and retains semantic information such as the role, name, state, and properties of each element.
Timeline (last 90 days)
Computer Use no longer relies only on a single action loop; instead, it directly writes JavaScript code to execute multi-step operations in batches within the browser environment, and fuses multiple modalities of information such as screenshots, the accessibility tree, and Playwright.
The accessibility tree is a structured UI description that the operating system maintains for assistive technologies (screen readers, voice control, automation tools), exposed by each platform's native APIs: on Windows it is UI Automation (UIA), on macOS it is AXUIElement, on Linux it is AT-SPI, and on Android it is AccessibilityNodeInfo.
After each important step, the accessibility tree or text should be re-read for post-action verification, and success of the click should not be assumed.
In the execution order of a computer-use agent, the tree should come first (listing interactive elements with names, roles and positions), text second (performing OCR on the window to match visible text when the tree has no labels), and screenshots last (for canvases, unnamed icons, or page verification).
In Canvas apps (design tool canvases, games, maps, chart controls), the accessibility tree usually has the entire window as just a single element with nothing inside.
For some applications (especially games with custom-drawn UI and some desktop software), accessibility information is sparse or even missing; in such cases SA-CUA's rule derivation fails and it must fall back to other mechanisms.
SA-CUA is a dual-LLM framework for establishing trust boundaries for computer-use Agents, whose core is to derive conservative deterministic rules from the accessibility tree, isolating trusted content from untrusted content on the privileged LLM's screenshot.
SA-CUA is highly dependent on the quality and coverage of the accessibility tree; for applications or interfaces with incomplete accessibility information, the effectiveness of rule derivation needs further verification.
Computer use capability allows a model to observe the screen state via screenshots or by reading the interface accessibility tree, and then output action instructions such as click coordinates and keyboard input
Playwright MCP controls the browser using accessibility snapshots—structured data from the web accessibility tree—rather than screenshots or visual models.
4 more timeline events
All Facts (15)
Computer use capability allows a model to observe the screen state via screenshots or by reading the interface accessibility tree, and then output action instructions such as click coordinates and keyboard input
65%VerifiedThe accessibility tree is a semantic tree structure built by the browser based on the DOM; it strips away visual style information and retains semantic information such as the role, name, state, and properties of each element.
65%UnverifiedComputer Use no longer relies only on a single action loop; instead, it directly writes JavaScript code to execute multi-step operations in batches within the browser environment, and fuses multiple modalities of information such as screenshots, the accessibility tree, and Playwright.
50%UnverifiedIn Canvas apps (design tool canvases, games, maps, chart controls), the accessibility tree usually has the entire window as just a single element with nothing inside.
50%UnverifiedIn the execution order of a computer-use agent, the tree should come first (listing interactive elements with names, roles and positions), text second (performing OCR on the window to match visible text when the tree has no labels), and screenshots last (for canvases, unnamed icons, or page verification).
50%UnverifiedAfter each important step, the accessibility tree or text should be re-read for post-action verification, and success of the click should not be assumed.
50%UnverifiedThe accessibility tree is a structured UI description that the operating system maintains for assistive technologies (screen readers, voice control, automation tools), exposed by each platform's native APIs: on Windows it is UI Automation (UIA), on macOS it is AXUIElement, on Linux it is AT-SPI, and on Android it is AccessibilityNodeInfo.
50%UnverifiedSA-CUA is a dual-LLM framework for establishing trust boundaries for computer-use Agents, whose core is to derive conservative deterministic rules from the accessibility tree, isolating trusted content from untrusted content on the privileged LLM's screenshot.
50%UnverifiedFor some applications (especially games with custom-drawn UI and some desktop software), accessibility information is sparse or even missing; in such cases SA-CUA's rule derivation fails and it must fall back to other mechanisms.
50%UnverifiedSA-CUA is highly dependent on the quality and coverage of the accessibility tree; for applications or interfaces with incomplete accessibility information, the effectiveness of rule derivation needs further verification.
50%UnverifiedPlaywright MCP controls the browser using accessibility snapshots—structured data from the web accessibility tree—rather than screenshots or visual models.
50%UnverifiedThere are three main ways for AI agents to perceive web pages: directly viewing rendered page screenshots (vision models), reading the rendered DOM, and reading the accessibility tree.
50%UnverifiedThe accessibility tree is automatically constructed by the browser based on the DOM and ARIA attributes, and each node contains three key properties: role, name, and state.
50%UnverifiedCompared to screenshot-plus-visual-recognition approaches, accessibility-tree-based browser interaction solutions consume significantly fewer tokens and are more reliable for fine-grained operations such as form filling and dropdown selection.
50%Unverified无障碍树的节点数量通常只有DOM节点的10%-20%，因为纯样式容器、不可见元素和装饰性图片会被过滤
50%