Computer Use
Computer Use is a technical capability that enables AI models to operate computer graphical user interfaces in a manner analogous to human users. Its core mechanisms include perceiving the current interface state through screenshots, identifying UI controls and interactive elements, and executing atomic operations such as mouse clicks and keyboard inputs. This capability allows AI to autonomously complete complex tasks across applications without dedicated API interfaces, with applications in automated workflows, software testing, and intelligent agent systems.
Core Facts
GPT-6 Astra is capable of operating modeling software and game engines, possessing agentic capabilities to view the screen, click on the interface, inspect work, and continue making modifications.
Anthropic introduced the Computer Use feature to allow AI to directly operate desktop applications
OpenAI的Operator项目专注于浏览器自动化操作
Timeline (last 90 days)
Codex provides a Computer Use plugin, whose purpose is to let Codex directly operate the user's computer.
There are two must-install plugins for Codex: Computer Use (which can invoke apps on the computer, automatically take screenshots, and read window contents) and a black plugin for making visualization charts (after the revamp it disappeared from the public interface and can only be found via search; in testing it made a dynamic chart successfully on the first try).
Hermes's second-layer tools need to be configured and enabled as needed, including WebSearch/WebExtract (web search and scraping), Vision Analyze (visual understanding), Video Analyze (video understanding), STT/TTS (voice interaction), Browser Exec (browser automation), and Computer Use (desktop control).
Computer Use no longer relies only on a single action loop; instead, it directly writes JavaScript code to execute multi-step operations in batches within the browser environment, and fuses multiple modalities of information such as screenshots, the accessibility tree, and Playwright.
Computer Use is built into the Agents API, so developers can directly call the same implementation as Codex and ChatGPT, avoiding the high barrier of building their own harness.
On the security level, Ari recommends that developers grant only the website and app access permissions necessary for the task, request user confirmation before performing critical operations such as payments, and gradually build user trust through reliability design.
Early computer-operating agent models could reliably start tasks, but then often got stuck on issues such as pop-ups, loading delays, or page structure changes.
In a task involving subscribing to a customized meal service, manual operation takes two hours, while Computer Use on GPT-6.1 Sol completes it in 15 minutes, about eight times faster than manual work.
Current Computer Use models have learned to retry, self-check, and judge what is wrong, thereby improving their debugging capabilities.
OpenSense supports browser operation and Computer Use, and can automatically download papers and collect data; if GPU resources are available, it can run computations locally to verify paper data.
40 more timeline events
All Facts (20)
GPT-6 Astra is capable of operating modeling software and game engines, possessing agentic capabilities to view the screen, click on the interface, inspect work, and continue making modifications.
90%VerifiedAnthropic introduced the Computer Use feature to allow AI to directly operate desktop applications
90%VerifiedOpenAI的Operator项目专注于浏览器自动化操作
90%VerifiedAnthropic's Computer Use feature allows the Claude model to directly control the entire computer desktop environment, including moving the mouse, clicking buttons, and typing text.
90%VerifiedComputer Use is a capability that allows AI models to directly control computer interfaces by taking screenshots to identify screen content and simulating mouse clicks and keyboard input.
90%VerifiedAnthropic于2024年率先在Claude中推出Computer Use功能
90%VerifiedOpenAI's GPT-4o with Tools and Anthropic's Claude Computer Use are representative implementations in the AI Agent direction.
85%VerifiedPerplexity's "Computer" feature falls under the category of AI Agent, allowing AI models to operate computer interfaces, including clicking buttons, filling out forms, browsing the web, and performing file operations.
80%VerifiedOpenAI的Operator和Google的Project Mariner也在探索AI操控计算机的类似路径
80%VerifiedAnthropic launched the Computer Use feature in its Claude model, allowing the AI model to directly interact with computer desktop environments.
80%VerifiedThe Computer Use technology was first publicly demonstrated by Anthropic in October 2024 at the launch of Claude 3.5 Sonnet.
80%VerifiedIn October 2024, Anthropic was the first to release the Computer Use feature for Claude, allowing AI to control desktop applications through screenshot recognition and simulated mouse and keyboard operations.
80%VerifiedAnthropic's Claude has added a "Computer Use" feature.
80%VerifiedAnthropic was the first to launch the Claude Computer Use feature in 2024, after which OpenAI released its Operator product.
80%VerifiedIn 2024, Anthropic released Claude's Computer Use feature, which understands screen content through screenshots and simulates mouse and keyboard operations using coordinate positioning
75%VerifiedCurrent AI agents operating web pages primarily rely on two technical approaches: computer vision (feeding page screenshots into multimodal large models to identify UI elements) and DOM parsing (reading the HTML structure tree to infer element functions).
75%VerifiedAnthropic's Claude launched a native desktop client and introduced Computer Use capability, enabling it to directly operate the user's desktop applications.
75%VerifiedThe Computer Use capability was first commercially deployed by Anthropic in Claude 3.5, and was subsequently followed by OpenAI.
75%VerifiedThe underlying technology of GUI agents relies on the collaborative operation of visual language models' screen comprehension capabilities and action planning modules: first performing semantic understanding of screen screenshots to identify interface elements, then having the planning module generate specific instruction sequences for execution.
70%VerifiedComputer Use is an AI capability first introduced by Anthropic in 2024 and subsequently followed up by OpenAI and others, allowing models to operate graphical interface applications like humans.
70%