GUI grounding
A foundational task for GUI agents, referring to locating the corresponding interactive element in an interface screenshot according to a natural language instruction and outputting coordinates for executing a click.
Timeline (last 90 days)
The implementation of computer use typically relies on two underlying technologies: screen understanding (GUI grounding) and action sequence planning (action planning).
GUI grounding is a foundational task in the GUI agent field, referring to an agent locating the corresponding interactive element in an interface screenshot given a natural language instruction and outputting coordinates to execute a click.
ScreenRef provides millions of paired GUI screenshots and text descriptions, and models trained on this data set records on screen-task benchmarks.
All Facts (3)
The implementation of computer use typically relies on two underlying technologies: screen understanding (GUI grounding) and action sequence planning (action planning).
50%UnverifiedGUI grounding is a foundational task in the GUI agent field, referring to an agent locating the corresponding interactive element in an interface screenshot given a natural language instruction and outputting coordinates to execute a click.
50%UnverifiedScreenRef provides millions of paired GUI screenshots and text descriptions, and models trained on this data set records on screen-task benchmarks.
50%