RT-2

Google DeepMind's RT-2 was the first to directly use a vision-language model to generate robot action commands.

VLA (Vision-Language-Action) models are a class of multimodal models that unify visual perception, natural language understanding, and action generation in a single end-to-end architecture, typically taking image sequences and language instructions as input and directly outputting robot joint angles, end-effector poses, or discrete action tokens.

VLA models integrate visual perception, language understanding, and robot action planning into a unified model, typically using a pretrained large vision-language model as the backbone, with representative works including Google's RT-2 and OpenVLA.

RT-2 first demonstrated the feasibility of the VLA architecture by discretizing robot actions into tokens and having the vision-language model directly output action sequences.

DeepMind developed the VLA (Vision-Language-Action model) system, which treats the robot's physical actions on equal footing with visual tokens and language tokens.

Google DeepMind's RT series includes RT-1, RT-2, and RT-X: RT-1 trains foundational manipulation capabilities, RT-2 links VLMs with robot actions, and RT-X achieves stronger generalization through cross-platform data sharing.

RT-2 (Robotics Transformer 2) directly uses a vision-language model to generate robot control instructions end-to-end.

RT-2, released by Google DeepMind in 2023, was the first to validate the feasibility of fine-tuning robot control on a 55-billion-parameter vision-language model, with generalization capability nearly doubling compared to its predecessor.

Representative VLA works include RT-2 and OpenVLA, which are typically pretrained on massive image-text-action triplet data and possess a certain degree of open-world generalization capability.

Representative works of VLA models include RT-2, OpenVLA, and others, with parameter scales ranging from several billion to several tens of billions.

Typical representatives of VLA models include RT-2, OpenVLA and GR00T, which are usually pretrained jointly on internet-scale image-text data and large-scale robotic manipulation data.

Typical representatives of action models are vision-language-action models (VLA), such as RT-2 and OpenVLA, which usually use a pretrained multimodal large model as the backbone and, after fine-tuning on robot data, directly output joint positions, end-effector trajectories or discrete action tokens.

In the field of robot foundation models, NVIDIA's GR00T series already exists, and in the open-source community, VLA models such as RT-2 and OpenVLA are iterating rapidly.

RT-2 is jointly trained on internet-scale image-text data and robot demonstration data, giving the model a certain degree of semantic generalization and commonsense reasoning ability.

Representative VLA works include RT-2, OpenVLA, π0, etc.

Representative VLA architectures include Google's RT-2 and Physical Intelligence's π0; the core challenge of VLA lies in the fact that VLM autoregressive decoding speed is lower than the 10–50 Hz real-time frequency required by the robot control loop.

RT-2 fine-tunes vision-language large models such as PaLM-E into controllers that output robot actions, demonstrating zero-shot transfer capabilities across tasks.

Google DeepMind's RT-2 and π0 (pi-zero) are representative VLA implementations.

Representative works in VLA models include Google's RT-2, OpenVLA, and π0.

Google's accumulation in 3D world understanding stems from its long-term robotics research (such as the RT-2 series) and its experience in 3D mapping with Google Maps.