NVIDIA Cosmos
A world foundation model platform developed by NVIDIA for physical AI. Built around video generation models, it produces future frame sequences consistent with physical laws and supports closed-loop evaluation and virtual testing of robot policy models.
Core Facts
Timeline (last 90 days)
World foundation models are a class of large generative models specifically for modeling the dynamics of the physical world, with typical representatives including NVIDIA Cosmos and Google's Genie series.
Cosmos comes in three sizes ranging from 4 billion to 64 billion parameters, is fully multimodal, commercially usable, and customizable, with its Super NanoEdge version optimized for NVFP4 and speculative decoding.
According to Clark, Cosmos is currently the leading open model for industrial visual analytics (dense captioning and anomaly detection reasoning).
Cosmos generator training is divided into two stages: pretraining to learn foundational capabilities, and mid-training where action data is then incorporated.
Cosmos uses a temporal position embedding scheme to normalize videos at different frame rates, audio at different hertz, and action signals to the same time scale.
Cosmos trains the three modeling perspectives of Forward Dynamics, Inverse Dynamics, and Policy within the same model, and imposes an information bottleneck to force them to share representations.
NVIDIA Cosmos is a world model for physical AI that can take text, video, audio, and actions as unified inputs and outputs.
All models, code, and partial training data for Cosmos have been open-sourced, with models and datasets hosted on Hugging Face and code located in a GitHub repository.
The team has performed post-training of Cosmos on the ZOE dataset, achieving SOTA-level grasp-and-place policy results.
Results in the Cosmos paper show that there is synergy among the three tasks of Forward Dynamics, Inverse Dynamics, and Policy, with learning one task benefiting the others.
2 more timeline events
All Facts (12)
NVIDIA Cosmos is a series of foundation models oriented toward physical world understanding, focused on visual reasoning tasks, and belongs to the category of physical AI foundation models.
75%UnverifiedWorld foundation models are a class of large generative models specifically for modeling the dynamics of the physical world, with typical representatives including NVIDIA Cosmos and Google's Genie series.
50%UnverifiedCosmos comes in three sizes ranging from 4 billion to 64 billion parameters, is fully multimodal, commercially usable, and customizable, with its Super NanoEdge version optimized for NVFP4 and speculative decoding.
50%UnverifiedThe Cosmos architecture includes a reasoning tower (VLM) that operates autoregressively and a bidirectional generation tower that operates via diffusion, with the generation tower initialized from the VLM's pretrained weights.
50%UnverifiedCosmos generator training is divided into two stages: pretraining to learn foundational capabilities, and mid-training where action data is then incorporated.
50%UnverifiedCosmos uses a temporal position embedding scheme to normalize videos at different frame rates, audio at different hertz, and action signals to the same time scale.
50%UnverifiedCosmos trains the three modeling perspectives of Forward Dynamics, Inverse Dynamics, and Policy within the same model, and imposes an information bottleneck to force them to share representations.
50%UnverifiedResults in the Cosmos paper show that there is synergy among the three tasks of Forward Dynamics, Inverse Dynamics, and Policy, with learning one task benefiting the others.
50%UnverifiedAll models, code, and partial training data for Cosmos have been open-sourced, with models and datasets hosted on Hugging Face and code located in a GitHub repository.
50%UnverifiedThe team has performed post-training of Cosmos on the ZOE dataset, achieving SOTA-level grasp-and-place policy results.
50%UnverifiedCosmos uses a single model architecture to simultaneously handle three types of tasks: visual-language understanding, scene generation, and action execution.
50%UnverifiedNVIDIA Cosmos is a world model for physical AI that can take text, video, audio, and actions as unified inputs and outputs.
50%