NVIDIA has released Cosmos 3 on Hugging Face, introducing an open-source omni-model architecture designed specifically for physical AI and robotics applications. Built on a Mixture-of-Transformers backbone, the release includes the 16-billion parameter Cosmos 3 Nano and the 64-billion parameter Cosmos 3 Super models. Developers can now run physical reasoning, world generation, and robot action prediction within a single forward pass rather than coordinating multiple separate pipelines.
Mixture-of-Transformers Architecture
Unlike previous model iterations that required separate models for generation and reasoning, Cosmos 3 processes text, image, video, audio, and action inputs within a single unified framework. The architecture splits input sequences into an autoregressive subsequence for reasoning via next-token prediction and a diffusion subsequence for iterative visual generation.
While the autoregressive and diffusion tokens utilize distinct parameter sets inside each transformer layer, they interact directly through joint attention mechanism. This design allows Cosmos 3 to function seamlessly as a vision-language model, synthetic video generator, world dynamics predictor, or robot policy model without architectural modifications.
Two Scales for Edge and Server Deployments
To accommodate diverse development requirements, NVIDIA has released Cosmos 3 in two specific configurations alongside supporting tools:
- Cosmos 3 Nano: A 16-billion parameter model combining an 8B reasoner and 8B generator, optimized for workstation GPUs like the RTX PRO 6000.
- Cosmos 3 Super: A 64-billion parameter variant featuring a 32B reasoner and 32B generator, designed for large-scale synthetic data generation on NVIDIA Hopper and Blackwell hardware.
- Diffusers Library Integration: Direct implementation via the
Cosmos3OmniPipelineclass enables single-frame and video generation in Python. - Open Synthetic Datasets: Datasets released on Hugging Face covering embodied robotics, physical interactions, human motion, autonomous driving, and warehouse safety.
Operational Requirements and Trade-offs
For autonomous driving and robotics developers, consolidating physical reasoning and action generation into one model simplifies simulation and deployment workflows. According to NVIDIA's release details, developers can execute forward dynamics, inverse dynamics, or direct policy generation using tailored text and image prompts.
However, a key operational constraint exists for specialized industrial environments. While Cosmos 3 provides baseline physical reasoning out of the box, enterprise deployments on unique hardware setups will still require post-training using custom domain datasets to achieve optimal operational accuracy.