NVIDIA and Hugging Face have partnered with 35 global organizations to launch Open-H-Embodiment, an open healthcare robotics dataset containing 778 hours of training data. The initiative targets physical AI by combining synchronized vision, force, and kinematics data with open-source foundation models designed for surgical tasks. This release provides researchers and roboticists with standardized tools to train closed-loop control systems.
Standardizing Healthcare Robotics Datasets
Healthcare AI historically focused on static perception tasks, such as classifying or segmenting pathological images. However, physical AI requires real-time control, contact dynamics, and closed-loop feedback that traditional static datasets cannot provide.
Comprising 778 hours of CC-BY-4.0 healthcare robotics training data, the Open-H-Embodiment dataset bridges this gap across multiple research and commercial platforms. Spearheaded by steering committee members from Johns Hopkins University, the Technical University of Munich, and NVIDIA, participating entities include Johns Hopkins, Stanford, CMR Surgical, and UC Berkeley.
Key components of the dataset release include:
- Commercial and research hardware: Data covers platforms including CMR Surgical, Rob Surgical, Tuodao, dVRK, Franka, and Kuka.
- Procedure diversity: Encompasses surgical robotics, ultrasound autonomy, and colonoscopy data.
- Multimodal environments: Spans simulation, benchtop exercises such as suturing, and real clinical procedures.
GR00T-H: Vision-Language-Action Policy for Surgery
Alongside the dataset, the collaboration released GR00T-H, a Vision-Language-Action (VLA) policy model derived from the Isaac GR00T N series. Trained on roughly 600 hours of Open-H-Embodiment data, GR00T-H utilizes NVIDIA's Cosmos Reason 2 2B as its underlying Vision-Language Model backbone.
To handle specialized hardware like cable-driven systems that make imitation learning difficult, GR00T-H uses four specific architectural adaptations:
- Unique Embodiment Projectors: Learnable MLPs map each robot's kinematics to a shared action space.
- State Dropout (100%): Proprioceptive input is dropped during inference to rely on learned system biases.
- Relative EEF Actions: Training uses a common relative End-Effector action space to overcome kinematic inconsistencies.
- Metadata Prompts: Instrument names and control index mappings are injected directly into VLM task prompts.
In benchmark testing on the SutureBot platform, a GR00T-H prototype successfully executed an end-to-end suture, demonstrating long-horizon physical dexterity.
Cosmos-H Generative Simulation
Traditional simulators frequently struggle with real-world clinical variables like soft-tissue deformation, blood, smoke, and specular reflections. To address this sim-to-real gap, the team released Cosmos-H-Surgical-Simulator, an action-conditioned World Foundation Model.