NVIDIA Unveils New Omniverse Libraries, Physical AI Models

NVIDIA reports new NVIDIA Omniverse libraries and NVIDIA Cosmos world foundation models (WFMs) that help advance development and deployment of robotics solutions.

Powered by new NVIDIA RTX PRO Servers and NVIDIA DGX Cloud, the libraries and models allow developers anywhere develop physically accurate digital twins, capture and reconstruct the real world in simulation, generate synthetic data for training physical AI models and build AI agents that understand the physical world, according to NVIDIA.

“Computer graphics and AI are converging to fundamentally transform robotics,” says Rev Lebaredian, vice president of Omniverse and simulation technologies at NVIDIA. “By combining AI reasoning with scalable, physically accurate simulation, we’re enabling developers to build tomorrow’s robots and autonomous vehicles that will transform trillions of dollars in industries.”

New NVIDIA Omniverse Libraries Advance Applications for World Composition New NVIDIA Omniverse software development kits (SDKs) and libraries are now available for building and deploying industrial AI and robotics simulation applications. New Omniverse SDKs introduce data interoperability between MuJoCo (MJCF) and Universal Scene Description (OpenUSD), enabling over 250,000 MJCF robot learning developers to simulate robots across platforms.

New Omniverse NuRec libraries and AI models introduce Omniverse RTX ray-traced 3D Gaussian splatting. NVIDIA Isaac Sim 5.0 and NVIDIA Isaac Lab 2.2 open-source robot simulation and learning frameworks are now available on GitHub. Isaac Sim now includes NuRec neural rendering and new OpenUSD-based robot and sensor schemas. Omniverse NuRec rendering is now integrated in CARLA, an open-source simulator used by over 150,000 developers.

Autonomous vehicle (AV) toolchain leader Foretellix is integrating NuRec, NVIDIA Omniverse Sensor RTX and Cosmos Transfer to enhance its scalable synthetic data generation with physically accurate scenarios. Voxel51’s data engine for visual and multimodal AI, FiftyOne, supports NuRec to ease data preparation for reconstructions.

Cosmos Advances World Generation for Robotics Cosmos WFMs, downloaded over 2 million times, let developers generate diverse data for training robots at scale using text, image and video prompts.

New models announced at SIGGRAPH deliver advances in synthetic data generation speed, accuracy, language support and control: Cosmos Transfer-2, coming soon, simplifies prompting and accelerates photorealistic synthetic data generation from ground-truth 3D simulation scenes or spatial control inputs like depth, segmentation, edges and high-definition maps, NVIDIA reports.

A distilled version of Cosmos Transfer reduces the 70-step distillation process to one so developers can run the model on NVIDIA RTX PRO Servers at high speed. Lightwheel, Moon Surgical and Skild AI are using Cosmos Transfer to accelerate physical AI training by simulating diverse conditions at scale.

NVIDIA Cosmos Reason—a new open, customizable, 7-billion-parameter reasoning VLM for physical AI and robotics—lets robots and vision AI agents reason like humans, using prior knowledge, physics understanding and common sense to understand and act in the real world. Cosmos Reason can be used for robotics and physical AI applications including: Data curation and annotation, which enables developers to automate curation and annotation of training datasets.

Robot planning and reasoning, acting as the brain for methodical decision-making in a robot vision language action (VLA) model. Cosmos Reason lets robots interpret environments and, given complex commands, break them down into tasks and execute them using common sense.

Video analytics AI agents built on the NVIDIA Blueprint for video search and summarization that can extract insights and perform root-cause analysis on volumes of video data. NVIDIA’s robotics and NVIDIA DRIVE teams are using Cosmos Reason for data curation and filtering, annotation and VLA post-training. Cosmos Reason adds world understanding to the vehicles’ long-term trajectory planner. 

NVIDIA has also announced AI infrastructure designed for demanding workloads. NVIDIA RTX PRO Blackwell Servers offer a single architecture for every robot development workload across training, synthetic data generation, robot learning and simulation.

NVIDIA DGX Cloud, available on Microsoft Azure Marketplace, now offers Omniverse developers a fully managed platform to simplify streaming OpenUSD- and NVIDIA RTX-based applications at scale from the cloud.

NVIDIA also announced: OpenUSD Curriculum and Certification, which addresses demand for USD expertise.

Sources: Press materials received from the company and additional information gleaned from the company’s website.