NVIDIA Omniverse and Cosmos: The Engine Behind Physical AI

A robot arm on a factory floor merging into a glowing wireframe digital twin with simulated sensor data and lidar point clouds
AI-generated illustration

Teaching a robot to pick up a part, or a car to handle a child running into the road, used to mean one of two things: collect mountains of real-world footage, or build a simulator and hope it looked real enough. NVIDIA is betting that the answer is now both at once. Its Omniverse libraries build physically accurate virtual worlds, and its Cosmos world foundation models learn how those worlds behave, fill them with endless variations and help machines decide what to do next. Together they form the core of what NVIDIA calls physical AI: AI that perceives, reasons and acts in the real world.

Here is how the two pieces fit, what changed in 2026, and who is already putting them to work.

Omniverse: Building Simulation-Ready Worlds

NVIDIA describes Omniverse as a set of modular libraries, APIs and agent-ready tools for building physical AI applications and simulation-ready 3D workflows. The key word is simulation-ready. As NVIDIA’s Omniverse product page puts it, a 3D scene can look complete and still lack the structure and physical properties a simulation needs.

Omniverse fills that gap with a handful of core capabilities:

  • OpenUSD to map data from design and engineering tools into structured 3D scenes that can move between applications and simulation pipelines.
  • SimReady validation to check whether assets meet the requirements for training and testing.
  • RTX rendering and sensor simulation to generate camera, lidar and radar outputs from those scenes.
  • GPU-accelerated physics to simulate motion, contacts and collisions.

Two 2026 shifts matter for buyers. First, NVIDIA says that as of May 2026 Omniverse is freely available for development and production use without an NVIDIA AI Enterprise subscription, with paid enterprise support still on offer. Second, in July NVIDIA folded Omniverse libraries into its Agent Toolkit, so AI agents can call them to prepare 3D content, assemble simulation workflows and flag issues for human review.

Cosmos: World Models That Reason, Simulate and Act

If Omniverse is the stage, Cosmos is the actor that has studied how the physical world works. A world model learns how environments behave, what might happen next and which actions make sense. NVIDIA launched Cosmos 3 at GTC Taipei on May 31, 2026, calling it the first fully open “omnimodel” that can understand and generate text, images, video, ambient sound and actions.

Under the hood, Cosmos 3 uses a mixture-of-transformers design that pairs a reasoning transformer with a generation transformer, so the model works out object interactions and motion before it produces video or action trajectories. NVIDIA says developers can use one model family in three ways: as a vision language model that understands scenes, as a world simulator that predicts future states and generates synthetic data, or as the backbone for “world action models” that train robots on specific tasks.

The family comes in three sizes, according to an NVIDIA blog post from August: Cosmos 3 Super (64B parameters) for high-fidelity world modeling, Cosmos 3 Nano (16B) for efficient reasoning and post-training, and Cosmos 3 Edge (4B) for on-device use on NVIDIA Jetson, RTX and DGX hardware. The models are released under the Linux Foundation’s OpenMDW 1.1 license, which lets teams post-train them on their own data and hardware.

The platform has kept growing since launch. In October NVIDIA added Cosmos Data Factory, five cloud-native blueprints for generating autonomous driving datasets, synthetic videos of rare or unsafe events, defect images for inspection models, and weather and lighting variations of existing footage. In November it published four new Cosmos 3 Nano checkpoints for interactive simulation, including models that stream synchronized camera and lidar data for driving simulation and predict video from a two-armed robot’s actions.

How Omniverse and Cosmos Work Together

NVIDIA’s own summary is simple: Omniverse provides the simulation technologies to build, render and test structured 3D environments, Cosmos provides models that reason, generate world states and learn actions, and developers use them together by feeding Omniverse simulation outputs into Cosmos models. In practice, a typical workflow looks like this:

  • Build the digital twin. Engineers bring CAD files, facility layouts and sensor setups into OpenUSD and use Omniverse to make the scene physically accurate.
  • Simulate the ground truth. Omniverse physics and sensor simulation produce depth, segmentation and motion data that is correct by construction.
  • Multiply and enrich with Cosmos. Cosmos turns those structured outputs into photorealistic, varied footage. NVIDIA’s Sim-Transfer checkpoint, for example, takes a simulator’s depth, segmentation and edge controls plus a text prompt and renders realistic video, while the physics stays with the simulator.
  • Train, test and repeat. Teams post-train robot or vehicle policies on the blended data, then evaluate them in closed-loop simulation before any real-world trial.

The payoff is coverage. NVIDIA argues that rare, long-tail events are exactly the data that is hardest and most dangerous to collect, and that the combination of OpenUSD and Omniverse cuts the rework that piles up every time an asset, sensor or condition changes.

Who Is Using It

Adoption is broad and spans the three big physical AI markets. In NVIDIA’s March GTC announcement, FANUC, ABB Robotics, YASKAWA and KUKA, with a combined installed base of more than 2 million robots, said they are integrating Omniverse libraries and Isaac simulation into their virtual commissioning tools. KION Group, working with Accenture, is using Omniverse to build warehouse digital twins for training autonomous forklifts. In healthcare, CMR Surgical is using Cosmos-based simulation to train and validate robotic intelligence for its Versius surgical system.

On the Cosmos side, NVIDIA lists Doosan Robotics, LG Electronics, Samsung Electronics and Skild AI in robotics, and Li Auto among autonomous vehicle developers. The October update added concrete numbers: Porsche is leading a shared autonomous driving data effort across Volkswagen Group brands using Cosmos Data Factory through Voxel51 on Microsoft Azure, which NVIDIA says is expected to cut repeated physical test drives by up to 90%. ANYbotics generated more than 1,500 synthetic video clips to teach its inspection robots to spot liquid spills.

Industrial software vendors are wiring Omniverse into tools engineers already use. On November 16, Rockwell Automation announced it has integrated Omniverse, RTX rendering and OpenUSD into its Emulate3D software, so manufacturers can simulate full-scale factories and test controls before commissioning equipment.

What It Means for Business and IT Leaders

For organizations exploring robotics, automation or vision AI, a few practical lessons stand out:

  • Treat 3D data as a strategic asset. Digital twins are only as useful as the data behind them. Standardizing on OpenUSD early makes CAD, facility and sensor data reusable across simulation, training and operations.
  • Synthetic data changes the economics. Rare failures, defects and hazards can be generated rather than staged. Early adopters report fewer physical tests and faster cycles, though results will vary by domain.
  • Open models mean ownership, and responsibility. Open weights let teams specialize models on proprietary data and run them on their own infrastructure, but they also put evaluation and governance squarely on the buyer.
  • Plan for the compute. World model training and simulation are GPU-intensive. NVIDIA and partners offer the tools through clouds including Microsoft Azure, Nebius and CoreWeave, so a pilot need not start with new hardware.
  • Validate before you trust. NVIDIA itself notes that the work ahead is measuring where these models faithfully support policy learning and where they fall short. Simulation results still need real-world checks.

Conclusion

Omniverse and Cosmos solve two halves of the same problem. One builds virtual worlds that obey physics; the other learns from those worlds, multiplies them and turns understanding into action. With Omniverse now free to use, Cosmos 3 released as open models, and companies from Porsche to Rockwell Automation building on the stack, physical AI is moving from research demos toward everyday engineering practice. The organizations that get their 3D data and simulation pipelines in order now will be best placed to benefit.

Sources