Simulation worlds: PhysX, Omniverse, Isaac Sim, and Cosmos

The world layer

This series has climbed a stack. Surrogates learn what physics does; differentiable engines expose how it responds; the world layer decides where it all happens, what observes it, and who can work with it.

Three vocabulary corrections save weeks of confusion. First, Omniverse is not an application; it is a platform of SDKs and libraries (Kit, RTX, USD tooling) on which applications are built, Isaac Sim being the flagship. Second, PhysX and Newton are colleagues, not rivals: Isaac Sim ships both as switchable backends, PhysX carrying two decades of production maturity and Newton carrying differentiability and open governance. Third, a Blueprint is a reference architecture, working code plus a documented workflow, not a product you install; you fork it the way you fork a PhysicsNeMo recipe.

The organizing question of this part: when your problem stops being "solve these equations" and becomes "stand up a believable world, populate it, observe it with realistic sensors, and run fleets of things inside it," what do you reach for, and in what order?

PhysX 5: the production engine

PhysX is the physics engine most people have used without knowing it, decades of games, and now the default engine of Omniverse and Isaac Sim. Version 5 is open source under BSD-3, GPU solver source included.

The repository at NVIDIA-Omniverse/PhysX now hosts the combined stack: the PhysX SDK itself (5.6.x at the time of writing, with GPU source released), the Flow volumetric fluid library, and the Omni PhysX layer, the Kit extensions, USD schemas, and runtime that let PhysX consume physics authored as USD in applications like Isaac Sim.

ovphysx: the engine without the platform

The newest development matters for anyone who wants PhysX without adopting a whole application: pip install ovphysx (0.4.x, explicitly pre-1.0) delivers a self-contained library exposing the PhysX SDK and Omni PhysX behind a C API with Python bindings. It loads USD scenes directly, including from remote URIs, exposes simulation state through DLPack tensors for zero-copy exchange with PyTorch and JAX, supports environment cloning for batched reinforcement learning, and distributes scenes across multiple CUDA devices. In other words: the Isaac Sim physics core as an importable library, the same positioning bare Warp holds in Part II.

Omniverse: anatomy of the platform

Strip away the marketing and Omniverse is four load-bearing pieces: a scene description, an application SDK, a renderer, and a fast data plane between them.

• OpenUSD is the world's file format. Everything in Omniverse is a USD stage: geometry, materials, physics (via the UsdPhysics schemas met in Part II), robots, sensors, and simulation output. Composition and layering are what make multi-team, multi-tool workflows survivable: the plant layout, the robot cells, and the simulation overrides are separate layers composed non-destructively.
• Kit is the application SDK. Every Omniverse application, Isaac Sim included, is a bundle of Kit extensions: Python or C++ modules contributing UI, simulation logic, importers, or services. Building your own tool means writing extensions, not forking an application, which is why the ecosystem now spans hundreds of them.
• RTX is the renderer, and more importantly the sensor engine. Hardware ray tracing gives physically-based light transport in real time, which is pleasant for humans and essential for sensors: the same machinery that draws the scene simulates cameras, lidar returns, and radar with physical fidelity (Section 08).
• Fabric is the data plane. A high-performance runtime representation of scene data that lets simulation write and rendering read at scale without round-tripping through USD's authoring layer every frame; the reason thousand-robot scenes are steppable at all.

The licensing map, plainly

The pieces you build on have different terms, and the differences are decision-relevant. PhysX is BSD-3. Isaac Sim is Apache 2.0 on GitHub. The Omniverse Kit SDK is free to develop with, but redistributing Kit inside your own shipped product runs through an Omniverse Enterprise license. For internal engineering tools this rarely bites; for an ISV shipping a simulator it is the first architectural question. The old Launcher-and-Nucleus distribution model has meanwhile given way to SDKs, containers, pip wheels, and cloud APIs; treat any tutorial that begins with the Omniverse Launcher as historical.

USD hands-on: your first physics-ready stage

Part II introduced OpenUSD as a format. Here it becomes a skill: ten lines of Python that author a physics-ready scene, and the five concepts those lines quietly use.

from pxr import Usd, UsdGeom, UsdPhysics, Gf

stage = Usd.Stage.CreateNew("cell.usda")
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)     # Isaac convention: Z up
UsdGeom.SetStageMetersPerUnit(stage, 1.0)           # and meters, not cm

UsdGeom.Xform.Define(stage, "/World")
UsdPhysics.Scene.Define(stage, "/World/PhysicsScene")

crate = UsdGeom.Cube.Define(stage, "/World/Crate")
crate.GetSizeAttr().Set(0.4)
UsdGeom.XformCommonAPI(crate).SetTranslate(Gf.Vec3d(0, 0, 1.0))

UsdPhysics.RigidBodyAPI.Apply(crate.GetPrim())      # it falls
UsdPhysics.CollisionAPI.Apply(crate.GetPrim())      # it collides
UsdPhysics.MassAPI.Apply(crate.GetPrim()).GetMassAttr().Set(12.0)

stage.GetRootLayer().Save()                          # open this in Isaac Sim

The file this writes is plain ASCII (.usda); open it in a text editor and every line of the Python maps to a visible line of scene description, which is the fastest way to build intuition. The concepts at work:

• Stage, prim, property. A stage is the composed scene; prims are its nodes (/World/Crate); properties are their attributes (size, translate) and relationships. Everything else in the platform is machinery for producing and consuming these three.
• API schemas versus typed schemas. UsdGeom.Cube is a typed ("IsA") schema: the prim is a cube. UsdPhysics.RigidBodyAPI is an API schema: a capability applied onto an existing prim. Physics in USD is almost entirely applied API schemas, which is why any imported CAD mesh can be made dynamic after the fact.
• Composition arcs. Real projects never author monoliths: a plant stage sublayers the layout, references each robot's asset file, payloads heavy geometry for deferred loading, and switches variants for configurations. Edits land in your layer; the referenced assets stay pristine. This is the mechanism behind every "non-destructive collaboration" claim in Section 03.
• ASCII versus crate. .usda is diffable text for things you author by hand; .usdc (crate) is the binary for heavy data; plain .usd can be either. Author small things as usda so git review works.
• Conventions before content. Z-up and meters-per-unit 1.0 are the Isaac Sim conventions; assets arriving Y-up in centimeters (most DCC exports) are the single most common source of "my robot is the size of a stadium" bugs, treated properly in Section 16.

Isaac Sim 6: the reference simulator

Isaac Sim is where the stack composes: PhysX or Newton for dynamics, RTX for rendering and sensors, USD for the scene, Kit for extensibility, all open source under Apache 2.0 and installable with pip.

The capability set, briefly: import from URDF, MJCF, CAD (including Onshape), or USD; tune robots for physics accuracy versus computational cost; simulate with either physics backend and switch between them; attach RTX and physics-based sensors; generate synthetic data through Replicator; bridge to ROS 2 for stack-in-the-loop testing; and hand off to Isaac Lab (Part II) for reinforcement learning at scale. Real-world capture flows in through NuRec neural reconstruction, closing the loop from physical facility to digital twin.

Architecturally, Isaac Sim is deliberately a reference application: a curated bundle of Kit extensions you are expected to extend, strip down, or embed. The three consumption patterns, in increasing order of commitment: use the GUI application as-is for interactive scene work; drive it headlessly from Python scripts for pipelines and CI; or cherry-pick its extensions into a custom Kit application that is yours. The pip distribution (an isaacsim metapackage, 6.x at the time of writing) makes the middle path, Python-first automation, dramatically less painful than it once was.

Scripting Isaac Sim: the standalone idiom

Everything durable you build will run headlessly from Python, not from the GUI. The standalone-script skeleton is short, but its two rules are non-negotiable.

from isaacsim import SimulationApp
simulation_app = SimulationApp({"headless": True})   # RULE 1: this line comes FIRST,
                                                      # before any other omni/isaac import

from isaacsim.core.api import World                   # namespaces moved from omni.isaac.*
from isaacsim.core.utils.stage import open_stage      # in recent releases; pin and check

open_stage("cell.usda")                               # the stage from Section 04
world = World(stage_units_in_meters=1.0)
world.scene.add_default_ground_plane()

world.reset()                                          # RULE 2: reset before stepping;
for i in range(600):                                  # it initializes physics handles
    world.step(render=False)

simulation_app.close()
# API sketch for the 6.x generation; the docs of your pinned version are truth.

Why the import-order rule exists: SimulationApp boots the Kit runtime, and every omni.* and isaacsim.* module is a Kit extension that only exists once the runtime is up. Importing them earlier fails with errors that do not mention this cause, which is why it is the first thing every Isaac veteran checks in someone else's broken script.

Bolting on synthetic data

import omni.replicator.core as rep

crates = rep.get.prims(path_pattern="/World/Crate.*")
cam = rep.create.camera(position=(2.5, 2.5, 2.0), look_at=(0, 0, 0.5))
rp = rep.create.render_product(cam, (1024, 768))

writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="_out/cell_ds", rgb=True,
                  bounding_box_2d_tight=True, semantic_segmentation=True)
writer.attach([rp])

with rep.trigger.on_frame(num_frames=200):            # 200 randomized frames
    with crates:
        rep.modify.pose(
            position=rep.distribution.uniform((-0.6, -0.6, 0.6), (0.6, 0.6, 1.4)),
            rotation=rep.distribution.uniform((0, 0, -180), (0, 0, 180)))
rep.orchestrator.run()

Two details that save an afternoon: semantic outputs (segmentation, labeled boxes) require prims to carry semantic labels, applied via the Semantics schema or the GUI's semantics tools, before the writer has anything to label; and the writer's output directory fills with per-annotator subfolders whose naming the docs specify, so inspect one frame end to end before generating fifty thousand.

Building your own: Kit apps and extensions

The step from "scripting Isaac Sim" to "shipping a tool" is the Kit template repository: scaffolded applications and extensions with the build, test, and packaging plumbing already wired.

# The whole lifecycle, four commands
git clone https://github.com/NVIDIA-Omniverse/kit-app-template.git
cd kit-app-template
./repo.sh template new      # pick: app (USD Composer / Explorer / Viewer)
                            #   or extension (Basic Python / Python UI / C++)
./repo.sh build
./repo.sh launch

The application templates are honest starting points, not toys: USD Composer for authoring-style tools, USD Explorer for large-scene review, USD Viewer for a viewport-only app built to be streamed into a web page. The extension templates scaffold the unit you will actually write most: a Kit extension is a folder with a extension.toml manifest (name, version, dependencies on other extensions) and a Python module exposing lifecycle hooks:

import omni.ext

class FleetHealthOverlay(omni.ext.IExt):
    def on_startup(self, ext_id):
        # build UI, subscribe to stage events, register commands
        print("[fleet.health.overlay] up")

    def on_shutdown(self):
        # tear down everything you created
        print("[fleet.health.overlay] down")

Because Isaac Sim itself is a bundle of extensions, anything you build this way loads into it directly: a telemetry overlay, a custom importer for your plant's asset conventions, a one-button scenario runner for the test team. The template's tooling also handles packaging and the streaming-deployment layers (Kubernetes-based Kit App Streaming, or NVCF for managed cloud), which is the sanctioned route when the tool needs to reach people without workstations, with the Section 03 licensing note applying the moment Kit is redistributed inside a product.

Sensors, rendering, and synthetic data

For perception work, the renderer is not cosmetics; it is the physics of observation. A simulated camera is a light-transport problem, a lidar is a laser-transport problem, and getting them right is what makes synthetic data worth training on.

Sensor RTX

Omniverse's sensor simulation runs the same ray-tracing machinery as rendering, but for sensor physics: camera models with lens effects and rolling shutter, lidar with beam patterns, multiple returns, and material-dependent reflectance, and radar-class sensors, exposed through Sensor RTX APIs that the Mega and AV-simulation Blueprints are built on. The fidelity argument is simple: perception models trained on naive renders learn the render's tells; sensors simulated as physics narrow that gap at the source.

Replicator: randomization as a pipeline

Replicator is the synthetic-data-generation framework inside Isaac Sim: programmatic control over scene content, materials, lighting, poses, and sensor parameters, with automatic ground-truth labeling (bounding boxes, segmentation, depth, normals) emitted alongside every frame. It is domain randomization (Part II, Section 15) industrialized: define distributions over the world, sample datasets from them, and version the generator instead of the data.

Assets that behave: SimReady

A photoreal mesh is not a simulation-ready asset; it needs mass, materials, collision geometry, and semantic labels. The SimReady convention packages exactly that, and generative tooling now accelerates the drudgery: USD Code and USD Search NIM microservices generate and retrieve OpenUSD assets from text, and the Edify SimReady model auto-labels existing 3D assets with physical attributes, NVIDIA's cited example being a thousand objects processed in minutes rather than a forty-hour week.

Blueprints and digital twins: the gallery

Eight reference architectures and workflows, filterable by domain, each a fork-and-adapt starting point rather than a demo.

The sample repository shelf

Every repository below was verified live at the time of writing. Together they cover the full path from first USD file to shipped Kit application; the third column says when to open which.

Cosmos: worlds that are generated, not constructed

Everything above builds worlds the classical way: model, compose, render. Cosmos, NVIDIA's world-foundation-model family, generates them: video-native neural models that predict, transfer, and reason about physical scenes.

Launched at CES 2025 and now at its third generation alongside models like Isaac GR00T and Alpamayo, the family divides by verb. Predict models generate physically plausible future video from prompts and context, synthetic worlds by sampling. Transfer models re-render structured input (a simulation layout, a segmentation, a coarse scene) into photoreal variants, which is how a single Omniverse scenario becomes a thousand weather-and-lighting variants. Reason models answer questions about physical scenes, the perception-and-judgment end. The AV pipeline is the flagship consumer: the AV-simulation ecosystem's public corpus includes tens of thousands of Cosmos-generated driving clips aimed at exactly the rare events real fleets under-sample.

The honest relationship between the two world-making methods

Constructed simulation gives controllability, ground truth by construction, and physics you can audit; it struggles with visual diversity and the cost of authoring the long tail. Generative world models give diversity and photorealism at negligible marginal cost; they struggle with physical guarantees, controllability, and labels. The emerging pattern is not a contest but a sandwich: construct the scenario in Omniverse (controllable, labeled), amplify it through Cosmos Transfer (diverse, photoreal), and validate perception against both. Treat WFM physics the way Part I taught you to treat surrogate physics: useful, fast, and to be cross-checked against the auditable model before anything safety-adjacent depends on it.

The automotive map

One table to place every OEM function on the stack, and a closing argument for the health-management reader.

The VHM argument

Vehicle health management enters this layer from three doors. First, the twin as context: telemetry interpreted against a live facility or vehicle twin is worth more than telemetry in a dashboard, because the twin supplies the operating context (duty, route, ambient, neighbors) that turns an anomaly into a diagnosis. Second, degraded-fleet simulation: the same Mega-style machinery that tests healthy fleets can inject the degradations Part II taught you to identify, a worn damper here, a drifting sensor there, and measure operational impact before deciding maintenance policy; failure-mode injection in a twin is DFMEA's quantitative sibling. Third, synthetic failure data: Replicator and Cosmos generate the labeled failure observations (visual, and increasingly signal-level) that real fleets are too healthy, or too poorly instrumented, to provide. The world layer, in short, is where health management stops being a per-signal discipline and becomes a per-system one.

Getting started

All three entry points are now pip-shaped or git-shaped. Versions verified on PyPI at the time of writing.

# Isaac Sim 6.x as Python packages (the metapackage pulls the stack)
pip install isaacsim --extra-index-url https://pypi.nvidia.com

# PhysX as a standalone library (pre-1.0; API still settling)
pip install ovphysx

# Or build Isaac Sim from source
git clone https://github.com/isaac-sim/IsaacSim.git

# PhysX SDK + Omni PhysX source
git clone https://github.com/NVIDIA-Omniverse/PhysX.git

A sensible first week

Day 1: install Isaac Sim, load a sample scene, and run the basic tutorials until the stage, prims, and play-button model feels native. Day 2: import one of your own assets, a URDF robot or a CAD fixture, and give it physics. Day 3: attach an RTX camera and a lidar, and write frames plus ground truth to disk with Replicator; this single exercise touches sensors, SDG, and the Python API at once. Day 4: script the whole thing headlessly, no GUI, from one Python file, because everything durable you build will run this way. Day 5: switch the physics backend from PhysX to Newton on the same scene and diff the trajectories, which is Parts II and III shaking hands.

Project Zero: a cell twin that writes labeled data

A complete first project that touches every layer of this guide once, sized for a focused weekend. Deliverable: a physics-ready stage you authored, a headless script that simulates it, and a labeled synthetic dataset it produced.

• Step 1 · Environment. Fresh Python environment, pip install isaacsim --extra-index-url https://pypi.nvidia.com (Section 13), and confirm the GUI launches and a sample scene plays. Do not skip the GUI check; every later mystery is easier to debug against a simulator you have seen working.
• Step 2 · Author the stage. Run the Section 04 script, then extend it: three crates instead of one (/World/Crate_01…03), a static table under them (collision, no rigid body), and semantic labels on the crates so Replicator has something to annotate. Open the result in the GUI, press play, and watch the crates drop; if they fall through the table, you have met collision-API-forgotten, the classic first bug.
• Step 3 · Simulate headlessly. The Section 06 standalone skeleton, pointed at your stage: open, reset, step 600 frames, close. Confirm it runs on a machine with no display attached, because that is where every pipeline eventually lives.
• Step 4 · Generate the dataset. Bolt on the Section 06 Replicator block: randomized crate poses, one camera, RGB plus tight 2D boxes plus segmentation, 200 frames to _out/cell_ds. Open frame 0 and frame 199 side by side and verify the labels track the randomization before scaling the frame count.
• Step 5 · Inspect like a skeptic. Plot the distribution of crate positions recovered from the labels against the distribution you requested; mismatches here are randomizer bugs that would silently bias a trained model. This ten-line pandas check is the habit that separates dataset generators from dataset believers.
• Step 6 · Stretch goals, pick one. (a) Import a URDF robot next to the table and step it; (b) switch the physics backend to Newton on the same stage and diff crate trajectories against PhysX, the Parts II–III handshake; (c) wrap the whole run as a Kit extension button using Section 07's skeleton, and you have shipped your first tool.

A 30-day playbook

Project Zero, industrialized: four weeks from installation to a defensible pilot twin, each week with an exit criterion so drift is visible.

Pitfalls the tutorials will not warn you about

Seven failure modes that account for most first-month misery on this stack, each cheap to avoid once named.

Choosing your stack, and the series in one view

The three parts compose into one sentence: learn the physics where solving is too slow (I), differentiate it where questions are gradients (II), and situate it in a world where systems, sensors, and people meet it (III). Any serious vehicle program will eventually run all three layers against the same USD scene, which is precisely why the scene format, not any single engine, is the load-bearing standard.

Glossary

The working vocabulary of the world layer, in one place.

Stage

The composed USD scene: the result of opening a root layer and resolving every composition arc it pulls in.

Prim

A node in the scene hierarchy (/World/Crate); carries a type, applied schemas, properties, and children.

Layer

A single USD file's worth of opinions. Stages are stacks of layers; stronger layers override weaker ones non-destructively.

Composition arcs

The mechanisms that assemble stages from layers: sublayer (stacking), reference (asset inclusion), payload (deferred-load reference), variant (switchable alternatives), inherits/specializes (class-like broadcast edits).

Typed vs API schema

Typed ("IsA") schemas define what a prim is (Cube, Mesh, Camera); API schemas apply capabilities onto existing prims (RigidBodyAPI, CollisionAPI, MassAPI). USD physics is almost entirely applied APIs.

usda / usdc

ASCII and binary (crate) encodings of USD. Author small things as diffable usda; let heavy data live in usdc.

Kit

The Omniverse application SDK. Every Omniverse app, Isaac Sim included, is a configured bundle of Kit extensions.

Extension

The unit of Kit functionality: a manifest (extension.toml) plus Python or C++ implementing lifecycle hooks; hundreds ship with the platform.

Fabric

The high-performance runtime data plane between simulation and rendering; the reason live state and the USD authoring layer are not the same thing (Pitfall 3).

RTX renderer / Sensor RTX

The hardware-ray-traced renderer, and the same machinery repurposed as physically-based camera, lidar, and radar simulation.

Replicator

Isaac Sim's synthetic-data framework: programmatic randomization of scenes plus automatic ground-truth writers (boxes, segmentation, depth, normals).

SimReady

The asset convention for simulation-usable content: geometry plus mass, materials, collision, and semantic labels, not just a pretty mesh.

Semantic labels

Class annotations applied to prims that Replicator's annotators read; no labels, no labeled data.

NuRec

Neural reconstruction: real-world captures turned into simulation-ready environments; the physical-to-digital direction.

Blueprint

A reference architecture with working code and a documented workflow (Mega, AV sim, DSX); forked and adapted, not installed.

NIM

NVIDIA Inference Microservice: a pretrained model behind a container and API (USD Code, USD Search, DoMINO Aero from Part I).

World foundation model (WFM)

A generative model of physical scenes (Cosmos Predict / Transfer / Reason); worlds by sampling rather than construction.

Digital twin

A maintained virtual counterpart of a physical asset or facility, coupled to it by data; distinguished from a mere model by the coupling.

Virtual commissioning

Validating control software against the twin before, or instead of, commissioning time on the physical equipment.

Learning resources

References

1. NVIDIA PhysX SDK 5 and Omni PhysX, open source (BSD-3), GPU solver source included; Flow fluid library. github.com/NVIDIA-Omniverse/PhysX
2. ovphysx: self-contained Python library for USD-based physics simulation with DLPack interoperability, batched environment cloning, and multi-GPU scene distribution (pre-1.0; 0.4.x on PyPI at time of writing). Documentation
3. NVIDIA Isaac Sim: open-source (Apache 2.0) robotics simulation reference application on Omniverse; PhysX and Newton switchable backends; URDF/MJCF/CAD/USD import; Replicator; ROS 2 bridges. github.com/isaac-sim/IsaacSim
4. Isaac Sim developer overview and licensing FAQ, including Kit redistribution terms and NuRec real-world capture. developer.nvidia.com/isaac/sim
5. NVIDIA-Omniverse GitHub organization: Blueprints and workflows, kit-cae, USD validation and profile tooling, and Agent Skills / MCP servers for Kit, USD, OmniUI, and the RTX SDK. github.com/NVIDIA-Omniverse
6. Pixar / AOUSD, OpenUSD: Universal Scene Description. openusd.org

7. NVIDIA CES 2025 announcements: Cosmos world-foundation-model platform; Mega, AV simulation, spatial streaming, and real-time CAE digital twin Blueprints; USD Code / USD Search NIMs; Edify SimReady. Coverage: Robotics 24/7, The Robot Report.
8. NVIDIA blog, "Into the Omniverse: World Foundation Models Advance Autonomous Vehicle Simulation and Safety" (AV Blueprint workflow, Cosmos-generated clip corpus, Halos). blogs.nvidia.com
9. NVIDIA blog, GTC 2026: Cosmos 3, Isaac GR00T N1.7, Alpamayo 1.5, the Physical AI Data Factory Blueprint (Cosmos + OSMO), the Omniverse DSX Blueprint, and the KION / Accenture / Siemens Mega deployment for GXO. blogs.nvidia.com
10. NVIDIA Cosmos world foundation models: Predict, Transfer, and Reason families. developer.nvidia.com/cosmos

11. Omniverse Kit App Template: scaffolded Kit applications (USD Composer / Explorer / Viewer) and extension templates with build, packaging, and streaming tooling. github.com/NVIDIA-Omniverse/kit-app-template
12. OpenUSD Code Samples (task-shaped snippets in Python, C++, USDA) and OpenUSD learning resources. OpenUSD-Code-Samples · OpenUSD
13. OpenUSD plugin samples: custom schemas, file-format plugins, dynamic payloads, with build tooling. github.com/NVIDIA-Omniverse/OpenUSD-plugin-samples · Exchange SDK samples: usd-exchange-samples
14. VFI Samples: CAD-to-USD conversion, asset validation, scene optimization, material assignment, and assembly automation for virtual facilities. github.com/NVIDIA-Omniverse/vfi-samples
15. Synthetic data examples for Replicator-based generation, and the IsaacSimZMQ external-process bridge. synthetic-data-examples · IsaacSimZMQ

16. Part I: NVIDIA PhysicsNeMo, a field guide for vehicle engineering (surrogate models, DrivAerML, crash recipes, deployment workflows).
17. Part II: Differentiable physics engines, Warp and Newton (adjoints, system identification, the half-car case study, sim-to-real discipline).