NVIDIA Empowers Developers to Build Advanced Simulations Using Frontier AI Agents and Omniverse Libraries

Image: Illustrative image · Amusements.global · CC BY-SA 4.0 · Source

NVIDIA showcases how developers combine frontier AI models like GPT-6 Astra with Omniverse libraries to create sophisticated simulations, facilitating robotics, autonomous vehicles, digital twins, and more with natural-language AI guidance.

NVIDIA highlights innovative uses of frontier AI models integrated with its Omniverse libraries to transform simulation development. Developers are directing AI agents, such as GPT-6 Astra, through natural-language commands to assemble assets, link physics engines, rendering modules, and sensor simulations, culminating in fully functional simulation applications. This approach simplifies creating environments for testing robotics, autonomous driving, digital twin validation, and space modeling.

Key demonstration projects reveal the breadth of this technology. For warehouse automation preparation, a humanoid robot simulation with first- and third-person views was generated using Astra to orchestrate physics (using ovphysx), rendering (ovrtx), scene management (ovstage), and user interfaces (ovui). Autonomous vehicle simulation workflows in environments like San Francisco's Market Street were created to test sensor and driving models through staged asset integration and environmental variability testing.

To enhance digital twins, AI agents compared simulated sensor outputs with recorded real-world camera and lidar data, iteratively refining scene accuracy by modifying OpenUSD assets and checking results against key performance metrics. Robotics research leveraged simulations to test complex movements and disassembly tasks, combining CAD models, physics engines, and AI-generated tool designs.

A notable example includes assembling an OpenUSD model of the International Space Station integrated with live telemetry, rendered and streamed through browser-based applications developed entirely via AI prompting. Similarly, stereo camera captures were turned into editable OpenUSD studios for interaction testing, enabling detailed physical behavior analysis.

Underpinning this ecosystem is ovstage, a high-performance C and Python library providing a shared, GPU-accelerated runtime scene data substrate compatible with Omniverse libraries. ovstage enables efficient manipulation and sharing of USD scene data across multiple simulation components without redundant parsing or memory overhead.

NVIDIA's approach offers a powerful workflow where developers guide AI agents to create, refine, and validate comprehensive simulation applications using advanced physics, rendering, and sensor simulation capabilities integrated seamlessly. This fusion of frontier AI coding models with Omniverse libraries heralds a significant leap in simulation efficiency and complexity, supporting research and development in robotics, autonomous vehicles, digital twins, and beyond.