At SIGGRAPH 2024, NVIDIA CEO Jensen Huang was first interviewed by Lauren Goode, Senior Writer of Wired, then he in turn interviewed Mark Zuckerberg, CEO of Meta. During his talk with Goode, Huang revealed how users can use text or verbal prompts to automatically generate 3D characters, objects, and scenes. It was part of a collaborative project with stock-art merchant Shutterstock and marketing firm WPP.
"We taught AI how to speak OpenUSD [3D file format for NVIDIA Omniverse]. So the girl [the user] is speaking to Omniverse, Omniverse can generate USD, then Omniverse uses the USD prompt to find the object from its catalog, and then Generative AI uses these conditions to generate the scene. So the work that you do will be much, much better controlled," said Huang.
Such on-demand content generation is now possible with 2D AI programs such as MidJourney, Dall-E, and Microsoft Image Creator. Replicating the same workflow in 3D is expected to radically change the way people design and model 3D objects in manufacturing, engineering, architecture, films, and games.
During his talk with Huang, Zuckerberg said, "Just like every business now has an email, in the future, every business will probably have an AI agent." Echoing this, Huang said, "Everybody can have an AI. In our company, I want every engineer, every software developer to have an AI."
In Meta's WhatsApp chat program, the [/imagine] prompt allows you to generate images based on text using Meta AI.
For more NVIDIA news from SIGGRAPH, read "NVIDIA Advances Humanoid Robotics Development."
During SIGGRAPH, NVIDIA announced the release of more NIMs (NVIDIA Inferencing Microservices) or APIs. They allow developers to create and offer applications built on NVIDIA technology components, such as Generative AI and OpenUSD. NVIDIA writes, "The world’s first generative AI models for OpenUSD development, developed by NVIDIA, will be available as NVIDIA NIM microservices. The models enable developers to incorporate generative AI copilots and agents into USD workflows, broadening the possibilities in 3D worlds and helping speed the adoption of USD across a new range of industrial sectors, like manufacturing, automotive and robotics."
The new Generative AI NIM uses USD as its 3D language to create objects and scenes based on user input. Though it originated in the film industry, USD is seeing growing adoption in manufacturing and engineering. Siemens, for example, has a partnership with NVIDIA to integrate OpenUSD workflows into its Simcenter portfolio. NVIDIA cofounded the Alliance for Open USD (AOUSD) along with Pixar, Adobe, Apple, and Autodesk.
Previously NVIDIA demonstrated the ability to generate standard warehouse items with natural language prompts with its AI Room Generator Extension. The code is available from GitHub, and is powered by NVIDIA DeepSearch and GPT 4. When creating environments such as warehouses and reception areas, instead of dragging and dropping items from a library, users can use prompts, such as “add common items found in a warehouse/reception area.”


Since its founding in 1993, NVIDIA (NASDAQ: NVDA) has been a pioneer in accelerated computing. The company’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics, ignited the era of modern AI and…
Cut Retrieval-Augmented Generation (RAG) Hallucinations by 50%
Most teams hit the same wall with enterprise AI: LLMs that hallucinate, pipelines that don’t scale, and infrastructure that’s harder to design than the models themselves.
Kenneth Wong is Digital Engineering's resident blogger and senior editor. Email him at [email protected] or share your thoughts or suggestions at digitaleng.news/facebook.
Follow DE
Join over 90,000 engineering professionals who get fresh engineering news as soon as it is published.