Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n Nvidia has released Cosmos-Transfer1, an innovative AI model that enables developers to create highly realistic simulations for training robots and autonomous vehicles. Available now on Hugging Face, the model addresses a persistent challenge in physical AI development: bridging the gap between simulated training environments and real-world applications.<\/p>\n\n\n\n \u201cWe introduce Cosmos-Transfer1, a conditional world generation model that can generate world simulations based on multiple spatial control inputs of various modalities such as segmentation, depth, and edge,\u201d Nvidia researchers state in a paper published alongside the release. \u201cThis enables highly controllable world generation and finds use in various world-to-world transfer use cases, including Sim2Real.\u201d<\/p>\n\n\n\n Unlike previous simulation models, Cosmos-Transfer1 introduces an adaptive multimodal control system that allows developers to weight different visual inputs\u2014such as depth information or object boundaries\u2014differently across various parts of a scene. This breakthrough enables more nuanced control over generated environments, significantly improving their realism and utility.<\/p>\n\n\n\n Traditional approaches to training physical AI systems involve either collecting massive amounts of real-world data \u2014 a costly and time-consuming process \u2014 or using simulated environments that often lack the complexity and variability of the real world.<\/p>\n\n\n\n Cosmos-Transfer1 addresses this dilemma by allowing developers to use multimodal inputs (like blurred visuals, edge detection, depth maps, and segmentation) to generate photorealistic simulations that preserve crucial aspects of the original scene while adding natural variations.<\/p>\n\n\n\n \u201cIn the design, the spatial conditional scheme is adaptive and customizable,\u201d the researchers explain. \u201cIt allows weighting different conditional inputs differently at different spatial locations.\u201d<\/p>\n\n\n\n This capability proves particularly valuable in robotics, where a developer might want to maintain precise control over how a robotic arm appears and moves while allowing more creative freedom in generating diverse background environments. For autonomous vehicles, it enables the preservation of road layout and traffic patterns while varying weather conditions, lighting, or urban settings.<\/p>\n\n\n\n Dr. Ming-Yu Liu, one of the core contributors to the project, explained why this technology matters for industry applications.<\/p>\n\n\n\n \u201cA policy model guides a physical AI system\u2019s behavior, ensuring that the system operates with safety and in accordance with its goals,\u201d Liu and his colleagues note in the paper. \u201cCosmos-Transfer1 can be post-trained into policy models to generate actions, saving the cost, time, and data needs of manual policy training.\u201d<\/p>\n\n\n\n The technology has already demonstrated its value in robotics simulation testing. When using Cosmos-Transfer1 to enhance simulated robotics data, Nvidia researchers found the model significantly improves photorealism by \u201cadding more scene details and complex shading and natural illumination\u201d while preserving the physical dynamics of robot movement.<\/p>\n\n\n\n For autonomous vehicle development, the model enables developers to \u201cmaximize the utility of real-world edge cases,\u201d helping vehicles learn to handle rare but critical situations without needing to encounter them on actual roads.<\/p>\n\n\n\n Cosmos-Transfer1 represents just one component of Nvidia\u2019s broader Cosmos platform, a suite of world foundation models (WFMs) designed specifically for physical AI development. The platform includes Cosmos-Predict1 for general-purpose world generation and Cosmos-Reason1 for physical common sense reasoning.<\/p>\n\n\n\n \u201cNvidia Cosmos is a developer-first world foundation model platform designed to help Physical AI developers build their Physical AI systems better and faster,\u201d the company states on its GitHub repository. The platform includes pre-trained models under the Nvidia Open Model License and training scripts under the Apache 2 License.<\/p>\n\n\n\n This positions Nvidia to capitalize on the growing market for AI tools that can accelerate autonomous system development, particularly as industries from manufacturing to transportation invest heavily in robotics and autonomous technology.<\/p>\n\n\n\n \n
\n<\/div>How adaptive multimodal control transforms AI simulation technology<\/h2>\n\n\n\n
Physical AI applications that could transform robotics and autonomous driving<\/h2>\n\n\n\n
Inside Nvidia\u2019s strategic AI ecosystem for physical world applications<\/h2>\n\n\n\n