Waymo is dramatically advancing the realism and scope of its virtual testing environments with a new frontier generative model, built upon Google DeepMind’s most advanced technology. The newly unveiled Waymo World Model leverages DeepMind’s Genie 3 to generate hyper-realistic, multimodal simulations, preparing its autonomous Driver for virtually any scenario—including exceedingly rare events like tornadoes, flooded streets, or even an elephant on the road.
This move underscores simulation’s critical role as a pillar of Waymo’s safety-focused AI development. While the Waymo Driver has now logged nearly 200 million fully autonomous miles on public roads, it has navigated billions more within virtual worlds. The new World Model aims to make these simulated miles more valuable, diverse, and controllable than ever before.
Harnessing Broad World Knowledge for Unseen Scenarios
A key differentiator of the Waymo World Model is its foundation on Genie 3, which was pre-trained on an extremely large and diverse set of videos from across the internet. This grants the system what Waymo terms “emergent multimodal world knowledge”—an understanding of visual and physical concepts far beyond the scope of data collected solely by its own fleet. Through specialized post-training, Waymo translates this vast 2D world knowledge into accurate 3D LiDAR outputs tailored to its proprietary sensor suite, creating complementary camera and LiDAR data that mirrors real-world perception.
The result is an unprecedented ability to generate and explore situations the fleet has never directly encountered, moving beyond reconstruction of logged events to the generation of entirely new, plausible scenarios.
Unprecedented Control for Rigorous Testing
Engineers are equipped with three powerful levers to shape simulations for targeted testing:
- Driving Action Control: Creates a responsive simulator for testing “what if” counterfactuals, allowing assessment of whether the Driver could have acted more confidently or taken alternative actions in past situations.
- Scene Layout Control: Enables customization of road layouts, traffic signal states, and the behavior of other road users, facilitating the creation of specific, challenging scenarios.
- Language Control: Offers the most flexible tool, using simple text prompts to adjust time-of-day, weather conditions, or generate completely synthetic scenes.
This high degree of controllability allows Waymo to systematically stress-test its systems against both uncommon edge cases and subtly altered versions of common events.
From Dashcam to High-Fidelity Simulation in a New City
The model also introduces a practical capability for expansion: converting ordinary dashcam or mobile phone videos into multimodal simulations. This process shows how the Waymo Driver would perceive a new scene with its own sensors, providing a pathway to generate high-fidelity, factual simulations in novel environments without first deploying a full sensor suite. For scaling to new cities, this represents a potentially powerful tool for early exposure and testing.
Acknowledging the computational challenge of simulating longer-duration scenarios, Waymo has also developed a more efficient variant of the model. This leaner version maintains high realism and fidelity while dramatically reducing compute requirements, enabling large-scale simulation of extended interactions, like negotiating a narrow lane.
While Waymo is not alone in pursuing generative world models—with notable efforts from companies like Wayve (with its GAIA-1 model), Waabi (Copilot4D), and Nvidia (Cosmos)—its deep integration with DeepMind’s foundational research and its focus on a full sensor suite (camera + LiDAR) positions the Waymo World Model as a significant frontier advancement.
