World Models: The Architecture Behind Spatial Intelligence
Summary
A language model's context is a one-dimensional sequence of tokens with no notion of where anything is. That is fine for text and structurally wrong for space, which is why video generators drift, forget rooms they just rendered, and cannot answer what is behind the camera.
- • Separated the four architectural families — latent dynamics, joint-embedding prediction, pixel-space video simulation, and geometry-native generation — by what each one actually stores as state.
- • Traced why frame-by-frame autoregression accumulates drift, and what an explicit 3D representation changes about consistency.
- • Examined spatial context: grounding every image and depth map in the context at an explicit camera pose rather than a sequence position.
- • Reviewed the benchmark asymmetries in current world-model evaluations, including native-interface advantages in camera-control comparisons.
- • Recurrent state-space models: deterministic recurrence plus stochastic latents for dynamics
- • Joint-embedding prediction in representation space instead of pixel space
- • Autoregressive diffusion transformers over rectified flow in a compressed latent
- • 3D Gaussian splatting, point clouds and depth maps as first-class model outputs
- • Persistence and drift: why consistency is a representation problem, not a sampling problem
- • The dividing question between architectures is what survives between frames — nothing, a latent vector, or an explicit geometric scene.
- • Generating convincing imagery of a world is not the same as modelling it; physics, metric scale and collision geometry are separate problems that current systems mostly do not solve.
- • Spatial grounding of context is the genuinely new idea, and it is the part most likely to persist across labs.
Introduction
A transformer's context is a line. Token 1, token 2, token 3, positions encoded along a single axis, and the whole apparatus of attention built on the assumption that "where something is" means "how far back in the sequence." For text that is exactly right. Language is a sequence.
Space is not. A room has no reading order. The chair is not "before" the window; it is two metres to the left of it and slightly nearer the camera. When you ask a video model to pan left and then pan back, nothing in its architecture obliges the chair to still be there. It re-generates what it thinks should plausibly appear, conditioned on recent frames, and the chair comes back a different colour — or does not come back at all.
That failure mode is the entire subject of this post. World models are the family of architectures built around the claim that an AI system needs an internal representation of an environment and its dynamics — not a description of it, and not a plausible-looking picture of it, but a state you can query, update, and act inside.
The idea is old. Schmidhuber proposed recurrent networks predicting future states from observations in 1990. Ha and Schmidhuber revived it in 2018 with agents that learned to drive and play games entirely inside self-generated simulations. What changed in 2025–2026 is that the representations got good enough to be useful outside a research environment, and a genuinely new architectural idea arrived: grounding the model's context in 3D space rather than in sequence position.
This post is about the architecture class, not about any one lab's model. Several groups are shipping in this space and converging on a small number of designs; the designs are the durable part.
The Four Families
Everything in this field can be sorted by one question: what survives between one moment and the next?
These are not competing answers to one question. They are answers to different questions, and the most interesting 2026 systems blur the last two.
1. Latent Dynamics: The Classical Form
The oldest working formulation splits the problem into three pieces:
The dynamics model is the world model. Everything else is plumbing.
The recurrent state-space model (RSSM) that underpins the Dreamer line splits the latent into two parts, and the split is the whole trick:
- a deterministic recurrent path (a GRU, typically) that carries long-range context forward reliably
- a stochastic latent sampled each step, which absorbs the genuine uncertainty about what happens next
Purely deterministic dynamics cannot represent "the die might land on any face." Purely stochastic dynamics forget. Carrying both lets the model be confident about the parts of the future that are determined and uncertain about the parts that are not.
What this buys is planning in imagination. Once dynamics are learned, an agent can roll out thousands of candidate action sequences in latent space — no renderer, no simulator, no environment — and pick the one with the best predicted return. Latent rollouts are orders of magnitude cheaper than stepping a real simulator, which is why this family dominated model-based RL.
What it does not buy is a world you can look at. The latents are a few hundred dimensions tuned for control, and the decoder output is usually blurry. That is not a defect; predicting exact pixels was never the objective.
2. Joint-Embedding Prediction: Refusing the Pixel Problem
The second family makes that refusal explicit and architectural.
The argument against pixel prediction is simple. Most of a future frame is unpredictable in detail and irrelevant to behaviour — the precise texture of gravel, the exact motion of every leaf. A model trained to minimise pixel error spends enormous capacity on that noise, and hedges by producing blur, because blur is the pixel-space optimum under uncertainty.
A joint-embedding predictive architecture instead predicts in representation space:
Both branches encode; the predictor maps one representation to the other; the loss lives entirely in latent space. The model is permitted to throw away whatever it cannot predict, because the target is itself a learned representation rather than ground-truth pixels.
The practical consequence is that these models are strong at understanding and planning and produce nothing viewable. Whether that is a limitation or the point depends on what you are building. For a robot policy, a representation that predicts "the object will be roughly there, oriented roughly that way" is worth more than a beautiful frame.
3. Video-Space Simulation: The Renderer Is the Model
The third family throws out latent abstraction and generates pixels directly, frame after frame, fast enough to be interactive.
The notable examples run at around 24 fps and accept continuous control input — you steer, and the next frame is generated conditioned on your input and the recent past. There is no explicit 3D representation anywhere in the system. No mesh, no radiance field, no splats. The "world" exists only as the model's conditioning on frames it recently emitted.
This is remarkable and it has a structural problem: the world's memory is a context window.
Turn around, walk back, and the room you left is not retrieved — it is re-imagined. Small errors compound because each frame conditions on the last, so drift is not a bug to be patched but the accumulated consequence of the generation scheme. Extending the context helps and costs quadratically; the failure gets postponed rather than fixed.
These systems are genuinely impressive within their horizon, and the horizon is the architecture.
4. Geometry-Native Generation: Making the Scene the State
The fourth family fixes persistence by refusing to leave the world implicit. The model's output is a 3D representation, and the images you see are rendered from it.
3D Gaussian splatting has become the dominant choice, and it is worth understanding why. A splat scene is a large collection of semi-transparent anisotropic Gaussians — position, covariance, colour, opacity. Rendering is a sort-and-blend rasterisation rather than the per-pixel ray marching that neural radiance fields require, which makes it fast enough for real-time viewing on ordinary hardware, including phones and headsets.
More importantly for architecture: a splat scene is explicit, persistent, and exportable. It exists as a file. Turning around cannot change it, because nothing is being regenerated — you are moving a camera through fixed geometry. Consistency stops being a property the model has to maintain and becomes a property of the representation.
Typical outputs in this family:
The cost is dynamics. A splat scene is furniture, not weather. Most systems in this family generate static or near-static environments, and motion, physics and interaction are supplied by whatever engine consumes the export.
5. Spatial Context: The Actual New Idea
Here is the 2026 development that seems most likely to outlive any particular model.
A language model encodes inputs into a context and generates conditioned on it, with each element carrying a position along one axis. The newer world models keep that structure and replace sequence position with 3D camera pose. Every image, every depth map in the context is tagged with where in space it was observed from.
The consequences are larger than they first appear:
Camera control becomes exact rather than described. You do not prompt for "a slow crane shot"; you supply the camera path as a geometric input, and the model generates the view from it. Control moves from natural language into the type system of the model.
Unrelated reference images become one scene. Six photographs of a building, taken at different times from different angles, can be placed in a common frame and interpolated between — because the context knows where each was taken from, not merely what order they arrived in.
2D and 3D stop being separate pipelines. A model that natively ingests images and depth can emit either: render a frame, or write out the geometry. Reconstruction and generation become the same operation run in different directions, which is why a single omni model can compete with specialised reconstruction systems.
The generator sitting on top of this is typically a multimodal autoregressive diffusion transformer: autoregressive across chunks or frames, diffusion (as rectified flow) within each, operating in a compressed latent rather than over raw pixels. The hybrid is deliberate — autoregression gives sequential conditioning and unbounded length, diffusion gives high-fidelity detail, and the latent keeps the compute affordable. Current published figures sit around a minute of 1440p output with pixel-accurate camera control.
6. What These Systems Do Not Do
This is where the field's marketing runs well ahead of its evidence, and it is worth being precise.
Generating a world is not simulating one. A model that produces a convincing kitchen from six photographs has demonstrated spatial reconstruction and view synthesis. It has not demonstrated that a dropped glass falls at 9.8 m/s², shatters, and leaves fragments that a robot arm must avoid. Physics, rigid and deformable dynamics, contact and friction are separate problems. In current robotics demonstrations, dynamics generally come from a conventional physics engine elsewhere in the stack, and the world model supplies geometry and sensor views.
Metric scale is usually undisclosed. For a robot, "this wall is 3.2 metres away" and "this wall is 3.2 arbitrary units away" are not the same statement. Reconstruction output that is not metrically calibrated needs an external scale reference before it can be planned against.
Geometry for rendering is not geometry for physics. Splats look right and are not watertight surfaces. Collision geometry, clean topology, UVs and material properties are additional work, not free by-products.
No uncertainty output. These models generally do not say which parts of a reconstruction were observed and which were confabulated to fill gaps. For a renderer, invisible. For an autonomous system planning a path, the difference between measured and imagined geometry is the difference between safe and not.
Benchmarks deserve scrutiny. Camera-control comparisons in this field have a recurring asymmetry: the model built to accept camera poses natively receives poses, while baselines receive a text description of the intended camera move. The winner then wins a camera-following contest conducted in its own native interface. The result is real and the framing is not a general video-quality leaderboard. Reconstruction numbers are frequently vendor-published without independent reproduction, disclosed evaluation code, or confidence intervals, and baseline checkpoints are often unidentified. Architecture details — attention pattern, context-selection policy, maximum frame count, parameter count, training compute — are largely unpublished, which makes efficiency claims unverifiable from outside.
Treat every number in this area as a hypothesis, and reproduce on your own scenes before designing around it.
7. Where They Actually Fit
The honest current shape of a production pipeline:
The three uses that currently justify themselves:
Synthetic training data. Reconstruct a real location once, then render thousands of labelled views under camera paths and lighting you choose. Perception models need data volume; this generates it with geometry known by construction.
Real-to-sim for robotics. Capture the environment, reconstruct it, import the geometry into a physics simulator, train the policy there. The world model handles the perception-to-geometry step, which used to require manual scanning and cleanup. It does not replace the simulator.
Content and previsualisation. Explorable environments for games, film previs and XR, exported as standard assets into existing tooling. This is the most mature application because a static, high-fidelity, exportable scene is exactly what those pipelines already consume.
8. Evaluation Checklist
Before building on a world model:
- Does your task need dynamics, or only geometry? If dynamics, what supplies the physics?
- Is the output metrically scaled, and against what reference?
- Are exports usable downstream — watertight meshes, collision geometry, materials — or only splats for viewing?
- Does the system distinguish observed geometry from generated in-fill, and can you get that mask?
- What is the consistency horizon: is the scene explicit and fixed, or re-generated per frame?
- Is interactivity required at frame rate, or is offline generation acceptable? These are currently different architectures.
- Were benchmark comparisons run through each model's native interface, or was one model handed its preferred input format?
- What is the licence on generated environments derived from captured real locations, and who owns the scene?
- How does quality degrade for regions the input never observed — behind objects, above the ceiling, outside the windows?
The framing worth keeping: language models learned to model text by predicting it. It does not follow that a model learns to understand space by predicting pictures of it. The interesting bet in this architecture class is that some explicit spatial structure — poses in the context, geometry in the output — is required, and that predicting pixels alone will not get there. That bet is not settled. It is, at least, now testable.
References
- Omni world model architecture, spatial context and camera-grounded generation: https://www.worldlabs.ai/blog/atlas
- Multimodal world model, Gaussian splat output and editing pipeline: https://www.worldlabs.ai/blog/marble-world-model
- Critical analysis of world-model benchmark methodology and undisclosed architecture: https://kingy.ai/blog/world-labs-atlas-world-model-deep-dive/
- World models overview and history: https://en.wikipedia.org/wiki/World_model_(artificial_intelligence)
- Ha and Schmidhuber, "World Models" (2018): https://arxiv.org/abs/1803.10122
- Dream to Control: learning behaviours by latent imagination (RSSM/Dreamer): https://arxiv.org/abs/1912.01603
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering: https://arxiv.org/abs/2308.04079
- Video generation models as world simulators: https://openai.com/index/video-generation-models-as-world-simulators/
- A Comprehensive Survey on World Models for Embodied AI: https://arxiv.org/pdf/2510.16732