NVIDIA · Stanford University · 2026
Cosmos Policy.
One video model. Policy, world model, and value function.
How a pretrained video diffusion model learns to act, imagine and judge: no new layers, one stage of fine-tuning.
The idea
Video knows physics.
Robots need actions.
A video generator learns how scenes evolve, but outputs pixels, not joint commands. Earlier systems added action heads, inverse-dynamics models and extra training stages. Cosmos Policy instead writes actions, robot state and value into the video itself, as extra latent frames.
Act
Given what the robot sees now, generate an action chunk, plus what will happen and how good it is.
Imagine
Given a proposed action, predict the robot's future camera views and joint state.
Judge
Estimate the expected reward-to-go from the imagined future state.
System model
The architecture.
Unchanged.
Cosmos Policy is NVIDIA's Cosmos-Predict2-2B video diffusion transformer, fully fine-tuned on robot data. Tap any block to see what it does.
Latent frame injection
Everything is
a frame.
Joint angles, a whole action chunk, even a single scalar value are written into the video's latent sequence as if they were frames. The transformer never needs to know the difference.
Ordering follows , so the model can decode left to right. The blank first frame exists because the VAE encodes the first frame alone and compresses the rest four at a time (paper, Sec. 4.1, App. A.1).
How a vector becomes a frame
Step through the four operations from the paper's Appendix A.1 for a tiny action chunk.
Joint training
One loss.
Three functions.
The model is always trained to denoise. What changes is which frames stay clean (conditioning) and which are noised (targets). Pick a mask to see which function is learned.
The denoising objective
Inherited unchanged from the video model (EDM, Karras et al. 2022). The paper's form, Sec. 3:
Policy and world model also predict extra targets (, ). Removing these auxiliary targets drops LIBERO success from 98.5% to 97.0% (Table 4).
A noise schedule built for precision
Generation starts at high σ, but video models rarely train there. Cosmos Policy mixes in a uniform tail so the first denoising steps are accurate.
Inference
From noise
to motion.
At test time the targets start as pure noise at and are denoised step by step. Cosmos Policy stops at , not ≈0, which the authors found more accurate.
Schedule shown: EDM-style spacing with between and (illustrative; the spacing exponent is not reported in the paper).
Model-based planning
Imagine first.
Then act.
With a world model and value function in the same network, the robot can try N actions in its head, picture each outcome, score it, and execute only the best.
Selection rule
Each candidate gets 3 imagined futures × 5 value predictions = 15 votes. The majority mean decides success vs. failure by a fixed threshold and averages only the majority group, which resists outliers when predictions are bimodal.
Why rollouts matter
Trained only on successful demos, the world model is too optimistic: below, it predicts the ziploc slider stays gripped when it actually slips. After fine-tuning on 648 policy rollouts, the "planning model" predicts the true outcome.
Figure 6, Kim et al. (2026)
Evidence
The results.
All numbers below are taken directly from the paper's tables and figures.
on the millimetre-precision ziploc task vs. π0.5 (85.4 vs. 61.5).
on RoboCasa when the policy no longer predicts the future state (62.5 → 44.4).
demos per task on RoboCasa, still the highest average success.
Open problems
What's left
to solve.
per action chunk with planning, using 8 H100 GPUs in parallel. Too slow for dynamic tasks.
real rollouts were needed to make the world model and value function reliable enough to plan.
of best-of-N search, one action chunk ahead. Deeper search is future work.
out-of-distribution ALOHA score: π0.5 still generalises slightly better (Table 3).