CoRL 2026 · Long-Video World Models

World Memory

Knownness-Conditioned Long-Video Generation

A frozen video diffusion model that reuses what it has seen — and generates only what it hasn't.

A 3D-grounded memory bank  ·  per-token noise scheduling  ·  write-back rollout

World Memory · working title

The problem

Video models re-hallucinate on revisit

Generate a long video where the camera moves away and comes back to a place it already saw.

  • Frame-by-frame diffusion has no memory of 3D.
  • On return, the same wall / table / sky is re-invented differently — flicker, drift, ghosting.
  • Naïvely copying old pixels (splatting) leaves holes, seams, and stale geometry.

GOAL

Revisit ⇒ recall, not re-generate

When the camera returns to a known place, the model should restore the same scene from memory — and still freely generate genuinely new regions.

KEY IDEA

One scalar per token: knownness k

Route each space-time token between reuse-and-repair and generate-from-noise — inside a single frozen diffusion model.

2

The recipe in one line

Memory sets the noise level; diffusion does the rest

OFFLINE

Build a 3D world memory

Depth + poses → a persistent voxel point cloud. Reproject it into every target camera to get z_mem (what should be there), k (how sure), and Xtok (where in 3D).

ONLINE

Per-token noise from k

Wan DiT natively supports a different diffusion timestep per token. Known → low noise (repair). Unknown → high noise (generate). New output is written back into memory.

high k  →  start near z_mem  ·  short denoise    |    low k  →  start from noise  ·  full generation
3

System overview

INPUT PHASE 1 · OFFLINE WORLD MEMORY MEMORY BUNDLE PHASE 2 / 4 · ONLINE DiT OUTPUT Posed Video 832 × 480 · 81 frames out-and-back trajectory → forced revisit Metric Depth + Global Scale Calibration depth matched to camera baseline → 3D addresses only PointMemory — world voxel point cloud • 3D position & average RGB • observation count • per-view directions • color variance → ghosting signal fuse ~11 observed frames · never forgets persistent across the whole rollout Reproject → target cameras z-buffer · adaptive splat radius · depth test z_mem VAE-encoded memory reprojection knownness k unknownknown Xtok per-token 3D world position memory slot bank compact 3D key/value memory Frozen Wan Video DiT + small zero-init trainable modules ① Per-token noise level from k low k → high noise · high k → low noise ② 3D positional encoding (Xtok) same world point = same identity under camera motion ③ Geometry cross-attn · every 5 blocks Xtok queries slot bank · Rope3D relative embedding ④ Confidence-gated shortcut pass memory where k high · suppress where k low Zero-init at start, modules output 0 → behaves like base Wan Generated Video write-back: new frames → hypothesis points unknown → known
4
Phase 1 · offline caching

Video → persistent 3D world memory

  • Out-and-back trajectory (~41 out / ~40 back) forces a revisit — the return path is real GT to score consistency.
  • Metric depth + one global scale matched to the camera baseline. Depth is only a 3D address, never copied as pixels.
  • Back-project ~11 frames into PointMemory: RGB, count, view directions, color variance (ghosting detector).
  • Reproject to each target cam (z-buffer, adaptive splat) → coverage / count / diversity / disagreement maps.

SAVED PER CLIP → one .pt file

x0 (GT video, VAE)z_memknownness kXtok3D slot bankcamera poses

WHY VARIANCE MATTERS

Consistent color across views ⇒ reliable memory. High disagreement ⇒ occlusion / dynamics / geometry error / ghosting — and lowers k.

5
Phase 1 · the knownness field

What makes a region known?

k  =  cover · exp(−β·disagree) · clip( tanh(count/n₀) + 0.3·diversity )

COVER

Is the region reprojected from memory at all?

COUNT

Seen many times?

DIVERSITY

Seen from many viewpoints?

DISAGREE

Do observations conflict? Penalized.

High k only when a region is covered, seen often, from many views, and observations agree.

6
Phase 2 · model

Frozen Wan DiT + four small hooks

① CORE

Knownness → per-token timestep

Wan natively supports a different diffusion timestep per token. Known = low noise (repair); unknown = high noise (generate). No surgery on the backbone.

② GEOMETRY

3D positional encoding (Xtok)

Each token knows its world position and viewing direction → the same world point stays the same identity as the camera moves.

③ RETRIEVAL

Geometry-keyed cross-attention

Every 5 blocks, Xtok queries the slot bank with Rope3D relative embedding — retrieve the same content from any viewing angle.

④ GATING

Confidence-gated shortcut

Residual that passes more memory where k is high, damped where k is low — bad memory can't leak in.

All new modules zero-initialized → day-0 behavior ≈ original Wan

7
Phase 3 · training

Anchor-mixture Diffusion Forcing

Per token: sample noise level t, pick a clean anchor, build the input on the model's familiar noise-to-image manifold:

x_t = (1 − t)·anchor + t·ε
  • Usually anchor = x₀ (real video latent).
  • For memory-covered tokens, ~40% use anchor = z_mem — but only when t ≥ 0.3.
  • Model always receives the true t.

THE TARGET IS ALWAYS x₀

Even when the anchor is z_mem, the loss points at the real frame. So the model learns to repair blurry / holey / ghosted memory into truth — not to copy it.

UNKNOWN REGIONS UP-WEIGHTED

Loss weighting stops the model from becoming a mere splat cleaner — it must truly synthesize content absent from memory.

8
Phase 4 · inference & rollout

Knownness → per-token start noise → write-back

τ = clip( 1 − λk,  0.45,  1 )
  • Known: x_τ = (1−τ)·z_mem + τ·ε — start from memory + a little noise, short denoise path.
  • Unknown: x_τ ≈ ε — start from ~pure noise, full generation.
  • Each token integrates from its own τ → 0 in a single diffusion pass.
  • Decode latents → frames with the VAE decoder.

WRITE-BACK · CHUNKED ROLLOUT

Back-project new frames into PointMemory as hypothesis points. Long videos run chunk-by-chunk; each chunk grows the memory.

When the camera returns, k is now high → it starts from the stored content and recovers the same scene instead of re-hallucinating.

9

Concretely, on DL3DV

Every space-time token gets memory, a 3D address, and a k

832×480
VIDEO RESOLUTION
81
FRAMES
[16,21,60,104]
VAE LATENT
30×52
TOKEN GRID
32,760
TOKENS = 21×30×52

Each of the 32,760 tokens carries its own memory content, 3D position, and knownness.

10

Takeaway

Memory decides how much to generate

Phase 1

Posed video → world memory content, per-region knownness, per-token 3D address.

Phase 3

Teach the model to repair memory, never blindly copy it. Target is always the real frame.

Phase 4

Knownness → per-token noise. Reuse+repair known regions, generate unknown, write back.

Revisit the same place ⇒ recover it from the same world memory — not a fresh hallucination.

World Memory · working title · CoRL 2026

11