Making a 3D-aware World Action Model

The hand skeleton is drawn as a second video and goes through the same VAE as the RGB, so the frozen Wan2.2-TI2V-5B (plus LoRA) attends over video and action tokens in one sequence.

Measured mocap facts keep the VLM honest, and Claude's review turns its draft into a robot-style instruction: 93 % correct in an independent Codex check, against 57 % for Qwen alone.

The same weights plan (image + text -> video + actions) or act as a world model (image + text + given actions -> video), chosen only by which tokens are noisy.

Depth, camera rays and 2.5D actions join the RGB as extra videos through the same VAE, trained on four datasets, with depth and action denoised 8 steps ahead of the RGB: decent, not perfect.

Both are drawn as 2.5D action videos and encoded by the same VAE. Next: transfer human hand poses to robot grippers.

Held-out clips · image + instruction → video + gripper action + depth · checkpoint 100k

Held-out clips · image + instruction → video + 2.5D hands + depth · checkpoint 100k

MuJoCo Franka FR3 + Robotiq 2F-85 with exo + wrist cameras, like our lab cell. Fine-tune the WAM on ~2,100 scripted picks, then execute its plans on the simulated robot.

With the WAM: 100 held-out scenes

	Exo only	Exo + wrist
Plan alone (open loop)	5 %	6 %
Plan grasp xy error (median)	72 mm	50 mm
Plan + wrist look (oracle mask)At 15 cm above the grasp, MuJoCo's segmentation mask picks the exact target pixels in the wrist RGB-D frame (not available on a real robot); the grasp is re-planned from those 3D points	74 %	93 %
Plan + wrist depth, 2 looks (no mask)No segmentation: keep the above-table depth cluster nearest the WAM grasp point. Look 1 at 15 cm, re-plan, move down, look 2 at 7 cm where the object fills the view, re-plan again (1 look: 79 %)	60 %	88 %

Without a WAM: grasp from head-camera depth with added error (200 trials)

Added depth error	Head camera only	+ wrist re-measure
0 %	98.5 %	100 %
2 %	78 %	100 %
4 %	57 %	100 %
8 %	31 %	100 %

Yes, near contact: a grasp needs ~1 cm, exo plans are 5–7 cm off, and the wrist look closes the gap. Not yet for the plan itself (open loop 5–6 %). Both wrist rows: the WAM plan brings the arm close, then a scripted look re-aims, descends, closes and lifts.

Note: the wrist camera scaling in these clips is wrong (older data); the newer data fixes the wrist depth scale.

The WAM generates 85 frames once per episode; the gripper glyph is decoded into pinch point, yaw and close frame, lifted to a 3D path with depth and camera, and tracked every sim tick by IK on the MuJoCo FR3, with the gripper closing at the decoded frame.

Each tick, damped least-squares IK turns the glyph's target pose into 7 joint targets for position servos with gravity compensation and a 0.1 s lead; IK error is 1.4 mm and the robot tracks the plan to 3.9 mm median, down from 13 mm.

Typical miss: the plan closes ~5 cm away from the object centre (median), so the fingers miss or hit the object.

The WAM plan only drives the approach and picks the object, grasp point and closing axis; a scripted wrist-camera look 15 cm above re-aims the grasp, taking picks from 6/100 to 93/100 (oracle segmentation) or 88/100 (wrist depth only, two looks).

The action stream now also draws the fingertip grasp goal in the wrist view, decodable with only the wrist intrinsics and kinematics; an executor fed noisy oracle goals picks 100/100 at 0–3 mm noise, 97 at 6 mm and 92 at 10 mm; WAM open loop goes from 4 % to 12 % after fine-tuning with wrist actions.

WAM open loop (same setting, 100 scenes): 4 % → 12 % after fine-tuning with wrist actions.

π0.5 re-observes exo, wrist and robot state every 8 steps and reaches 87/100, while the WAM generates the whole episode once from frame 0 and executes it open loop for 12/100, so the next steps are an autoregressive WAM and using the exo–wrist camera pose.

WAM generated video + plan | our execution of that plan (one shot, open loop) | π0.5 execution (closed loop). Our successes first, then our failures.

Closest is SA-WAM: actions, future RGB and depth in one video model with one loss, depth through the VAE as an image; but its actions are numbers (8-D end-effector pose) in latent slots, ours are drawn as video. DreamZero: arXiv 2602.15922. X-WAM: arXiv 2604.26694. SA-WAM: arXiv 2609.02531. Our and π0.5 numbers: same 100 sim scenes; others on their own benchmarks.