Code-as-World: MirroS Converts Real Videos Into Editable Physics Programs

MirroS has released Code-as-World, a research system that takes real video as input and outputs an executable physics scene rather than more pixels or text. The scene is stored as a scene.json file that runs in MuJoCo, the robotics physics simulator. A five-round agentic loop recovers the physical parameters from footage, and the verified scenes become labeled training data. Two open-weight models, a 4B and a 9B, are now available on GitHub under Apache 2.0, with the 9B scoring 55.4 MRA on the QuantiPhy-validation benchmark, just above Gemini-3.1 Flash at 54.8.

What happened

Detail	Value
Models released	Code-as-World-VL-4B and Code-as-World-VL-9B
Base models	Fine-tuned from Qwen3.5-4B and Qwen3.5-9B
License	Apache 2.0
Physics engine	MuJoCo (scene.json format)
Agentic loop rounds	Up to 5 (K = 5)
Training GPUs	8x NVIDIA H100
QuantiPhy-validation 9B score	55.4 MRA
Gemini-3.1 Flash score	54.8 MRA
ChatGPT-5.1 score	48.4 MRA
Qwen3-VL-32B-Instruct (best open baseline)	40.2 MRA
API format	OpenAI-compatible /v1 endpoint via vLLM
Max model length	4,608 tokens
Frames sampled per video	16

MirroS shipped the GitHub repository alongside both checkpoints as BF16 safetensors. The models are served by vLLM behind an OpenAI-compatible /v1 endpoint, which means any team already calling GPT-style APIs can swap in these weights with minimal integration work.

Why does this matter? The case against pure pixel prediction

Standard video generation models predict plausible frames without representing the underlying physics. They have no built-in concept of mass, friction, or gravity. MirroS argues this is a fundamental gap: a model can generate a convincing clip of a ball rolling without knowing the ball’s actual mass or the floor’s friction coefficient.

Code-as-World sidesteps this by representing a scene as an “executable world representation” (EWR), a triple made up of three parts:

• Composition: objects, geometry, metric dimensions, mass, friction, and gravity. Floors and walls are static physical entities, not background pixels.
• Evolution: initial states, forces, contacts, collisions, and termination conditions. Executing this produces a full state trajectory.
• Appearance: camera, lighting, materials, and render settings. Changing appearance never affects the physics.

The result compiles into a scene.json that MuJoCo can execute with either an animation engine (kinematic poses) or a physics engine (forces and contacts). Crucially, the file is human-editable: change the mass, re-simulate, and compare.

How the agentic loop works

Recovering a physical scene from a video is an inverse problem with no unique solution. MirroS frames it as abductive search: an agent cycles through propose, instantiate, execute, render, and verify for up to five rounds.

On the input side, SAM 3 supplies instance masks and image-plane tracks, VGGT-Omega estimates depth and camera geometry, and SAM 3D generates per-object meshes. Each candidate rollout is projected back into the source video’s viewpoint and compared against key frames on RGB, depth, masks, and trajectories. Frame-level discrepancies are bundled into structured feedback that guides the next revision. If no candidate passes verification within five rounds, the hypothesis is rejected.

At a matched five-evaluation compute budget, this loop beats Best-of-5 independent sampling on Visual Alignment, Object IoU, Trajectory-ADE, and Accuracy at 2% of the scene dimension. The gain holds under both execution engines.

Training: two phases, one physical prior

Phase 1 is supervised fine-tuning on 73,335 image-space question-answer pairs built from RefCOCO, RefCOCO+, RefCOCO/g, RefCLEF, and GOT-10K. These cover extent, position, displacement, velocity, and acceleration in raw pixels.

Phase 2 applies GRPO (a reinforcement-learning method for language models) to world-space visual question answering drawn from 1,585 text-driven and 988 video-driven executable worlds. The reward combines scale-normalized numerical accuracy with unit and format terms.

The ablation numbers show how much world-space supervision adds:

Training regime	4B MRA	9B MRA
Image-space only	44.2	50.9
Image + world-space	50.6	55.4

Adding world-space supervision also improves pixel-level grounding: the 9B model jumps from 63.7 to 68.3 on RefCOCO and from 20.1 to 26.6 on GOT-10K.

Our take

The benchmark result is tighter than the framing suggests. A 9B open-weight model beating Gemini-3.1 Flash by 0.6 MRA points on a 159-item validation set is a meaningful result, but it is not a decisive gap. The more interesting number is the ablation: world-space supervision adds roughly six points at the 4B scale, which is a clear signal that the physical labeling pipeline is doing real work.

For business teams, the practical question is whether rigid-body physics scenes are the representation they actually need. This system is strong for robotics, simulation, and training data generation. It is not yet useful for fluid dynamics, deformable materials, or anything that falls outside MuJoCo’s rigid-body scope, which MirroS acknowledges directly.

The Apache 2.0 release and the OpenAI-compatible endpoint lower the barrier for internal prototyping significantly. If you are working on AI integration for robotics or simulation pipelines, standing up the 4B checkpoint via vLLM to test on your own footage is a reasonable afternoon experiment. The 27B reasoning variant scores 58.6 MRA but is not yet released.

We have been watching the broader trend of agentic loops replacing single-pass inference across research and production systems. For context on how similar reasoning-loop approaches are being deployed in adjacent domains, see our coverage of Anthropic’s automated alignment researcher.

What to do about it

1. Check the MirroS GitHub repo and confirm your compute environment supports vLLM with BF16 safetensors before investing time.
2. Start with the 4B checkpoint and 16-frame video clips. The 4B is faster to iterate on and already scores 50.6 MRA.
3. Run the agentic loop with the default K=5 budget. Only increase it if your use case justifies the additional inference cost.
4. Evaluate the scene.json outputs manually on a small set of your own videos before using verified worlds as training data for downstream tasks.
5. Watch for the 27B reasoning checkpoint release, which scores 58.6 MRA and could be the right tier for production-grade simulation work.

If rigid-body physics scenes fit your data pipeline, the Apache 2.0 license and OpenAI-compatible endpoint make Code-as-World one of the more deployment-ready research releases of the month.

Source: Marktechpost