[Paper Notes] Orca: The World is in Your Mind
Published:
Orca is an early attempt to define a general world foundation model around one central modeling target: learn a unified latent state of the world, then expose that state through downstream readouts for language, images, and robot actions.
My read: the paper’s main move is the shift from isolated next-token, next-frame, or next-action objectives to Next-State-Prediction. Orca treats text, image, and action as interfaces to a shared world latent. The backbone is trained to model state transitions, then frozen; lightweight decoders test whether the latent can be read out into reasoning, future images, and embodied control.
Paper Info
The paper is “Orca: The World is in Your Mind” by the Orca Team, Beijing Academy of Artificial Intelligence. The arXiv page is arXiv:2606.30534, with v1 submitted on June 29, 2026 and v2 revised on June 30, 2026. The project page is orca-wm.github.io.
The core contributors listed in the paper are Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, and Runze Xiao, with Yuheng Ji marked as project lead and Zhongyuan Wang and Pengwei Wang as corresponding authors.
The Big Idea
Many current foundation-model families are organized by output format. Language models predict the next token. Image and video models predict or generate frames. VLA models predict actions. Orca proposes a different center of gravity: the model should learn an internal world state and its transitions, and downstream tasks should read from that latent state.
The paper writes this as latent world-state modeling. Given multimodal world signals (X={X^m}_{m\in M}), the encoder maps them into a latent state:
[ S = f_\theta(X). ]
The state then evolves forward or backward under implicit dynamics and explicit conditions:
[ S_{t+\Delta}\sim p_\Theta(S_{t+\Delta}\mid S_t,z_t,c_t),\quad \Delta\in\mathbb{Z}_{\ne 0}. ]
Here (z_t) represents unobserved dynamics such as physical laws, object properties, scene dynamics, and environmental forces. (c_t) represents explicit conditions such as human instructions or language-described events. Positive (\Delta) predicts future states; negative (\Delta) backtracks to past states.
In this first Orca version, the world signals are vision and language. The authors frame them as two complementary channels: visual signals capture how the world evolves, while language signals describe meaning, intention, causal premises, and event structure.
Two Learning Modes
Orca uses a VLM backbone, based on Qwen3.5, and learns through two complementary paradigms.
Unconscious learning captures dense natural transitions from continuous video. Given a video frame (v_t), Orca predicts the latent of the adjacent next frame (v^l_{t+1}). The target is obtained from a frozen vision encoder, so supervision happens in latent space instead of pixel space. This mode is meant to absorb motion, occlusion, object interaction, and local physical regularities from observation alone.
Conscious learning captures sparse meaningful transitions under language conditions. Videos are segmented into events, and each event has a caption. Given a frame (v_t) and an instruction or event description (e_{t+\Delta}), Orca predicts the latent of a frame sampled from the specified adjacent event. This lets language describe target transitions such as a future event, past event, task intention, or causal premise.
The same conscious branch also includes VQA response generation. Given a video (V) and question (l_q), the LM head generates answer (l_a), preserving a natural-language interface to the latent.
Pre-Training Objective
The pre-training objective combines three terms:
[ \mathcal{L}=\lambda_{\mathrm{obs}}\mathcal{L}{\mathrm{obs}} +\lambda{\mathrm{evt}}\mathcal{L}{\mathrm{evt}} +\lambda{\mathrm{vqa}}\mathcal{L}_{\mathrm{vqa}}. ]
The appendix gives the concrete weights:
[ \mathcal{L}{\mathrm{pre}} =0.1\mathcal{L}{\mathrm{obs}} +0.5\mathcal{L}{\mathrm{evt}} +0.4\mathcal{L}{\mathrm{vqa}}. ]
The latent matching loss is a blend of MSE and cosine distance:
[ \ell_{\mathrm{lat}}(\hat v^l,v^l) =0.1\lVert \hat v^l-v^l\rVert_2^2 +0.9\left(1-\frac{\langle \hat v^l,v^l\rangle}{\lVert \hat v^l\rVert_2\lVert v^l\rVert_2}\right). ]
(\mathcal{L}{\mathrm{obs}}) predicts the adjacent next-frame latent. (\mathcal{L}{\mathrm{evt}}) predicts event-conditioned previous or next event latents. (\mathcal{L}_{\mathrm{vqa}}) is the standard next-token loss for answers. The paper mixes state-transition samples and VQA samples at roughly 5:1.
The query mechanism is simple. Observation-only transition uses <Query 1> to read out a predicted visual latent. Event-conditioned transition adds an instruction and <Query 2> to read out the instruction-conditioned target latent. All learnable queries are trained from scratch.
Data and Training Scale
The paper constructs a world-learning inventory with:
| Data type | Scale | Role |
|---|---|---|
| General video | 125K hours | Dense visual transitions |
| Event annotations | 160M | Language-described state transitions |
| General VQA | 11.5M | Language grounding and world understanding |
The video data covers ego-centric interaction, exo-centric manipulation, action-free robot execution, and natural dynamics. Event data is derived from video through coarse and fine event segmentation plus captions. VQA data teaches the model to describe and interpret observed world states.
The current release uses only about one-tenth of the video inventory. The appendix reports 12.5K hours of video used for both Orca-4B and Orca-0.8B training, over 10,844 steps on 32 nodes / 256 GPUs. The base VLMs are Qwen3.5-4B and Qwen3.5-0.8B. The vision encoder is frozen; the LLM part and visual transition head are trainable.
The infrastructure section is practical. The authors use FlagScale with FSDP2, chunked cross-entropy loss, activation recomputation, and communication prefetching. Throughput rises from 0.66 samples/sec/GPU in the StarVLA pipeline to 2.91 samples/sec/GPU in full Orca training, a reported 4.4x improvement.
Readout Design
After pre-training, Orca freezes the backbone and trains only modality-specific readout modules.
Language readout reuses the LM head. It exposes the latent as answers, explanations, and event-level reasoning.
Vision readout maps Orca’s predicted visual latent into image space through a frozen Stable Diffusion 3.5 MMDiT. The trainable parts are an MLP adaptor and LoRA modules. The target image size is 768 x 768, and vision readout training runs for 200K steps.
Action readout maps the latent to robot action chunks. A DiT-based Action Expert with flow-matching loss receives Orca latent, noisy action with time embedding, and proprioception. Only the MLP adaptor and Action Expert are trained. For real-robot tasks, the Action Expert sees 200 trajectories per task across five tasks.
This readout protocol is important because it isolates the question the paper wants to test: does pre-training produce a latent that downstream decoders can use?
Results
The first scaling result is straightforward: total pre-training loss decreases as video data grows, and the 4B model reaches lower loss than the 0.8B model. More importantly, frozen-backbone readouts improve as pre-training scales. The paper reports gains in text generation, image prediction, and action generation as the world latent strengthens.
Text generation. Orca is evaluated on MVBench, TemporalBench, 3DSRBench, and SWITCH. Orca-4B reaches an average score of 51.8, compared with 46.7 for Qwen3.5-4B in the same table. The paper also groups questions into capability dimensions: state transition, commonsense reasoning, spatial relations, and dynamic motion. Orca-4B improves most clearly on state transition and dynamic motion, which matches the training objective.
Image prediction. The paper introduces PRICE-V0.1, a real-world interaction-conditioned image prediction benchmark. Given an initial image and instruction, the model predicts the target state image after the action. Orca-4B+2B reaches 59.8±10.9 average score across Gemini 3.1 Pro, GPT 5.4, Doubao-Seed-2.0, and Gemma 4-31B judges. The closest baseline in the main table, Flux.2 [klein], reaches 56.1±18.1. The qualitative claim is that Orca better preserves scene consistency, robot morphology, contact relationships, and instruction-conditioned state change.
Action generation. The robot evaluation uses five tasks on a dual-arm wheeled humanoid: Take Book, Stacked Bowls, Pull Out Tissue, Stamp, and Scoop Sugar. The benchmark tests environment OOD and object OOD. Orca is compared with V-JEPA 2.1, Qwen3.5, and (\pi0.5). For overall rule-based score, Orca reaches 32.4, compared with 29.4 for (\pi0.5), 17.0 for V-JEPA 2.1, and 10.5 for Qwen3.5. Orca is especially strong in environment OOD, with 36.6 rule-based score versus 27.6 for (\pi0.5). Object OOD is closer: Orca scores 28.2, while (\pi0.5) scores 31.2.
The diagnostic metrics matter more than the single success rate. Orca tends to advance further before failure and recover better after progress drops. The paper uses PRM-as-a-Judge metrics such as MaxProcess in Failure, Failure Near-Success Score, and Drawdown Recovery Ratio to show that even failed Orca trajectories often remain useful and structured.
Ablation
The ablation studies the three losses:
| Pre-training losses | Text | Image | Action | Average |
|---|---|---|---|---|
| VQA only | 48.4 | - | 10.2 | 29.3 |
| Obs + Event | - | 58.2 | 30.9 | 44.6 |
| Obs + VQA | 50.5 | - | 32.6 | 41.6 |
| Event + VQA | 50.1 | 54.7 | 23.0 | 42.6 |
| Obs + Event + VQA | 51.8 | 59.8 | 32.4 | 48.0 |
The conclusion is nicely interpretable. Observation-only transition is especially helpful for action readout because it teaches dense temporal and physical dynamics. Event-conditioned transition is crucial for image prediction because image readout needs instruction-conditioned target states. VQA keeps the language interface alive and adds semantic grounding. The full combination gives the most balanced latent.
Strengths and Limitations
The strength of Orca is its framing. It offers a concrete recipe for world-state learning that can be probed across three different downstream surfaces while the backbone stays frozen. The paper also makes a useful empirical point for robotics: video-only world pre-training can improve action readout even without action labels during pre-training.
The limitations are equally important. Orca currently uses mainly vision and language, leaving out audio, tactile, force, light, proprioception, and many scientific signals. Visual state prediction is supervised in a frozen ViT/VLM latent space, so the learned world state remains tied to a pre-trained semantic embedding. The model scale is modest, 0.8B and 4B, and the authors use only one-tenth of their video inventory. State-transition supervision is mostly short-horizon and event-local. The action tasks are still relatively short, and success rates remain low in strict real-robot OOD settings.
Takeaway
Orca’s reusable message is: learn a world latent first, then ask different decoders what that latent contains. The paper is strongest as a system-level proposal: train a backbone through dense observation-only transitions, sparse language-conditioned event transitions, and VQA grounding; freeze it; then probe text, image, and action readouts. The current results leave the full definition of a general world foundation model open, while making a clear case that Next-State-Prediction is a productive organizing principle for multimodal and embodied intelligence.