[Paper Notes] RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
RynnValue is an open-source robotic value foundation model built around a simple target: given a language instruction and robot observations, predict the remaining time in seconds until task completion. This temporal distance acts as a directed, goal-conditioned cost-to-go. Its labels come from timestamps and relabeled completion cutoffs, allowing training on more than 7,000 hours, 1.67M original episodes, and 3.09M instruction-conditioned segments without trajectory-preference labels or per-task normalized-progress annotations.
The model uses absolute and relative temporal heads, irregular frame sampling, temporal-order shuffling, and value-isolation attention. These choices prevent easy shortcuts such as inferring progress from frame index, regular sampling intervals, or neighboring value queries. On the six-dataset RBM-EVAL-OOD trajectory-ranking benchmark, RynnValue-8B reaches 0.675 average Kendall’s (\tau_a), above preference-supervised Robometer at 0.655 and far above its progress-only variant at 0.292.
RynnValue does not emit an action reward directly. It predicts an observation potential (\Phi_t=-v_t), where (v_t) is remaining time. Potential differences provide dense reward shaping while a manually annotated sparse terminal signal preserves the final task objective. In four unseen real-robot tasks, the resulting reward interface achieves 72.5% average online-RL success versus 52.5% for Robometer, and 82.5% offline-RL success versus 63.8%.
Paper Info
“RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance” is by Dongchi Huang, Hongyin Zhang, Bohan Hou, Siteng Huang, Zhian Su, Hang Guo, Tong Lu, Zhaofeng Xu, Jiahao Tang, Jianfei Yang, Donglin Wang, Peixi Peng, Mingxiu Chen, Deli Zhao, and Xin Li from DAMO Academy, Alibaba Group, and Hupan Lab. This note covers arXiv:2608.09853v1, dated August 11, 2026. The authors release a project page, code, and 4B/8B checkpoints.
1. Is RynnValue a Reward Model?
RynnValue can serve as a reward model, though its learned object is more precisely a goal-conditioned state-value potential. Its input is a task instruction, embodiment metadata, and a short sequence of visual observations. Its main output is the time remaining until the instruction is completed. The model does not score actions, predict transition rewards, or estimate an action-conditioned (Q(s,a)).
The complete pipeline is:
instruction + observation history
↓
predicted remaining time v_t
↓
potential Φ_t = -v_t
↓
dense shaping κ(γΦ_{t+1} - Φ_t) + sparse task reward
↓
offline or online robot RL
This distinction matters. RynnValue learns a reusable notion of “how far this visual state is from the language goal.” A downstream RL algorithm turns changes in that potential into rewards.
2. Temporal Distance as the Scaling Target
General-purpose robotic reward models commonly use one of three anchors: human or model preferences between trajectories, comparison with reference demonstrations, or normalized task progress in ([0,1]). Each anchor creates friction at scale. Preferences require comparison labels; references require an appropriate successful trajectory; normalized progress has no shared physical meaning across a ten-second pick and a one-minute bimanual task.
RynnValue labels an observation at timestamp (t_i) using the relabeled goal-completion cutoff (t_G):
[ v_i^*=\max(0,t_G-t_i). ]
Observations at or after completion receive zero. A second target captures signed local displacement between consecutively presented frames:
[ \Delta_i^*=t_{i+1}-t_i. ]
Positive (\Delta_i^*) indicates forward movement through the recorded trajectory; negative values arise when training frames are deliberately presented in reverse temporal order. Absolute distance anchors each state to completion, while relative distance teaches local temporal direction.
Temporal distance preserves seconds as a common unit. It also avoids a separate progress-normalization recipe for every dataset. This choice gives the model a cost-to-go interpretation under an approximately minimum-time objective.
3. Architecture: One Backbone, Three Outputs
RynnValue builds on the embodied multimodal model RynnBrain. The main configuration samples (K=8) observations. Each observation receives an absolute-value query group with (N=8) repeated tokens; observations after the first also receive an eight-token relative-value group. Tokens within a group exchange information, and their hidden states are concatenated before prediction. The grouped representation can preserve cues about object configuration, robot–object interaction, task stage, and completion evidence.
Two distributional heads produce numerical outputs:
- The absolute head predicts remaining time on ([0,512]) seconds.
- The relative head predicts signed displacement on ([-256,256]) seconds.
Each range is divided into 256 symlog-spaced bins. Continuous targets are represented with two-hot labels over adjacent bins. At inference, the expected symlog-space bin center is transformed back to seconds. This classification formulation compresses large values, preserves precision near zero, and prevents large temporal targets from dominating gradient scale.
The original language-model head generates a structured analysis after the temporal queries:
Video Description: ...
Match: Yes / No
Success: Yes / No
These language outputs help the shared representation understand events, instruction alignment, and completion. They are not fed back into the temporal predictions.
4. Scaling Heterogeneous Robot Data
The training mixture spans real robots, simulation, egocentric human videos, single- and dual-arm systems, dexterous hands, mobile manipulation, and many camera layouts.
| Source | Original episodes | Instruction-conditioned segments |
|---|---|---|
| AgiBot | 167,535 | 1,166,042 |
| EgoDex | 338,234 | 338,234 |
| Galaxea Open-World | 16,979 | 95,671 |
| InternData-A1 | 320,905 | 320,905 |
| Open X-Embodiment | 693,037 | 693,037 |
| RDT | 6,109 | 6,109 |
| RoboCOIN | 67,420 | 410,877 |
| RoboMIND | 32,138 | 32,138 |
| RoboTwin | 27,414 | 27,414 |
| Soft-FOLD | 1,542 | 1,542 |
| Total | 1,671,313 | 3,091,969 |
The resulting corpus contains 223,395 unique instructions and more than 7,000 hours. Long episodes are split using native subtask annotations when available; otherwise, the full episode remains a coarse segment. Each segment then receives a completion cutoff. The endpoint is the default, with dataset-specific ratio or duration trimming used where post-completion motion would corrupt the target.
Source-aware curation removes placeholders, data-quality metadata, malformed commands, and pure locomotion segments without a manipulation goal. In the four corpora analyzed in the appendix, the pipeline retains 83.35% of trajectory units and 98.99% of unique valid instructions, suggesting that removal is concentrated in repeated annotation noise.
5. Preventing Temporal Shortcuts
Timestamp labels are cheap, but a multi-frame model can solve the training task without understanding manipulation. Uniformly spaced chronological frames expose several shortcuts: later sequence positions usually have lower remaining time, neighboring values often follow an arithmetic pattern, and successful training videos usually progress monotonically.
RynnValue uses four mechanisms to break these correlations:
- Random temporal sampling: eight observations are drawn at irregular timestamps, removing fixed value increments.
- Temporal-order shuffling: half of the sequences are unsorted; the remainder use a forward-biased walk with 0.3 rewind probability. Frame position no longer implies progress.
- Value-isolation attention: value-query groups from different observations cannot attend to one another, and context tokens cannot absorb earlier value queries. Each estimate must use its own visual-language context.
- Instruction-mismatch augmentation: 10% of samples receive an instruction from another trajectory. The language branch predicts
Match: NoandSuccess: No; the invalid absolute target is masked while the instruction-independent relative target remains active.
The final loss is
[ \mathcal L=\mathcal L_{\mathrm{abs}}+ \mathcal L_{\mathrm{rel}}+2\mathcal L_{\mathrm{lang}}. ]
The absolute and relative heads plus the shared backbone receive gradients. The LM output projection stays frozen, while the language loss still updates backbone representations through it.
6. From Value to Dense Reward
At inference, observations return to chronological order and RynnValue predicts non-negative remaining time (v_t). Sign reversal produces a higher-is-better potential:
[ \Phi_t=-v_t. ]
Potential approaches zero as the task reaches completion. The RL experiments use
[ r_t’=r_t^{\mathrm{sparse}}+ \kappa\left(\gamma\Phi_{t+1}-\Phi_t\right), ]
with
[ r_t^{\mathrm{sparse}}= \begin{cases} 0,&\text{if the action completes the task successfully},
-1,&\text{otherwise}. \end{cases} ]
The shaping term rewards reductions in predicted remaining time and penalizes regressions. Potential-based shaping has a policy-invariance motivation, while the sparse completion term guards the final objective against reward-model noise.
The terminal signal is still manually annotated. Operators also determine task success and episode termination; reward models supply only the shaping potential. RynnValue therefore improves sparse reward density without eliminating task-level success supervision in the reported policy-learning experiments.
For comparison, the shaping coefficient is fixed across tasks at (\kappa=0.1) for RynnValue and (\kappa=1.0) for Robometer. Each trajectory is scored from the task instruction and one third-person RGB stream. Histories are uniformly subsampled to four frames during RL reward inference to match Robometer’s protocol.
7. Reward-Model Benchmark Results
RBM-EVAL-OOD contains 976 failed, suboptimal, and successful trajectories across six held-out robot datasets. The benchmark measures Kendall’s (\tau_a) between human trajectory-quality ordering and model ordering. RynnValue scores a trajectory using the negative predicted remaining time at its final queried observation.
| Model | Preference supervision | Average Kendall’s (\tau_a) |
|---|---|---|
| Robometer, progress only | No | 0.292 |
| RoboReward-4B | No preference pairs | 0.502 |
| Robometer, RBM-1M | Yes | 0.655 |
| RynnValue-4B | No | 0.670 |
| RynnValue-8B | No | 0.675 |
The 4B and 8B variants are nearly tied, suggesting that the supervision recipe and anti-shortcut design contribute more than parameter count in this range. In an instruction–trajectory confusion test, RynnValue obtains a 0.79 normalized diagonal margin, compared with 0.67 for the strongest baseline, showing stronger language-goal grounding.
The ablation table makes the shortcut problem especially clear:
| Variant | Average Kendall’s (\tau_a) |
|---|---|
| Without temporal-order shuffling | 0.189 |
| Uniform temporal sampling | 0.379 |
| Without value-isolation attention | 0.482 |
| Without language supervision | 0.537 |
| Without relative temporal prediction | 0.627 |
| Full RynnValue-8B | 0.675 |
Removing shuffling causes the largest collapse. A model that always sees ordered sequences can learn “later frame means better” and still fit its labels. Random gaps and isolated queries force more visual reasoning.
8. Data Diversity Matters More Than Repetition
The paper separately scales episode count and the number of training tasks while keeping total episode counts comparable at each fraction. Additional episodes from the same tasks saturate quickly. Increasing task diversity reduces held-out temporal-distance error across the full scaling range.
This result changes the interpretation of “more robot data.” For value learning, repeated successful executions of a narrow skill family provide limited new information after basic visual dynamics are learned. New goal structures, failure modes, objects, embodiments, and viewpoints expose the model to qualitatively different cost-to-go relationships.
9. Real-World Policy Learning
The reward model is evaluated zero-shot on four tasks using a dual-arm Franka system; their tasks, objects, and scenes are absent from RynnValue training. Each policy is tested over 20 trials.
Online RL
| Reward | Bread basket | Steak + spatula | Box + drawer | Bimanual transfer | Average |
|---|---|---|---|---|---|
| Sparse | 40% | 45% | 40% | 70% | 48.8% |
| Robometer | 35% | 45% | 65% | 65% | 52.5% |
| RynnValue | 45% | 75% | 70% | 100% | 72.5% |
Online training uses DSRL to optimize latent noise for a frozen (\pi_{0.5}) action decoder. Bread Basket and Steak Serving start from SFT policies; Box-in-Drawer and Bimanual Transfer start from the same Robometer-based offline-RL checkpoints for every online reward variant. Each task uses 60 online trajectories and 6,000 training steps.
Offline RL
| Reward / policy | Bread basket | Steak + spatula | Box + drawer | Bimanual transfer | Average |
|---|---|---|---|---|---|
| SFT | 70% | 25% | 0% | 0% | 23.8% |
| Sparse IQL | 70% | 20% | 0% | 0% | 22.5% |
| Robometer IQL | 80% | 80% | 50% | 45% | 63.8% |
| RynnValue IQL | 100% | 90% | 90% | 50% | 82.5% |
Offline IQL relabels the same 410-trajectory mixed-expertise dataset for every reward variant. The dataset is heavily success-dominated: 398 successes and 12 failures. RynnValue still provides enough intra-trajectory structure to outperform sparse IQL and Robometer, especially on Box-in-Drawer, where SFT and sparse IQL never succeed.
The smallest online gap appears on Box-in-Drawer: RynnValue reaches 70% and Robometer 65%. Precise grasp stability and box–drawer alignment can look similar in third-person RGB even when physical outcomes differ. The paper uses this case to expose a sensing limitation of visual reward models.
10. Strengths and Limitations
RynnValue’s main strength is a supervision target that can be generated across heterogeneous corpora with minimal annotation. Temporal seconds form a shared interface across datasets, and the ablations convincingly show that this apparent simplicity needs aggressive shortcut suppression. The work also closes the loop from intrinsic value evaluation to offline and online policy improvement, with released code, models, benchmark adapters, and RL infrastructure.
The central assumption deserves care. Logged remaining time is produced by the behavior policy, operator speed, control frequency, hesitation, and the chosen completion cutoff. It equals an optimal minimum-time cost-to-go only under strong conditions. Two visually identical states can receive different labels when one trajectory pauses or follows a slower controller. Task diversity may help average over these effects, but the paper does not explicitly identify an embodiment-invariant optimal hitting time.
Completion cutoffs are also a source of supervision noise. Native subtask annotations are used when available; coarse episodes may retain their endpoint, and some datasets receive ratio- or duration-based trimming. This recipe is cheaper than preference labeling, yet it still depends on source-specific curation and cutoff assumptions.
The real-world reward pipeline retains manually annotated success and termination. RynnValue supplies shaping, so the experiments do not demonstrate a fully autonomous replacement for terminal rewards. Reward models use only a third-person RGB history and language; they lack proprioception, force, tactile input, and explicit geometry. The Box-in-Drawer result shows the consequence for precision-sensitive contact states.
Evaluation uses 20 rollouts per policy, four real tasks, and one Franka platform. RynnValue and Robometer use different fixed shaping coefficients, which is practical tuning but complicates a pure comparison of raw potential quality. The short observation window and four-frame RL inference also limit long-horizon memory. The authors identify streaming inference, longer horizons, dexterous hands, mobile manipulation, and richer energy/safety/precision costs as future directions.
Takeaway
RynnValue treats reward modeling as learning a reusable temporal potential. Timestamps provide cheap supervision; absolute and relative heads learn global and local temporal structure; shuffled sampling and isolated queries prevent positional shortcuts; potential differences convert remaining time into dense reward for robot RL.
The paper’s most useful contribution is the interface boundary. The foundation model answers, “How many seconds of goal-directed work appear to remain?” The RL system decides how changes in that estimate should shape behavior, while a sparse terminal label secures the final objective. The strong OOD ranking and real-policy gains make temporal distance a compelling scaling target, while the gap between logged time and true optimal cost-to-go remains the key conceptual question.