[Paper Notes] Flora: An Open Model Suite for Scaling Robotics Data Engines
Published:
TL;DR
Flora turns existing human and robot videos into timestamped action instructions and reward annotations. It combines a segmenter, a captioner, and a reward model. The segmenter and captioner receive supervised fine-tuning and reinforcement learning; the released reward model uses supervised fine-tuning.
This identifies the “Flora” associated with Jim Fan: a robotics data engine, with video labeling as its central job. Its approximately 125× cost advantage over maximum-effort Astra is an engineering estimate with explicit deployment assumptions. The strongest reason to read the report is how it separates event boundaries, intended actions, and execution quality.
Report and source version
Flora: An Open Model Suite for Scaling Robotics Data Engines — Sicong Leng, Yunze Man, Zhiqi Li, Jing Wang, Yuhe Zhang, Ryan Punamiya, Siddharth Patki, Kaiyuan Zheng, Zi-ang Cao, Zhiding Yu, Sifei Liu, Danfei Xu, Ben Levin, Jan Kautz, Yuke Zhu, Johan Bjorck, and Linxi “Jim” Fan, NVIDIA. These notes follow the technical report linked by the official project, downloaded October 25, 2026: PDF. The source URL carries the revision marker 2026-10-19c; it is not an arXiv version number.
These notes focus on Section 4 and Appendices D, F, and G. Model weights are linked in the official Flora collection.
Segment first, then describe the intended action
A useful action label needs both a description and the interval over which it applies. Flora produces
\[\mathcal A(X)=\{(s_i,e_i,c_i)\}_{i=1}^{N},\]where $c_i$ describes the action intended during $[s_i,e_i)$. An attempted insertion can receive an insertion instruction even when it fails; execution quality is labeled separately.
The segmenter processes overlapping video windows. Each window has a central region from which predictions are emitted and surrounding context used to interpret boundaries. Predicted intervals carry flags indicating whether an action continues beyond either window edge. A deterministic procedure converts local timestamps to episode time and stitches compatible fragments. Concurrent actions can retain separate intervals. Shifted windows provide another pass for boundary refinement.
For each resulting interval, the captioner sees target frames plus frames from neighboring non-overlapping events:
\[c_i=g_\varphi(X_i^{-},X_i,X_i^{+},q_{\rm cap}).\]Before, target, and after blocks are explicitly identified. Context helps resolve the manipulated object and local intent, while the generated instruction stays tied to the target interval. Synchronized camera views can be tiled into a single image per sampled time. This division lets temporal localization and language quality be evaluated separately.
Three reward outputs, with different meanings
One task-conditioned model supports outcome, dense, and rubric-based rewards. The outcome mode assigns one of five completion levels. A continuous score can be computed from the expected score-token value, rescaled to $\lbrack 0,1\rbrack$.
Dense labeling assigns normal, suboptimal, or failure to half-second steps. These labels are retrospective. Later observations can reveal an earlier mistake, while a later recovery does not erase it. Windows cover 16 seconds and advance by ten; the last four seconds of each non-final window are discarded so retained labels have later evidence. These annotations are designed for offline processing. Using them as a causal live reward would require changing the information available to the model.
Rubric mode receives user-defined task steps and allowed credit values. The maximum credits sum to one, and the model assigns one permitted value per step:
\[R_\rho(X)=\sum_j\hat y_j,\qquad \hat y_j\in V_j,\qquad \sum_j\max V_j=1.\]This allows the evaluation target to specify partial completion without changing model weights. It also makes the rubric part of the measurement protocol: two rubrics need not produce comparable scores.
The training recipe follows the output structure
The segmenter starts from Qwen3.5-9B; captioning and reward models start from Qwen3.5-27B. Data combines public, licensed proprietary, and internally collected sources with human annotations.
All three use token cross-entropy for supervised adaptation. The segmenter trains LoRA parameters with its visual encoder frozen. Captioning updates the language model and visual merger. Reward training also updates the visual encoder at a lower learning rate; outcome, dense, and rubric examples are shuffled together for one epoch.
The segmenter then receives GRPO with a verifiable reward:
\[r_{\rm seg}=0.45F_{0.5}+0.15F_{0.75}+0.25P_{0.5}+0.15B,\]combining temporal F1, precision, and boundary/continuation accuracy. Captioner GRPO uses a reference-grounded point score,
\[r_{\rm cap}=\frac{2M}{R+M+F},\]where $R$ is supported reference mass, $M$ is covered mass, and $F$ penalizes factual errors supported by the evidence. Both retain KL regularization to the fixed supervised policy. Validation selects the two checkpoints independently.
Good segmentation does not guarantee the best caption
On WGO-Bench’s 100 episodes, Flora reaches 50.3% segmentation F1 at temporal IoU 0.75, ahead of Astra’s 46.0%. Its joint semantic F1 is 34.5%, below Astra’s 39.0%; gold-window caption accuracy is 64.7%, versus 77.4%. The advantage is therefore more pronounced in temporal localization.
On the internal DenseCaptionBench, Flora’s joint point F1 is 37.4% on Human100 and 51.0% on Robot200, compared with Astra’s 29.7% and 57.9%. DenseRewardBench macro F1 is 46.2% versus 40.7%, while rubric pooled L1 error favors Astra, 3.6 versus Flora’s 6.5 in the table’s scaled units. The report supports a useful suite of complementary capabilities, with clear remaining errors.
These comparisons evaluate complete systems with different visual input packaging and inference compute. They do not isolate an architecture effect under an identical compute budget.
Read the cost claim through Appendix D
The report compares an estimated US$0.7224 per source-video hour for Flora with US$88.4922 for maximum-effort Astra, a ratio of 122.49 summarized as approximately 125×. Flora assumes eight GB300 GPUs, US$10 per GPU-hour, and a conditional serving rate of 1.23495 episodes per second. The throughput uses an earlier workload and a deployment model; the report explicitly says its production GPU-hour denominator has not been verified.
The API accounting uses October 4, 2026 prices and excludes training, reference construction, and judging. Some requests lack priced usage. Flora’s estimate excludes storage and deployment overheads, and its scenario range is not a confidence interval. The separate claim of labeling 200,000 video hours daily across 512 GB300s is reported production scale, not a single-GPU throughput measurement.
I would first test Flora on a small labeled sample of the intended dataset, separating boundary errors, action-description errors, and failure-label errors. For policy training, the next experiment should measure whether these labels improve the downstream policy. Annotation benchmark accuracy alone does not quantify that benefit.