[Paper Notes] DexHoldem: Playing Texas Hold’em with Dexterous Embodied System
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
DexHoldem is a real-world benchmark for dexterous embodied systems built around Texas Hold’em tabletop interaction. The platform uses a ShadowHand mounted on a UR10e arm, three RealSense RGB-D cameras for policy-side observation, and a separate agent-view camera for system-level state parsing. The benchmark contains 1,470 teleoperated demonstrations across 14 card and chip manipulation primitives, a physical policy benchmark, a 36-problem agentic perception benchmark, and three released closed-loop system case studies.
My read: the paper is useful because it evaluates the full stack at the point where many embodied-agent demos become vague. A robot needs to decide that it should call, raise, reveal a card, or collect chips, then parse a changing tabletop state, translate the decision into executable low-level primitives, manipulate thin cards and small chips with a dexterous hand, preserve the table for the next decision, and recover when perception or execution is slightly wrong. DexHoldem makes those failure surfaces measurable.
Paper and Resources
The paper is “DexHoldem: Playing Texas Hold’em with Dexterous Embodied System” by Feng Chen, Tianzhe Chu, Li Sun, Pei Zhou, Zhuxiu Xu, Shenghua Gao, Yuexiang Zhai, Yanchao Yang, and Yi Ma. It is available as arXiv:2605.18727, with the project page at dexholdem.github.io/Dexholdem. The dataset is released as TexasPokerRobot, the policy code is in DexHoldem/Dexholdem-Policy, and the embodied-agent runtime is in DexHoldem/DexHoldemSKills.
The policy repository is organized around six public training recipes: DP(DINO), DP_transformer_resnet, DP_unet, ACT, RDT_small, and RDT_FT. The released pipeline downloads the Hugging Face dataset, organizes demonstrations with five validation trajectories per primitive, optionally precomputes DinoV2 or SigLIP visual features, and serves trained checkpoints through a ZeroMQ policy server. The skills repository exposes the agent loop as a coding-agent-native workflow: install the skill, launch a game loop, parse the table, route the state, and dispatch robot primitives.
Why Poker Is a Good Benchmark Shape
Texas Hold’em is not used here as a test of poker strategy. It is a controlled tabletop domain where semantic state and physical state are tightly coupled. Cards are thin, chips are small and denomination-specific, bets change the legal action space, and every physical movement can disturb later decisions. The robot may need to pick up a left or right hole card, show a card, place it back face down, push a 50-chip, pull a 100-chip, or compose several such actions into a higher-level poker move.
This makes the benchmark more demanding than isolated hand skills. A local manipulation success can still be a system failure if a non-target card shifts, a chip stack becomes unreadable, or the next agent step cannot trust the table state. DexHoldem therefore scores primitive rollouts with four outcome levels: scene-preserving success, disruptive completion, retryable task failure, and disruptive failure requiring reset. The distinction between task completion and scene preservation is the main metric design choice.
Benchmark Structure
DexHoldem separates the system into two levels. The primitive level defines callable dexterous skills for data collection, policy training, and physical rollouts. The agent level defines the perception, routing, verification, recovery, and human-help branches that arise when those primitives are composed into a poker hand.
The policy benchmark covers 14 instruction-conditioned primitives:
| Group | Primitives |
|---|---|
| Card pickup | pick_up_left, pick_up_right |
| Chip push | push_5, push_10, push_50, push_100 |
| Chip pull | pull_5, pull_10, pull_50, pull_100 |
| Card placement/reveal | put_down_left, put_down_right, show_left, show_right |
Each primitive has 105 accepted teleoperated demonstrations, split into 100 training and 5 validation trajectories. During deployment, each low-level policy receives top-down, third-person, and wrist RGB-D observations, 30-dimensional arm-hand proprioception, and a task condition. The action is a short-horizon sequence of 30-dimensional joint-position targets: 6 dimensions for the UR10e arm and 24 for the ShadowHand.
The agentic perception benchmark isolates the visual parsing problem. Each of 36 problems asks the perceiver to recover a structured game state from a real tabletop image plus predecessor context. The schema is split into eight scored challenges: loop stage, turn ownership, blind information, community cards, current bet chips, robot chip inventory, opponent chip inventory, and showdown outcome. The important detail is that overall correctness is a strict exact match over all applicable fields, so a single wrong chip dictionary can make a state unusable for routing.
The Closed-Loop Agent
The embodied loop is capture -> perceive -> route -> execute. A dedicated agent-view camera captures the table. The perceiver writes structured state. A deterministic router loads persistent game-state memory, validates fields, handles waiting, verifies primitive outcomes, retries harmless failures, and escalates unsafe states to human help. The main agent is queried only when multiple high-level branches are legal, such as a new poker decision at an idle state.
High-level agent primitives are translated into low-level policy primitives. For example, view_card(L) becomes pick_up_left -> perceive -> put_down_left, while call dispatches chip-push primitives according to the difference between the opponent’s bet and the robot’s bet. Chip actions use a min-count rule over denominations in the order 100, 50, 10, 5, so a failed atom can be retried without discarding the whole high-level action.
This architecture is deliberately mundane in the right places. The router carries workflow constraints that should not be left to the language model at every frame, while the learned policy handles contact-rich motion. The result is a benchmark where an agent’s perception errors, policy errors, verification delays, and recovery decisions all show up in the same physical trace.
Main Results
On the 80-trial primitive schedule covering all 14 skills, π0.5 obtains the best task completion rate at 61.2%. On the stricter scene-preserving success rate, π0.5 and π0 tie at 47.5%. RDT reaches 30.0% SPSR and 46.2% TCR; DP(DINO) is the strongest task-specific imitation baseline at 26.2% SPSR and 36.2% TCR. Smaller or simpler baselines remain much lower: ACT reaches 10.0% SPSR, BAKU 6.2%, and DP-UNet 1.2%.
The gap between SPSR and TCR is the key signal. For π0.5, the score rises from 47.5% to 61.2% when disruptive completions are counted. For RDT, it rises from 30.0% to 46.2%. These are not small bookkeeping differences; they show that a policy can complete a local primitive while damaging the table state needed by the next agent step.
The primitive-group breakdown is also revealing. π-series models reach 100.0/100.0 SPSR/TCR on pickup, but chip motion remains much harder: π0.5 reaches 25.0/35.0 on chip push and 15.0/30.0 on chip pull. Put-down/show has a larger completion-preservation gap, reaching 50.0/80.0 for both π0.5 and π0. Cards can be placed or revealed while still disturbing the scene enough to hurt continuation.
For agentic perception, Opus 4.7 gives the best strict problem-level exact match at 34.3%, while GPT 5.5 gives the best average field-wise accuracy at 66.8%. This split is important. Individual fields can be recognizable, yet full routing-relevant state recovery remains fragile because many fields must be right together. Blind information is nearly saturated, and turn ownership is high for several models, while current bet chips and opponent chip inventory are weak: the best CB score is 45.8%, and the best OCI score is 43.8%.
The three system-level case studies pair GPT 5.5 with the π0 dexterous policy. They are not presented as a statistically powered success-rate estimate. Their value is operational: the traces show repeated waiting, verification, continuation of multi-atom actions, recovery dispatches, and occasional human-help requests. One 23-state trajectory views both hole cards, raises, checks, calls a 200-chip bet, and reveals both cards, with about a third of states spent in the wait branch.
RDT Fine-Tuning Diagnostic
The RDT data-scaling study is a useful caution against overreading pretraining. The authors compare random initialization with a gripper-pretrained RDT checkpoint under 10%, 20%, 50%, and 100% of the DexHoldem training split. At 10% data, pretrained initialization reduces validation loss by only 1.2% relative to random initialization. The reductions grow to 9.0%, 10.7%, and 11.3% at 20%, 50%, and 100%, but both curves follow similar scaling trends.
The interpretation is conservative: gripper-centric pretraining gives an optimization or initialization benefit once enough dexterous-hand data exists, without creating a qualitatively new low-data regime for this ShadowHand poker setting. That is a helpful result because it keeps the benchmark grounded in physical embodiment mismatch instead of assuming that large robot pretraining automatically transfers to dexterous card and chip manipulation.
Limitations
DexHoldem is tightly scoped. The hardware, table layout, camera arrangement, cards, and chip denominations are fixed. The benchmark measures performance under a standardized real-world interface; cross-embodiment transfer and arbitrary-object dexterity are outside its claim. The dataset is also modest compared with modern pretrained robot-policy corpora. 1,470 demonstrations are enough to define and compare the task suite, while large-scale policy scaling remains open.
The system-level evaluation is intentionally small because current components are still unreliable. The paper reports three closed-loop case studies instead of a large success-rate table. That is a limitation, but it is also a useful honesty boundary: once perception, routing, policy execution, verification, and recovery are composed on physical hardware, evaluation becomes expensive and errors compound quickly.
Takeaways
DexHoldem’s strongest idea is the scene-preserving embodied benchmark. It evaluates whether a policy leaves the world in a usable state, and whether an agent can keep enough structured memory to route the next action. For dexterous manipulation, that is closer to deployment than asking whether a hand can complete one primitive in isolation.
For embodied-agent research, the perception result is a warning. Field-wise scores can look passable while strict full-state recovery remains weak. Poker makes this visible because chip counts, bets, cards, turn markers, and loop stages must all agree before an action is legal and executable.
For robot-learning practice, the released policy code and dataset matter. The benchmark provides a concrete recipe for comparing pretrained VLA policies, diffusion policies, ACT-style policies, and RDT variants under a shared 30-dimensional ShadowHand-UR interface. The open question is how to reduce the cost of real-contact evaluation without losing the very property that makes DexHoldem useful: small physical disturbances matter.