Contact-rich manipulation / Open environment & dataset

Beyond a single shape.
Push Any Letter.

Lixin XuEstimated reading time: 13 min

What changes when the object is an a, a Q, or a rotated T? Explore how geometry and goal orientation change the way a policy learns to push.

Explore the code ↗Get the data ↗

52Uppercase & lowercase letters

24Goal orientations per letter

100Demonstrations per condition

124,800Episodes in V3

01 / The question

Same task.
A richer distribution.

Push-T offers a focused setting for learning contact-rich manipulation. Push Any Letter expands the object distribution to all 52 uppercase and lowercase ASCII letters, with different contours, holes, symmetries, and contact opportunities.

02 / Dataset versions

Coverage you can
actually inspect.

Letters × orientations × demonstrations

52 × 12 × 10 = 6,240

Released ↗

52 × 12 × 100 = 62,400

Released ↗

52 × 24 × 100 = 124,800

Released ↗

Balanced by letter and target orientation. All goals are at the arena center; initial object and pusher states vary across demonstrations.

Read the dataset card ↗

V1 preview Select a letter and goal orientation to play a released demonstration.

Goal orientation30°

Loading recorded trajectory…0.0 s

03 / Research directions

One environment.
Three ways to study learning.

Imitation learning

Train on planner demonstrations and compare behavior cloning, diffusion policy, flow matching, and an ACT-style model. Inspect success by letter and angle, with both state and image observations.

Reinforcement learning

Use the Gymnasium environment to learn through interaction, or initialize from demonstrations and fine-tune. Study sample efficiency, exploration, and robustness across object geometries.

World models

Compare MLP, GRU, and state JEPA models without velocity or task goals. Inspect prediction errors and recorded rollouts in the world-model report.

04 / Evaluation

Compare learning.
Test the dynamics.

Within a specified training distribution, compare algorithm baselines on task performance, training efficiency, and prediction accuracy.

Imitation learning & reinforcement learning. Compare success rates and training efficiency across algorithm baselines under matched training distributions and evaluation conditions. Measure how performance changes with training data, environment interactions, and compute.
World models. Evaluate how accurately state–action inputs predict the next state and longer sequences of states. Examine whether predictions capture contact events and their effects on object motion, including translation and rotation, and how dynamics errors accumulate over time.

05 / Start exploring

Run it.
Change the letter.

A CPU-based 2D simulator that runs on macOS. Python 3.9+ for simulation; Python 3.10+ for optional training dependencies.

Setup & reproducibility notes ↗

git clone https://github.com/DavidLXu/Push-Any-Letter.git
cd Push-Any-Letter
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m push_any_letter.app --letter Q

Exact physics replay requires the matching Arial Bold font; font files are not redistributed. See the repository for font hashes, dataset downloads, and training commands.

06 / Experimental reports

What the models
have learned so far.

Results, working hypotheses, and the next experiments. Open a report to inspect the evidence.

V1 · 6,240 demonstrations

ACT leads on state; image ACT and Flow differ by one success.

100k updates · batch 256 · seed 42
State + Image · four algorithms each

Final unseen-angle success · 520 cases per model
Algorithm	State	Image
Vanilla BC	0.00%0 / 520	0.96%5 / 520
Diffusion Policy	6.35%33 / 520	36.73%191 / 520
Flow Matching	4.81%25 / 520	43.46%226 / 520
ACT	61.35%319 / 520	43.65%227 / 520

Training progress 0 → 100k

Vanilla BC Diffusion Policy Flow Matching ACT

State · Loss0.9 EMA · log scale

Image · Loss0.9 EMA · log scale

State · Seen52 cases

State · Unseen52 cases

Image · Seen52 cases

Image · Unseen52 cases

Success curves: 52 seen starts / 52 unseen-angle cases. Loss objectives differ across algorithms.

Loading recorded curves…

Success: ≥95% coverage within 2,400 steps. State: two pose frames. Image: two 96 × 96 RGB frames. Both include pusher, goal and letter identity.

State metrics · JSON ↓Image metrics · JSON ↓ Final results · CSV ↓ Raw loss · CSV ↓ Success curves · CSV ↓ V1 dataset ↗

V1 · ACT · State

Success and failure cases

ACT · State · 100k

T, S, a · unseen goal angles · recorded trajectories.

Speed Pusher history Object-center history

ObjectTargetPusher pathObject-center path (optional)

Shared simulation time0.0 / 120.0 s

Recorded episodes load when this report opens.

Successful episodes

95% coverage reached

Failed episodes

2,400-step time limit

Six recorded episodes · JSON ↓

V2 · 62,400 demonstrations

ACT leads both. Image ACT gains 8.27 percentage points over V1.

100k updates · batch 256 · seed 42
State + Image · four algorithms each

Final unseen-angle success · 520 cases per model
Algorithm	State	Image
Vanilla BC	0.00%0 / 520+0.00 pp vs V1	0.38%2 / 520-0.58 pp vs V1
Diffusion Policy	4.04%21 / 520-2.31 pp vs V1	28.46%148 / 520-8.27 pp vs V1
Flow Matching	6.92%36 / 520+2.12 pp vs V1	38.08%198 / 520-5.38 pp vs V1
ACT	58.85%306 / 520-2.50 pp vs V1	51.92%270 / 520+8.27 pp vs V1

10× data, the same 100k-update budget and evaluation cases. Gains vary by model; this is one seed, not an equal-epoch comparison.

Training progress 0 → 100k

Vanilla BC Diffusion Policy Flow Matching ACT

State · Loss0.9 EMA · log scale

Image · Loss0.9 EMA · log scale

State · Seen52 cases

State · Unseen52 cases

Image · Seen52 cases

Image · Unseen52 cases

Success curves: 52 seen starts / 52 unseen-angle cases. Loss objectives differ across algorithms.

Loading recorded curves…

Success: ≥95% coverage within 2,400 steps. State: two pose frames. Image: two 96 × 96 RGB frames. Both include pusher, goal and letter identity.

State metrics · JSON ↓Image metrics · JSON ↓ Final results · CSV ↓ Raw loss · CSV ↓ Success curves · CSV ↓ V2 dataset ↗

V2 · ACT · State

Success and failure cases

ACT · State · 100k

T, A, a · unseen goal angles · recorded trajectories.

Speed Pusher history Object-center history

ObjectTargetPusher pathObject-center path (optional)

Shared simulation time0.0 / 120.0 s

Recorded episodes load when this report opens.

Successful episodes

95% coverage reached

Failed episodes

2,400-step time limit

Six recorded episodes · JSON ↓

Methods: Diffusion Policy · Flow Matching · ACT. V1 technical report ↗

No velocity · No target · V1 data

Does history help
predict motion?

We omit velocity from the input and use four recent states and past actions to supply motion context. We expected this history to help recover movement trends. On V1, the four-frame model instead has larger rollout errors than the single-frame control.

25.50 vs 41.69

Single-frame vs four-frame · position error in pixels

100-step prediction · held-out V1 trajectories

Inputs and prediction. Both models receive letter identity, pusher and object positions, object orientation encoded as sine/cosine, and the current absolute pusher-target action. The history model adds the preceding three states and actions. Both predict the next state; neither receives velocity, a task goal, reward, or images.

Data split. V1 contains 6,240 episodes and 4,736,343 transitions in total. Within each letter–angle condition, slots 00–07 train, 08 validates, and 09 tests: 4,992 / 624 / 624 whole episodes. Test letters and goal angles are familiar; trajectories are held out.

Matched training. Three 256-unit MLP layers, a letter embedding, and 30,000 updates per model, with the same sampling seed. The objective combines one-step prediction with a five-step rollout term every fourth update. One seed was tested. Checkpoints are selected by 50-step validation position and angle error.

Prediction error grows with the horizon.

Lower is better

Mean object position error (px) / wrapped orientation error (degrees), at 20 Hz
Horizon	Single frame	Four frames	Keep initial pose
1 step 0.05 s	0.43 / 0.15°	0.84 / 0.17°	0.33 / 0.14°
10 steps 0.5 s	3.39 / 1.32°	6.93 / 1.52°	3.17 / 1.33°
50 steps 2.5 s	13.69 / 5.39°	26.92 / 8.89°	22.73 / 8.40°
100 steps 5 s	25.50 / 10.66°	41.69 / 18.55°	46.46 / 18.01°

Evaluated at up to two fixed starts per test episode: 1,246 windows at 1/10 steps and 1,245 at 50/100 steps. These are endpoint errors, not averages over the rollout. Selected checkpoints: single-frame 28k, four-frame 2k. The baseline keeps the object's initial pose unchanged.

Many short windows contain little object motion, so the stationary baseline is strong. In the 91 moving one-step windows, position error is 1.56 px for one frame, 2.90 px for four frames, and 3.29 px for the baseline. “Moving” means more than 2 px displacement or 0.03 rad rotation over the evaluated window.

Training progress 0 → 30k

Single frameFour frames

Training objective100-update mean · log scale

1-step validationObject position error · px

10-step validationObject position error · px

50-step validationObject position error · px

100-step validationObject position error · px

Loading recorded curves…

The four-frame model's single-step validation error falls during training, but its long rollouts do not improve with it. This run did not turn history into more reliable dynamics predictions. It does not establish that history is harmful: the representation, objective, and data coverage may all matter. Both models accumulate substantial errors over full episodes.

One trajectory. Two predictions.

Recorded states · Canvas playback

Select one of the six fixed test episodes. All three arenas share the same time and actions. After four initial recorded frames, predictions use only their own states; the single-frame model sees only the latest frame. No later ground-truth correction.

Speed Pusher history Object-center history

Shared simulation time0.0 s

Recorded states load when this report opens.

Recorded truth and predictions · JSON ↓

Test metrics · JSON ↓Single-frame loss log ↓Four-frame loss log ↓Single-frame validation log ↓Four-frame validation log ↓

V3 · State dynamics · No velocity or target

Better local motion.
Drift remains.

History improves one-step prediction, but the four-frame MLP still drifts more than the single-frame model over long rollouts. GRU helps modestly; the state JEPA variant has the lowest object errors at all four tested horizons.

38% lower

100-step position error · JEPA vs single-frame MLP

15.00 vs 24.33 px · 5 seconds ahead

One trajectory. Four predictions.

Recorded states · Canvas playback

Select one of the three test examples. All five arenas share the same time and actions. After four initial recorded frames, predictions use only their own states; the single-frame model sees only the latest frame. No later ground-truth correction.

Speed Pusher history Object-center history

Shared simulation time0.0 s

Recorded states load when this report opens.

Recorded truth and predictions · JSON ↓

Object position error (px) / orientation error (degrees) · lower is better
Horizon	MLP · 1 frame	MLP · 4 frames	GRU · 4 frames	State JEPA · 4 frames	Keep initial pose
1 step0.05 s	0.36 / 0.13°	0.17 / 0.11°	0.16 / 0.08°	0.10 / 0.05°	0.32 / 0.13°
10 steps0.5 s	2.95 / 1.11°	2.66 / 1.28°	2.21 / 0.95°	1.39 / 0.70°	3.28 / 1.33°
50 steps2.5 s	12.69 / 5.16°	13.67 / 6.48°	11.70 / 4.90°	7.77 / 4.12°	22.50 / 8.42°
100 steps5 s	24.33 / 9.73°	26.65 / 12.19°	22.90 / 9.52°	15.00 / 7.95°	44.94 / 17.01°

12,480 held-out trajectories; up to two fixed windows each. Errors are measured at the prediction endpoint. In moving 100-step windows, JEPA reaches 18.62 px versus 30.54 px for single-frame MLP and 62.30 px for persistence.

Training progress 0 → 100k

MLP · 1 frameMLP · 4 framesGRU · 4 framesState JEPA · 4 frames

State prediction loss100-update mean · log scale

1-step validationObject position error · px

10-step validationObject position error · px

50-step validationObject position error · px

100-step validationObject position error · px

Loading recorded curves…

Shared protocol. 124,800 V3 episodes; 99,840 / 12,480 / 12,480 whole-episode train / validation / test split. Letter ID, positions, orientation and actions only. Each model trains for 100k updates with batch size 1,024; checkpoints are selected by 50-step validation error. Curves use a fixed validation subset.

State JEPA. An action-conditioned latent predictor with an EMA target encoder, variance and state-reconstruction losses, and a residual state head. This is a state-based adaptation, not a reproduction of V-JEPA 2. Charts compare the shared state loss, not the different total objectives.

Limits. One seed; JEPA has 484k trainable parameters versus 141–159k for the baselines. Test angles are in the V3 training distribution. Larger data, training budgets and test splits differ from V1, so this is not an isolated data-scaling experiment. Long rollouts still accumulate error.

Test metrics · JSON ↓Training and validation curves · JSON ↓Five-arena comparison · MP4 ↓

Same task.A richer distribution.

Coverage you canactually inspect.

One environment.Three ways to study learning.

Imitation learning

Reinforcement learning

World models

Compare learning.Test the dynamics.

Run it.Change the letter.

What the modelshave learned so far.

Same task.
A richer distribution.

Coverage you can
actually inspect.

One environment.
Three ways to study learning.

Compare learning.
Test the dynamics.

Run it.
Change the letter.

What the models
have learned so far.