act-so101-pickplace-v2 — a documented negative result

ACT (~52M) trained on so101-sim-pickplace-v2 (160 SO-ARM100 pick-and-place episodes, 12.5% of which contain deliberate first-grasp perturbations followed by verified expert recovery). The hypothesis: the nominal baseline's failures were all grasp-miss-then-empty-transport, so adding recovery demonstrations should teach the policy to retry.

It didn't work — and the way it failed is the useful part.

Results (same eval protocol as every model in this series)

Policy	plain	+ temporal ensembling	robustness n/v/p/f (15 eps each)
ACT v1 (100 nominal eps)	65%	75% (+10)	60 / 60 / 53 / 40
ACT-v2 (this model)	50%	35% (−15)	47 / 33 / 40 / 33
ACT-DR (150 domain-randomized eps)	90%	—	87 / 87 / 87 / 100

The ensembling sign flip is the diagnostic. Temporal ensembling averages overlapping action chunks: on v1's behaviorally unimodal data it smooths noise (+10); on recovery data — where near-identical states continue either "proceed to lift" or "re-approach" — it averages two incompatible modes into compromise actions (−15). Several v2 successes run ~40% longer than the stereotyped episode, consistent with the retry skill partially transferring, but the loss in grasp precision dominates.

The data-centric lesson, paired with the companion ACT-DR result: diversity that multiplies visual/physics variation while keeping behavior unimodal (domain randomization) improved everything (+25 nominal, flat-robust); diversity that multiplies behavioral modes (recovery branches) hurt this chunked-BC architecture. Handling multimodal continuations is exactly what diffusion/flow policies and VLA action experts are for.

Confound, noted honestly: at fixed 60k steps, v2 sees 24 epochs of 160 episodes vs v1's 43 epochs of 100; per-episode exposure is lower. The consistent direction across plain, ensembled, and randomized evals makes under-training an unlikely full explanation.

Training

LeRobot 0.6.0 lerobot-train, single Kaggle T4, identical recipe to v1/DR: --policy.type=act --policy.chunk_size=45 --policy.n_action_steps=15 --batch_size=8 --steps=60000

Provenance

Data recorded by a verify-and-retry scripted expert (success-filtered) in the author's MuJoCo work-cell; robot model from MuJoCo Menagerie (trs_so_arm100, Apache-2.0).