ACRFT — YAM lego-taxi policy-extraction arms
Policy-extraction methods applied to the same pi0.5 base and the same frozen patch critic, as a
method-only-diff comparison ring: identical BC init
(yam_bc_s300_h30_successonly/100000), critic patch_critic_yam_s347_fixed_tau9_min_200k frozen,
and only the extraction objective differs between arms.
Runs suffixed _bb match the BC fine-tune's own training budget: the whole model is
trainable (the BC config sets no freeze filter, so matching its budget means matching what it was
allowed to move, not only steps and batch), batch 32, constant lr 5e-5, 30k steps. These are whole
openpi checkpoints, final step only — load them the way you load any pi0.5 checkpoint.
Runs suffixed _run1 are the earlier pass with the backbone frozen and the action expert
only trainable. They are a smaller budget than BC's and are kept for comparison, not as the
headline result.
| folder | method | step | size | provenance | what the objective swaps |
|---|---|---|---|---|---|
awr_bb/30000 |
AWR | 30000 | 12 GB | xbpeng/awr awr_agent.py:403,407,41 | advantage-weighted flow-BC |
How the weights are delivered follows the suffix, not the arm: a _bb folder is a whole openpi
checkpoint (load it directly); a _run1 folder is an action-expert subtree that is overlaid
on the BC checkpoint at serving time. CFGRL additionally carries an optimality embedding and is
served through its own config.
Each implementation carries file/line-level provenance comments from the official code (or the paper + appendices where no code exists). Two further arms need no weights — QPILOTS-U (test-time critic steering, arXiv 2606.14801) and IDQL/BoN (N-sample argmax of min-ensemble Q, philippe-eecs/IDQL) — they run from the BC checkpoint plus the critic.
Serving
There is ONE serving entry point, scripts/serve_policy.py. Arms reach it two ways, and which
way depends on whether the arm changed the policy's weights or only how a chunk is chosen.
Weight-only arms (awr, flowdpg, qam, dql, fqlx) fine-tune the pi0.5 action expert,
so they are exported to ordinary openpi checkpoints and served like any checkpoint:
uv run python scripts/export_extraction_checkpoint.py --arm dql # -> exported/dql_30000
uv run python scripts/serve_policy.py --port 8000 policy:checkpoint --policy.config pi05_yam_lego_taxi --policy.dir <exported>/dql_30000
CFGRL additionally carries the optimality embedding and samples with classifier-free
guidance, which are model properties, so it travels in its own config (with_cfgrl builds the
variant of any pi0.5 task config; the guidance weight is cfg_w):
uv run python scripts/serve_policy.py --port 8000 policy:checkpoint --policy.config pi05_yam_lego_taxi_cfgrl --policy.dir <exported>/cfgrl_30000
Critic-consuming arms need no policy of their own; they are modes of the critic wrapper:
# selection: bon executes the argmax of N draws (this is also IDQL's argmax rule -- label it by N)
uv run python scripts/serve_policy.py --port 8000 --critic <critic_dir> --critic-mode bon --num-samples 8 policy:checkpoint --policy.config pi05_yam_lego_taxi --policy.dir <BC checkpoint>
# implicit: IDQL's implicit policy -- one draw sampled with expectile weights on the advantage
# --critic-mode implicit --num-samples 64
# qpilots: test-time Q-steering of the sampler, no weights at all
# --critic-mode qpilots --alpha 0.2
# lps / lpsd / flowdagger: pass their small head
# --critic-mode lpsd --extraction-head <latent_actor_*.msgpack>
# --critic-mode flowdagger --extraction-head <flowdagger_run1 dir>
adaptive (execute only the best commitment prefix, then replan) needs a critic trained with
several commitment groups, i.e. macro_group_size < horizon; the fixed-chunk critic this ring
trains against has a single group, where adaptive is bon under another name.
Caveats
- Offline metrics only so far (critic-Q, held-out demo-MSE, chunk jerk); on-robot success rates are pending. Critic-Q is self-refereed for critic-ascending arms — read it with that in mind.
- Batch sizes differ per arm (4-32) because of VLA-scale memory limits, so at equal step counts the arms have consumed different sample counts.
- Code:
600dba6c581009369f1a927f5117e815d4b1c5ee