corn-inspection-pi0-checkpoints

Fine-tuned π0 / π0-FAST flow-matching policies for an eye-in-hand visual-servoing task: close-range inspection of corn cobs for insect damage from a mobile manipulator. Trained with openpi on an anonymized teleoperation dataset (LeRobot format).

This repo publishes the π0 sweep of the thesis experiments — 29 arms, {h5,h10,h20,h50} × {state-real,state-zero} × seed{42,43,44} (24) + one camera-slot ablation + four π0-FAST arms. It holds, per run, the final checkpoint and the step-2500 campaign rung: the two checkpoints the reported numbers come from. See What's in this repo and Storage policy.

What's in this repo

Prefix	Per run	Purpose
runs/<arm>_<jid>/	step 7999 params/ + assets/ (no optimizer state) + norm_stats* + manifests + SLURM log	the checkpoint consumed downstream (Thor inference, closed-loop eval, open-loop eval)
campaign-rungs/<arm>/	step 2500 params/ + assets/ + norm_stats* + manifests — the 24 flow-π0 arms only (no π0-FAST, no camslot)	the rung the closed-loop campaign servers load; pre-registered as the headline comparison point

The intermediate rungs (0500…7500) are not published here. They exist in full on the project cluster and feed the post-hoc checkpoint-ladder analysis; the per-step training curves they would illustrate are already public as text in the project's run-ledger/ (full SLURM logs).

<jid> is the SLURM job id — see the decode table below.

Canonical dataset

Every run here was trained on the canonical dataset build:

	fingerprint (sha256)
LeRobot (π0 input)	3532405f02eb4a8866f585102a8d01a4fc12e85c4f75db2076d789e66716233e
RLDS twin	30d7cefab9da9107f3068ebe5e0bec9029a345deb71df0045501056cc2e075ac

Built by cluster/build_canonical_dataset.sh (SLURM job 149026) from 60 raw teleoperation episodes, pinned toolchain (lerobot 0.1.0, datasets 3.6.0). The fingerprint is recorded in every run's manifest_pre_run.json and RLDS integrity is gated by sha256sum -c at job start. Dataset repo: lucaosti/corn-inspection-vla-dataset (private + gated), with the RLDS build under its rlds/ prefix.

Training

Framework	openpi (physical-intelligence/openpi, JAX), pinned per run
Method	LoRA fine-tune of π0 / π0-FAST, 8000 steps
Ladder	checkpoint every 500 steps → 16 rungs (0500…7999)
Sweep	{h5,h10,h20,h50} × {state-real,state-zero} × seed{42,43,44} (24) + pi0_h10_state-real_seed42_camslot-base (1) + pi0fast_{h5,h10,h20,h50}_seed42 (4) = 29
Hardware	SLU Libra, 1× NVIDIA H100 NVL per arm (machinelearning partition, array 149039)
Final loss	π0 state arms ≈ 0.053–0.084; π0-FAST ≈ 0.19–0.83 (FAST-tokenizer cross-entropy, different scale)

Per-run reproducibility metadata is in each run directory: manifest_pre_run.json / manifest_post_run.json (git SHAs, dataset fingerprint, hardware), manifest_*_trainconfig.txt (the resolved TrainConfig dataclass — train_config.captured: true), manifest_pre_run_pip_freeze.txt, norm_stats.json, slurm_<jid>.log.

canonical run → SLURM job id

arm	jid	arm	jid
pi0_h5_state-real_seed42	149053	pi0_h20_state-real_seed42	149133
pi0_h5_state-real_seed43	149067	pi0_h20_state-real_seed43	149139
pi0_h5_state-real_seed44	149073	pi0_h20_state-real_seed44	149144
pi0_h5_state-zero_seed42	149079	pi0_h20_state-zero_seed42	149151
pi0_h5_state-zero_seed43	149086	pi0_h20_state-zero_seed43	149154
pi0_h5_state-zero_seed44	149095	pi0_h20_state-zero_seed44	149162
pi0_h10_state-real_seed42	149099	pi0_h50_state-real_seed42	149166
pi0_h10_state-real_seed43	149105	pi0_h50_state-real_seed43	149188
pi0_h10_state-real_seed44	149111	pi0_h50_state-real_seed44	149195
pi0_h10_state-zero_seed42	149116	pi0_h50_state-zero_seed42	149199
pi0_h10_state-zero_seed43	149121	pi0_h50_state-zero_seed43	149205
pi0_h10_state-zero_seed44	149128	pi0_h50_state-zero_seed44	149213
pi0_h10_state-real_seed42_camslot-base	149039	pi0fast_h5_seed42	149216
pi0fast_h10_seed42	149223	pi0fast_h20_seed42	149226
pi0fast_h50_seed42	149232

Layout

runs/<arm>_<jid>/
  7999/
    params/                     openpi checkpoint params (orbax/ocdbt)
    assets/                     norm-stats assets
    _CHECKPOINT_METADATA
  norm_stats.json               action/state normalization stats
  norm_stats_stop_override.json
  manifest_pre_run.json         git SHAs, dataset fingerprint, hardware
  manifest_post_run.json
  manifest_{pre,post}_run_trainconfig.txt
  manifest_pre_run_pip_freeze.txt
  slurm_<jid>.log
  wandb_id.txt

campaign-rungs/<arm>/            24 flow-π0 arms only
  2500/{params,assets,_CHECKPOINT_METADATA}
  norm_stats.json  norm_stats_stop_override.json
  manifest_{pre,post}_run{,_trainconfig}.txt

Both prefixes omit the optimizer train_state/ — inference reads only params/ and assets/. The intermediate rungs (0500…7500) are not published here; see Storage policy.

Usage

Load one arm's final checkpoint with openpi, using the arm's row in pi0-oscillation/cluster/sweep_manifest.csv to reconstruct the exact TrainConfig it was trained under:

python pi0-oscillation/serving/pi0_server.py \
    --checkpoint <local>/runs/pi0_h10_state-real_seed42_149099/7999 \
    --arm-name pi0_h10_state-real_seed42

For the closed-loop campaign, point the server at the pre-registered rung instead: campaign-rungs/pi0_h10_state-real_seed42/2500.

--arm-name must match the checkpoint's name column in sweep_manifest.csv; a mismatch silently builds the wrong architecture / normalization for that checkpoint.

Storage policy

See docs/superpowers/plans/2026-09-06-artifact-storage-policy.md. Rule 0: the project cluster keeps every artifact at full fidelity and is never pruned for space. Rule 1 (2026-09-10): this repo is the publishable showcase, not a backup — it carries the final checkpoint and the pre-registered step-2500 rung, and nothing else. The complete 16-rung training ladders and the pre-canonical runs are kept on the cluster only. GitHub run-ledger/ carries the text record (manifests, SLURM logs, norm-stats, decode table); no weights.

Provenance and supersession

• The provisional pre-canonical checkpoint tree (full ladders from the first L40S / H100 sweep, SLURM jobs 147551 / 147688 / 148116 / …) was superseded by the canonical re-training (array 149039): their training-dataset checksums were never committed and the resolved TrainConfig was never dumped. It was held as the tag provisional-full-ladders-2026-09-04 until 2026-09-10, then removed from this repo and kept on the project cluster only.
• The canonical runs published here close that gap: dirty-tree gate at submit, resolved TrainConfig captured (train_config.captured: true), canonical dataset fingerprint in every manifest.
• Published content carries no personal name of the demonstrator.

Companion OpenVLA sweep: lucaosti/corn-inspection-openvla-checkpoints.