pi05-real-clutter-60-droid-refined-lora

LoRA fine-tune of pi0.5 (pi05_droid warm-start) on IDEAS-Lab-Northwestern/real-clutter-60-droid-refined — 60 real Franka teleop trajectories of a cluttered-table pick-and-place task, DROID schema.

• Task prompt: "pick up the smallest mug and place it in the bowl"
• Warm-start: gs://openpi-assets/checkpoints/pi05_droid/params
• LoRA: gemma_2b_lora (rank 16) backbone + gemma_300m_lora (rank 32) action expert, batch 4, EMA off.
• Norm-stats: reuses pi05_droid's bundled DROID assets (no recompute) — bundled per checkpoint at assets/droid/norm_stats.json.

Checkpoint ladder

Dir	Cumulative steps	Train loss
10000/	10k	0.0056
20000/	20k	0.0036
30000/	30k	0.0032
40000/	40k	0.0038
50000/	50k (saved at step 49999, openpi 0-indexed)	0.0039

Train loss converges by ~20–30k (min 0.0032 at 30k), flat thereafter. 60 traj × 50k steps ≈ ~13 epochs, so eval-sweep the earlier checkpoints — the best real-robot step is likely 10k–30k, before overfit.

Refined real setup

Trained on data from the refined real setup — refined wrist & main-camera poses and gripper — distinct from earlier real datasets (e.g. cab / jar). The wrist camera is raw cam1, exterior third-view is raw cam0 (routed at conversion via --wrist-cam cam1). Collector: yypeng666.

Observation / action schema (DROID)

Stream	→ pi0.5
exterior third-view (raw cam0)	exterior_image_1_left
(zero-pad)	exterior_image_2_left
wrist (raw cam1)	wrist_image_left
joint_position (7) + gripper_position (1)	state
actions (8) = joint_velocity(7) + next gripper target(1)	actions

Inference

Serve with openpi pi05_droid-family config (pi05_droid_finetune_lora). train_state/ is omitted (inference-only). Eval runs on a real Franka / RT-capable client, not the SFT server.

Paper & Citation

Part of ManiGuard: paper (arXiv:2608.17386) · code · docs

@misc{peng2026maniguard,
  title         = {{MANIGUARD}: A Benchmark and Data Suite for Specification-Grounded
                   Safety Evaluation and Improvement of Robotic Manipulation},
  author        = {Peng, Yiyan and Wang, Philip and Zhan, Simon Sinong and Lyu, Yiqi
                   and Ni, Zhenyang and Yan, Jixin and Wong, Fiorelli and Jiao, Ruochen
                   and Yin, Hang and Cao, Xinyu and Shao, Huajie and Li, Manling
                   and Zhang, Ruohan and Zhu, Qi},
  year          = {2026},
  eprint        = {2608.17386},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2608.17386},
}

License

The fine-tuned weights derive from a Physical Intelligence openpi base model whose VLM backbone is PaliGemma; use of these weights is therefore subject to the Gemma Terms of Use (including the Gemma Prohibited Use Policy), which downstream users must pass on. The openpi training code and ManiGuard's own contributions are Apache-2.0.