engage3D placement correction v1
A small network (MLP ensemble of 3, about 140k parameters each) that corrects where two interacting people are placed in 3D from one camera. It moves and turns each body so the pair's arrangement matches reality.
Pipeline: SAM 3D Body (one body per person) -> combined depth (MoGe-2 depth map + SAM 3D Body gap) -> this model.
Inputs per view: both bodies' 19 key points, per-region depth-map disagreement, ViTPose+ 2D residuals
(engage_place_features.py, overlap block dropped). Output: per body a centre shift and a small rotation.
Held-out test (Harmony4D, 4 unseen sequences, 1,068 views; pre-registered):
| SAM 3D Body | combined depth | + this model | |
|---|---|---|---|
| distance between the two people, median error | 11.2 cm | 3.95 | 3.17 |
| relative depth error | 16.0 cm | 5.8 | 4.3 |
| which body surfaces touch (F1) | 0.46 | 0.55 | 0.60 |
Limits: trained on one pair of athletes (standing grappling) + MMA; ground-truth person boxes; known camera intrinsics. On new people about two-thirds of the gain transfers. SAM 3D Body was trained on Harmony4D (all 24 subjects; arXiv 2602.15989, Table 1), so the absolute errors above, including on new people, are likely optimistic; comparisons within the experiment are unaffected.
Use: from engage_place_train import load_model; predict = load_model('placement_v1.pt')
(code: https://github.com/dhruvinci/engage3d, src/). Experiment: experiments/E06-placement-model-v1.
Data and upstream models: Harmony4D (MIT). The pipeline uses SAM 3D Body (SAM License), MoGe-2 and ViTPose+; their licences apply to the pipeline, not to these weights.