InstantHMR
Paper: InstantHMR: Real-Time Human Mesh Recovery on Your Phone
Real-time 3D human mesh recovery from a single image, in the Momentum Human Rig (MHR). Code, demo and training: github.com/Naturalpad-studio/InstantHMR.
| File | Model | Parameters | EMDB-1 PA-MPJPE (mm) | 3DPW PA-MPJPE (mm) | Galaxy S23 NPU (ms) | Licence |
|---|---|---|---|---|---|---|
instanthmr-nano.onnx |
nano | 20.0 M | 61.7 | 46.6 | 4.60 | SAM License, non-commercial |
instanthmr-tiny.onnx |
tiny | 20.3 M | 58.0 | 44.0 | 4.84 | SAM License, non-commercial |
instanthmr-small.onnx |
small | 24.4 M | 53.3 | 41.7 | 5.38 | SAM License, non-commercial |
instanthmr-medium.onnx |
medium | 32.2 M | 48.8 | 40.0 | 5.49 | SAM License, non-commercial |
instanthmr-medium-commercial.onnx |
medium, commercially usable data | 32.2 M | 72.2 | 50.7 | 5.49 | SAM License |
mhr-body.onnx |
MHR body, shared by all models | SAM License |
New in v1.1: the kinematic fit. Each model also comes with the network and a kinematic fit in one
graph: the MHR pose is refined so that the body's keypoints, projected into the image, land on the
network's own 2D keypoints. The fit has no learned parameters and adds about 1 ms on a phone NPU.
-fit1 runs it on one crop of each person, -fit2 on two crops (a tight one and a zoomed-in one,
in the same batch).
| File | EMDB-1 PA-MPJPE (mm) | 3DPW PA-MPJPE (mm) | RTX 4070, TensorRT (ms) | Galaxy S23 NPU (ms) |
|---|---|---|---|---|
instanthmr-nano-fit1.onnx / -fit2.onnx |
57.2 / 55.9 | 44.5 / 44.1 | 0.85 / 1.10 | 5.49 / 8.73 |
instanthmr-tiny-fit1.onnx / -fit2.onnx |
53.8 / 52.6 | 42.4 / 42.0 | 0.87 / 1.13 | 5.84 / 9.20 |
instanthmr-small-fit1.onnx / -fit2.onnx |
49.2 / 48.1 | 40.3 / 40.0 | 1.00 / 1.30 | 6.37 / 10.36 |
instanthmr-medium-fit1.onnx / -fit2.onnx |
45.2 / 44.4 | 39.4 / 39.3 | 1.12 / 1.53 | 6.40 / 10.22 |
instanthmr-medium-commercial-fit1.onnx / -fit2.onnx |
68.8 / 68.3 | 47.7 / 47.5 | 1.12 / 1.53 | 6.40 / 10.22 |
Use
git clone https://github.com/Naturalpad-studio/InstantHMR.git && cd InstantHMR
pip install -r requirements.txt
python demo.py --image photo.jpg --model medium # network + fit on 2 crops (default)
python demo.py --video clip.mp4 --model small --fit 1 --pose-filter one-euro
The demo downloads the files it needs from this repository.
The files
A model graph takes a person crop and returns the MHR parameters:
- inputs:
image(B, 3, S, S), an RGB crop normalised with the ImageNet mean and standard deviation, S in theimage_sizemetadata;cliff_cond(B, 3), the box position in the full frame; - outputs:
mhr_params(B, 204),shape_params(B, 45),cam_trans(B, 3) in metres,joints_2d(B, 70, 2) in [-1, 1] over the crop.
A fit graph (-fit1, -fit2) takes one person's N crops (N = 1 or 2, in its crops metadata), cut
at the scales in its crop_scales metadata from the person box enlarged to the ground-truth
convention (a detector box x 1/0.866):
- inputs:
image(N, S, S, 3) uint8 RGB crops (normalised inside the graph);cliff_cond(N, 3);crop_box(N, 4), each crop's pixel square (x1, y1, w, h) in the frame;cam(1, 3), the focal length and principal point in pixels; - outputs:
mhr_params(1, 204),shape_params(1, 45),cam_trans(1, 3),joints_2d(1, 70, 2) in pixels.
It is fp16 inside (inputs and outputs are float32 or uint8) and runs on onnxruntime's CPU, CUDA and TensorRT providers and on phone NPUs.
mhr-body.onnx turns mhr_params and shape_params into the 70 3D keypoints joints_3d
(B, 70, 3) and the 18,439-vertex mesh vertices (B, 18439, 3), in metres, centred on the body
(add cam_trans for camera space). The mesh faces are in its mesh_faces metadata. The crop and
the cliff_cond vector are built by instanthmr/inference.py in the code repository, for both kinds
of graph.
PyTorch checkpoints (for fine-tuning)
pytorch/<model>/ holds each model's PyTorch weights and the run_config.json of its training run,
to continue training on your own data with the training code of the
GitHub repository:
| Folder | Backbone | Training mix | Checkpoint epoch |
|---|---|---|---|
pytorch/nano/ |
hgnetv2_b0 |
SA-1B, AI Challenger, Harmony4D, COCO, MPII | 307 |
pytorch/tiny/ |
hgnetv2_b1 |
SA-1B, AI Challenger, Harmony4D, COCO, MPII | 386 |
pytorch/small/ |
hgnetv2_b2 |
SA-1B, AI Challenger, Harmony4D, COCO, MPII | 394 |
pytorch/medium/ |
hgnetv2_b4 |
SA-1B, AI Challenger, Harmony4D, COCO, MPII | 474 |
pytorch/medium-commercial/ |
hgnetv2_b4 |
Harmony4D, COCO (CC BY / no known copyright) | 159 |
They are the checkpoints the ONNX graphs above were exported from (the best 3DPW validation epoch of each run), with the weights only, no optimiser state.
hf download momolesang/InstantHMR --include "pytorch/medium-commercial/*" --local-dir .
torchrun --standalone --nproc_per_node=1 training/train_ddp.py \
--init-from pytorch/medium-commercial/instanthmr-medium-commercial.pth \
--data_root <data> --output_dir runs/my-finetune --mix mydata=0.5,harmony4d=0.25,coco=0.25 \
... # the other flags of the model's training command, including --cliff-focal
python tools/export_onnx.py --ckpt runs/my-finetune/best_student_model_v3.pth --fit 2 \
--output models/my-model-fit2.onnx
--init-from starts a new run from these weights with a fresh optimiser and schedule; the model
built from your flags must match the checkpoint, so start from that model's training command
(see Fine-tuning a released model).
The exporter reads cliff_focal from the run_config.json beside the checkpoint, so keep the
two together. Each file is a dict (model_state_dict, epoch, val_pa_mpjpe, source) that
loads with torch.load(path, weights_only=True).
A fine-tuned model is a derivative of these weights and stays under the SAM License, with the same non-commercial restriction for the models trained on non-commercial images. If you can, please share your fine-tuned models: it helps other users and keeps this project going.
Training data and licences
The models are trained on the released annotations of the
SAM 3D Body dataset, which Meta
distributes under the SAM License, and the graphs contain tables from the SAM 3D Body
checkpoint (the bone-scale and hand pose bases, and in mhr-body.onnx the 70-keypoint readout).
Every file in this repository is therefore distributed under the SAM License. It
allows commercial use, under the conditions it lists (trade controls, no military use, among
others).
In addition, the nano, tiny, small and medium weights are for non-commercial use only, because
some of their training images (SA-1B, AI Challenger, and COCO photos under non-commercial
Flickr licences) are licensed for research or non-commercial use only. medium-commercial was
trained only on Harmony4D and the COCO images under CC BY or with no known copyright, and has no
such restriction. The body model inside mhr-body.onnx is Meta's
MHR (Apache-2.0).
Earlier version
The first InstantHMR release (RepViT backbone), a distillation of
facebook/sam-3d-body-dinov3, is kept in
legacy/; instanthmr.onnx, instanthmr.pth and instanthmr.gif at the top level are
identical copies, kept so that existing links to them keep working. It is under the SAM License.