InstantHMR

Paper: InstantHMR: Real-Time Human Mesh Recovery on Your Phone

Real-time 3D human mesh recovery from a single image, in the Momentum Human Rig (MHR). Code, demo and training: github.com/Naturalpad-studio/InstantHMR.

File Model Parameters EMDB-1 PA-MPJPE (mm) 3DPW PA-MPJPE (mm) Galaxy S23 NPU (ms) Licence
instanthmr-nano.onnx nano 20.0 M 61.7 46.6 4.60 SAM License, non-commercial
instanthmr-tiny.onnx tiny 20.3 M 58.0 44.0 4.84 SAM License, non-commercial
instanthmr-small.onnx small 24.4 M 53.3 41.7 5.38 SAM License, non-commercial
instanthmr-medium.onnx medium 32.2 M 48.8 40.0 5.49 SAM License, non-commercial
instanthmr-medium-commercial.onnx medium, commercially usable data 32.2 M 72.2 50.7 5.49 SAM License
mhr-body.onnx MHR body, shared by all models SAM License

New in v1.1: the kinematic fit. Each model also comes with the network and a kinematic fit in one graph: the MHR pose is refined so that the body's keypoints, projected into the image, land on the network's own 2D keypoints. The fit has no learned parameters and adds about 1 ms on a phone NPU. -fit1 runs it on one crop of each person, -fit2 on two crops (a tight one and a zoomed-in one, in the same batch).

File EMDB-1 PA-MPJPE (mm) 3DPW PA-MPJPE (mm) RTX 4070, TensorRT (ms) Galaxy S23 NPU (ms)
instanthmr-nano-fit1.onnx / -fit2.onnx 57.2 / 55.9 44.5 / 44.1 0.85 / 1.10 5.49 / 8.73
instanthmr-tiny-fit1.onnx / -fit2.onnx 53.8 / 52.6 42.4 / 42.0 0.87 / 1.13 5.84 / 9.20
instanthmr-small-fit1.onnx / -fit2.onnx 49.2 / 48.1 40.3 / 40.0 1.00 / 1.30 6.37 / 10.36
instanthmr-medium-fit1.onnx / -fit2.onnx 45.2 / 44.4 39.4 / 39.3 1.12 / 1.53 6.40 / 10.22
instanthmr-medium-commercial-fit1.onnx / -fit2.onnx 68.8 / 68.3 47.7 / 47.5 1.12 / 1.53 6.40 / 10.22

Use

git clone https://github.com/Naturalpad-studio/InstantHMR.git && cd InstantHMR
pip install -r requirements.txt
python demo.py --image photo.jpg --model medium              # network + fit on 2 crops (default)
python demo.py --video clip.mp4 --model small --fit 1 --pose-filter one-euro

The demo downloads the files it needs from this repository.

The files

A model graph takes a person crop and returns the MHR parameters:

  • inputs: image (B, 3, S, S), an RGB crop normalised with the ImageNet mean and standard deviation, S in the image_size metadata; cliff_cond (B, 3), the box position in the full frame;
  • outputs: mhr_params (B, 204), shape_params (B, 45), cam_trans (B, 3) in metres, joints_2d (B, 70, 2) in [-1, 1] over the crop.

A fit graph (-fit1, -fit2) takes one person's N crops (N = 1 or 2, in its crops metadata), cut at the scales in its crop_scales metadata from the person box enlarged to the ground-truth convention (a detector box x 1/0.866):

  • inputs: image (N, S, S, 3) uint8 RGB crops (normalised inside the graph); cliff_cond (N, 3); crop_box (N, 4), each crop's pixel square (x1, y1, w, h) in the frame; cam (1, 3), the focal length and principal point in pixels;
  • outputs: mhr_params (1, 204), shape_params (1, 45), cam_trans (1, 3), joints_2d (1, 70, 2) in pixels.

It is fp16 inside (inputs and outputs are float32 or uint8) and runs on onnxruntime's CPU, CUDA and TensorRT providers and on phone NPUs.

mhr-body.onnx turns mhr_params and shape_params into the 70 3D keypoints joints_3d (B, 70, 3) and the 18,439-vertex mesh vertices (B, 18439, 3), in metres, centred on the body (add cam_trans for camera space). The mesh faces are in its mesh_faces metadata. The crop and the cliff_cond vector are built by instanthmr/inference.py in the code repository, for both kinds of graph.

PyTorch checkpoints (for fine-tuning)

pytorch/<model>/ holds each model's PyTorch weights and the run_config.json of its training run, to continue training on your own data with the training code of the GitHub repository:

Folder Backbone Training mix Checkpoint epoch
pytorch/nano/ hgnetv2_b0 SA-1B, AI Challenger, Harmony4D, COCO, MPII 307
pytorch/tiny/ hgnetv2_b1 SA-1B, AI Challenger, Harmony4D, COCO, MPII 386
pytorch/small/ hgnetv2_b2 SA-1B, AI Challenger, Harmony4D, COCO, MPII 394
pytorch/medium/ hgnetv2_b4 SA-1B, AI Challenger, Harmony4D, COCO, MPII 474
pytorch/medium-commercial/ hgnetv2_b4 Harmony4D, COCO (CC BY / no known copyright) 159

They are the checkpoints the ONNX graphs above were exported from (the best 3DPW validation epoch of each run), with the weights only, no optimiser state.

hf download momolesang/InstantHMR --include "pytorch/medium-commercial/*" --local-dir .
torchrun --standalone --nproc_per_node=1 training/train_ddp.py \
    --init-from pytorch/medium-commercial/instanthmr-medium-commercial.pth \
    --data_root <data> --output_dir runs/my-finetune --mix mydata=0.5,harmony4d=0.25,coco=0.25 \
    ...   # the other flags of the model's training command, including --cliff-focal
python tools/export_onnx.py --ckpt runs/my-finetune/best_student_model_v3.pth --fit 2 \
    --output models/my-model-fit2.onnx

--init-from starts a new run from these weights with a fresh optimiser and schedule; the model built from your flags must match the checkpoint, so start from that model's training command (see Fine-tuning a released model). The exporter reads cliff_focal from the run_config.json beside the checkpoint, so keep the two together. Each file is a dict (model_state_dict, epoch, val_pa_mpjpe, source) that loads with torch.load(path, weights_only=True).

A fine-tuned model is a derivative of these weights and stays under the SAM License, with the same non-commercial restriction for the models trained on non-commercial images. If you can, please share your fine-tuned models: it helps other users and keeps this project going.

Training data and licences

The models are trained on the released annotations of the SAM 3D Body dataset, which Meta distributes under the SAM License, and the graphs contain tables from the SAM 3D Body checkpoint (the bone-scale and hand pose bases, and in mhr-body.onnx the 70-keypoint readout). Every file in this repository is therefore distributed under the SAM License. It allows commercial use, under the conditions it lists (trade controls, no military use, among others).

In addition, the nano, tiny, small and medium weights are for non-commercial use only, because some of their training images (SA-1B, AI Challenger, and COCO photos under non-commercial Flickr licences) are licensed for research or non-commercial use only. medium-commercial was trained only on Harmony4D and the COCO images under CC BY or with no known copyright, and has no such restriction. The body model inside mhr-body.onnx is Meta's MHR (Apache-2.0).

Earlier version

The first InstantHMR release (RepViT backbone), a distillation of facebook/sam-3d-body-dinov3, is kept in legacy/; instanthmr.onnx, instanthmr.pth and instanthmr.gif at the top level are identical copies, kept so that existing links to them keep working. It is under the SAM License.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for momolesang/InstantHMR

Finetunes
1 model

Paper for momolesang/InstantHMR