- Model Card for RTMPose-m AP-10K (animals) β Apple Core AI (fp16)
Model Card for RTMPose-m AP-10K (animals) β Apple Core AI (fp16)
Format conversion of OpenMMLab's RTMPose model
rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.pth (animal (mammal) pose estimation) to Apple Core AI (.aimodel) in fp16, for on-device inference on Apple silicon, where it
runs on the Neural Engine in under 2 ms per crop. Tested on macOS 27 (Apple M5) in Python and Swift; not tested on iOS.
Model Details
Model Description
RTMPose is a real-time top-down pose estimator: given a crop around one animal, it predicts 17 AP-10K keypoints (eyes, nose, neck, tail root, shoulders, elbows, front paws, hips, knees, back paws) as SimCC classification logits along x and y. This repository contains the network as one Core AI function with the input normalization built in; the detector, the crop and the decoding run in the host (reference code included).
- Original model developed by: OpenMMLab (RTMPose, Jiang et al., 2023).
- Conversion by: silv4souz4 (independent; not affiliated with or endorsed by OpenMMLab or Apple).
- Model type: animal (mammal) pose estimation, RTMPose-m (CSPNeXt-P5 backbone, RTMCC head with gated attention unit, SimCC), input 256Γ256 (WΓH).
- Base model:
rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.pthfrom the MMPose model zoo (SHA-256896e3665d849ef7eb9b6ec0995955796cc9810f024fa0aa0bdc18acb0d68bf52). Not hosted on the Hugging Face Hub by the authors, so nobase_modellink is declared. - Relation to base: format conversion to fp16 (BatchNorm folded into the convolutions); no quantization.
- License: Apache License 2.0, inherited from MMPose.
Model Sources
- Original code: https://github.com/open-mmlab/mmpose (RTMPose project)
- Paper: https://arxiv.org/abs/2303.07399
- AP-10K: https://github.com/AlexTheBad/AP-10K
- Sibling conversion: silv4souz4/rtmpose-m-wholebody-dwpose-coreai-fp16 β RTMPose-m whole-body (DWPose-m)
Uses
Direct Use
On-device animal (mammal) pose estimation from a bounding box: fitness and motion analysis, animation and avatars, gesture/interaction prototypes, animal behaviour observation, annotation tools.
Downstream Use
Keypoint pre-labelling for datasets, conditioning for image generation (pose-guided models), tracking pipelines combined with a detector.
Out-of-Scope Use
- Identifying people, biometric identification, or surveillance; inferring sensitive attributes from posture.
- Safety-critical or medical decisions without validation.
- Images without a box (the detector is not included) or species outside the AP-10K mammals (birds, reptiles, fish).
- Any use that violates applicable law.
Bias, Risks, and Limitations
- Inherits RTMPose's limitations: occlusion, unusual poses, motion blur and small crops reduce accuracy; performance across demographics and species follows the training data (AP-10K (23 mammal families, 54 species; pre-trained on AIC + COCO)).
- fp16 moves some decoded points by one 0.5 px bin against fp32; on our check one low-confidence point moved up to 12 px with the Neural Engine or CPU. No flip test (MMPose averages with a mirrored crop in its evaluation).
- Our accuracy check is tiny (2 animals (a jaguar and an antelope) from the MMPose test images (
tests/data/ap10k)); treat it as a sanity check, not a benchmark β see the published RTMPose results for accuracy. iOS is untested. - Debug information embedded in the asset contains build-time file paths of the converting machine; it has no effect on inference.
Recommendations
Load with the Neural Engine preferred; supply tight boxes from a good detector; filter points by score; validate on your own data and device.
How to Get Started with the Model
Download the whole repository (asset, host reference, config.json and checksums):
hf download silv4souz4/rtmpose-m-ap10k-coreai-fp16 --local-dir rtmpose-m-ap10k-coreai-fp16
config.json is a machine-readable summary of the function, tensor shapes and host steps below (it is also the file the Hub
uses to count downloads of this repository).
| File | Size | Function |
|---|---|---|
RTMPose_m_animal_ap10k_256x256_fp16.aimodel |
27.3 MB | main: image float16 [1,3,256,256] RGB in [0,1] β simcc_x [1,17,512], simcc_y [1,17,512] |
host_reference/pipeline_rtmpose.py |
3 KB | reference host (box β crop β model β decode), numpy/torch, comments in Portuguese |
Host steps (as in MMPose):
- Box β center and scale with padding 1.25 and the fixed input aspect (256:256); 3-point affine to 256Γ256, bilinear, zero border; RGB in [0,1] (the mean/std normalization is inside the model).
- Run
mainβsimcc_x,simcc_y. - Decode: argmax along each axis Γ· 2 (0.5 px bins); score = minimum of the two peak values; drop the point if score β€ 0.
- Back to the image:
p = p_crop / (256, 256) Β· scale + center β 0.5 Β· scale.
from coreai.runtime import AIModel, NDArray, SpecializationOptions, ComputeUnitKind
opts = SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.neural_engine())
m = await AIModel.load("RTMPose_m_animal_ap10k_256x256_fp16.aimodel", opts)
out = await m.load_function("main")({"image": NDArray(crop)}) # crop: float16 [1,3,256,256], RGB in [0,1]
simcc_x, simcc_y = out["simcc_x"].numpy(), out["simcc_y"].numpy()
Ahead-of-time compilation (.aimodelc) β tutorial
The asset here is a portable .aimodel. Apple's documentation, Compiling Core AI models ahead of time,
explains that every Core AI model must be specialized to the device before inference; the coreai-build tool moves the
most expensive part of that specialization, model compilation, to your build machine, so the model loads faster in your
app. It produces one .aimodelc per device architecture; you ship (or download) only the one that matches the device.
According to Apple, AOT compilation targets devices that support Apple Intelligence (iPhone/iPad with A17 Pro or later, Mac
with M1 or later, Apple Vision Pro); it is not supported on tvOS and watchOS.
1. Install the Metal Toolchain (Xcode > Settings > Components > Metal Toolchain, or):
xcodebuild -downloadComponent MetalToolchain
2. Compile (all architectures of a platform, or one with --architecture):
xcrun coreai-build compile RTMPose_m_animal_ap10k_256x256_fp16.aimodel --platform macOS --min-deployment-version 27.0 --output compiled/
xcrun coreai-build compile RTMPose_m_animal_ap10k_256x256_fp16.aimodel --platform iOS --min-deployment-version 27.0 --output compiled/
Output files are named <asset>.<arch>.aimodelc (for example .h17g for the Apple M5 used here); each works on any OS at
or above --min-deployment-version. Run coreai-build compile --help for --preferred-compute and the other options.
3. Load the matching bundle β the same API as for .aimodel:
let arch = AIModel.deviceArchitectureName
let url = compiledDir.appending(path: "RTMPose_m_animal_ap10k_256x256_fp16.\(arch).aimodelc")
let model = try await AIModel(contentsOf: url, options: .default)
Apple recommends hosting the per-architecture bundles and downloading only the matching one (for example with the BackgroundAssets framework). Some on-device specialization still remains after AOT, depending on the model and compute units.
What we measured (Apple M5, macOS 27.2, Swift runtime, first load with an empty specialization cache, two rounds):
| Step | Portable .aimodel |
AOT .aimodelc |
|---|---|---|
Compile on the Mac (coreai-build, h17g) |
β | 0.8 s |
| First load (model + function), cold cache | 0.5β0.7 s | 0.2 s |
| Later loads (cache warm) | 0.1β0.2 s | 0.1β0.2 s |
For a model this small the gain is modest, and in our runs the first inference call (8β10 ms on the Neural Engine) and the steady-state latency were the same with and without AOT. AOT is not an inference speed-up.
Training Details
Training Data
No training or fine-tuning. Weights are the official checkpoint trained on AP-10K (23 mammal families, 54 species; pre-trained on AIC + COCO); those datasets have their own terms.
Training Procedure (conversion)
- CSPNeXt + RTMCC head re-written in plain PyTorch (no mmpose/mmcv/mmengine) with the checkpoint's module names and strict loading (no missing or unexpected keys); input normalization moved inside the model.
- BatchNorm folded into the convolutions (exact in fp32, β€ 3e-6).
torch.exportβ decompositions β fp16 cast βcoreai-torchTorchConverterβ portable.aimodel.- Int8 variants were measured and rejected: int8 per channel keeps accuracy but runs 1.6β2.1Γ slower on the Neural Engine; int4 breaks the whole-body model. fp16 is the fastest and smallest-error option on Apple silicon here.
Evaluation
Testing Data, Factors & Metrics
(1) Parity with the official ONNX export (onnxruntime) on 3 noise inputs and real crops; (2) a small ground-truth sanity
check on 2 animals (a jaguar and an antelope) from the MMPose test images (tests/data/ap10k): mean error as % of the box's longer side and PCK at 5 % / 10 % of that side. Apple M5, Core AI runtime,
default and Neural-Engine-preferred.
Results
| Check | Result |
|---|---|
| fp32 re-write vs official ONNX | 112β119 dB, 0 decoded points different |
This fp16 asset vs official ONNX (simcc_x / simcc_y) |
58.5 / 57.2 dB; 10 / 85 decoded points moved by one 0.5 px bin |
| Ground truth, fp32 reference: error / PCK@0.05 / PCK@0.10 | 1.164 % / 0.969 / 1.000 |
| Ground truth, this fp16 asset (Neural Engine) | 1.177 % / 0.969 / 1.000 |
Swift vs Python, same asset (default) |
identical |
| Latency per crop, Swift (median of 3 rounds Γ 100 calls) | 1.60 ms Neural Engine preferred; 3.45 ms default (GPU) |
Latency was measured on a busy, fanless machine (load 4β5); treat it as indicative.
Summary
Task-equivalent to the fp32 original, 27 MB, under 2 ms per crop on the Apple M5 Neural Engine.
Environmental Impact
No training. Conversion and evaluation ran locally on one Apple M5 laptop; negligible.
Technical Specifications
Compute Infrastructure
- Hardware: Apple M5 (Mac17,3), 24 GB unified memory.
- Software: macOS 27.2, PyTorch 2.13,
coreai-torch0.4.3,coreai-opt0.3.0,coreai-core1.0.0b3,coreai-build3605.5.4; ONNX reference with onnxruntime 1.31.
Citation
If you use these assets, please cite RTMPose and AP-10K:
@misc{https://doi.org/10.48550/arxiv.2303.07399,
doi = {10.48550/ARXIV.2303.07399},
url = {https://arxiv.org/abs/2303.07399},
author = {Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai},
title = {RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
publisher = {arXiv},
year = {2023}
}
@inproceedings{yu2021ap,
title={AP-10K: A Benchmark for Animal Pose Estimation in the Wild},
author={Yu, Hang and Xu, Yufei and Zhang, Jing and Zhao, Wei and Guan, Ziyu and Tao, Dacheng},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021}
}
License and Attribution
- License: Apache License 2.0 β full text in
LICENSE, the license of MMPose (the RTMPose checkpoints are distributed in its model zoo). - Copyright of the original code and model: OpenMMLab.
- Modifications (Apache 2.0 Β§4(b)): plain-PyTorch re-write, BatchNorm folding, normalization inside the model, fp16
conversion to the Core AI format; the host reference is our re-implementation of MMPose's pre/post-processing. No
retraining. See
NOTICE. - "Apple", "Core AI" and "OpenMMLab" are trademarks of their respective owners; their use here is descriptive only.
Model Card Authors
silv4souz4
Model Card Contact
Please open a discussion in the Community tab of this repository.
- Downloads last month
- 19