metadata
base_model: vikhyatk/moondream2
library_name: transformers
tags:
- moondream
- lora
- ui-grounding
- click-prediction
- vision-language
language:
- en
license: apache-2.0
Moondream2 LoRA v22 — UI grounding
LoRA adapter for Moondream2 (revision 2025-06-21) fine-tuned for UI click localization. This is the latest iteration in the v* series; for the version used in the published end-to-end benchmarks see Khabner/moondream-lora-v12.
Hyperparameters
LoRA rank 16, alpha 32, dropout 0.05; 2 epochs, lr 1e-4, cosine decay with 10% warmup; bfloat16 (GPU/MPS), float32 (CPU).
Usage
import importlib, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"vikhyatk/moondream2", revision="2025-06-21",
trust_remote_code=True, torch_dtype=torch.bfloat16,
)
model.model._setup_caches()
ckpt = torch.load(hf_hub_download("Khabner/moondream-lora-v22", "adapter.pt"), map_location="cpu", weights_only=True)
def _nest(flat):
tree = {}
for k, v in flat.items():
d = tree
for p in k.split(".")[:-1]:
d = d.setdefault(p, {})
d[k.split(".")[-1]] = v
return tree
inner = model.model
pkg = inner.__class__.__module__.rsplit(".", 1)[0]
flat = {k: v.to(device=str(inner.device), dtype=torch.bfloat16) for k, v in ckpt["lora"].items()}
importlib.import_module(f"{pkg}.moondream").variant_state_dict = lambda *a, **kw: _nest(flat)
if "coord_decoder" in ckpt:
cd = {k.removeprefix("coord_decoder."): v.to(device=str(inner.device), dtype=torch.bfloat16)
for k, v in ckpt["coord_decoder"].items()}
inner.region.coord_decoder.load_state_dict(cd)
model.eval()
result = model.model.point(image, "Send button", settings={"variant": "custom"})
Full reference inference and serving code: github.com/VLM-WEBTEST/magnitude_integration.