neelvarma/dinov2-saes

DINOv2 sparse autoencoder checkpoints

This collection contains 15 project-trained TopK sparse autoencoders, their training configurations, and available evaluation and latent-frequency metadata. checkpoint_manifest.json lists each checkpoint's base model, layer, dimensions, sparsity, and membership in the 15-checkpoint paper release. The checkpoint paths and original weights are preserved.

The private research artifact is associated with The Null Problem in SAE Ablations at Vision Transformer Register Tokens. Its source repository is https://github.com/NVarma77/null-problem-sae-ablation (access may be restricted).

Historical selector provenance

Checkpoints named registers_only were trained using a historical selector that selected the final four sequence positions [257:261), while Hugging Face DINOv2-with-registers places its true register tokens at [1:5). These are terminal patch positions, and should not be interpreted as true-register-only SAEs. The collection preserves the historical names and behavior. See results/registers_only_provenance_audit/ for the recorded audit. The SAEs used for corrective analyses have separate all-token training configurations.

Checkpoint validation

All 15 checkpoints load and match the dimensions and sparsity in their configurations. 14 produce finite output for a zero-input smoke test with exact top-k encoding; 1 fail that test because saved parameters contain non-finite values. These failed runs are retained for provenance and are unsuitable for inference:

• saes/facebook_dinov2-with-registers-small/enc_res_out_layer_8_top_k_128_6_0.05_33958568_registers_only/trainer_0/ae.pt

Also, 0 checkpoints have a non-finite saved threshold buffer; threshold-based encoding is unsupported for those checkpoints. The default loader uses exact top-k encoding, which does not use this threshold. Each checkpoint's validation fields are recorded in checkpoint_manifest.json. This smoke test does not establish reproduction of the reported metrics.

Download and load

Install huggingface_hub and PyTorch, then download the collection and import the included inference-only loader. Replace neelvarma/dinov2-saes with this repository ID.

import sys
from huggingface_hub import snapshot_download

directory = snapshot_download("neelvarma/dinov2-saes", allow_patterns=[
    "saes/**", "dictionary_learning/**", "checkpoint_manifest.json"
])
sys.path.insert(0, directory)
from dictionary_learning.trainers.top_k import AutoEncoderTopK

sae = AutoEncoderTopK.from_pretrained(
    f"{directory}/saes/facebook_dinov2-with-registers-base/enc_res_out_layer_8_top_k_2048_6_1_32232117/trainer_0/ae.pt", device="cpu"
)
# Feed residual-stream activations from the model and layer in config.json.

These custom SAE state dictionaries use the included loader; they do not use the Transformers AutoModel interface.

Supporting files and verification

release-assets/ contains the checksum-pinned v0.1.0 research release: the raw results archive, the 15-checkpoint archive, and the complete offline ZIP. Its inventory identifies the files and source commit. The archives contain the curated 15-checkpoint release even when this collection includes extra SAEs.

From a complete downloaded snapshot, run sha256sum -c SHA256SUMS to verify all files, or run the same command inside release-assets/ to verify the three original archives. Base-model weights and ImageNet samples are obtained separately for reproduction.

Project-produced SAE weights and research material use CC BY 4.0. The included loader code uses MIT; see LICENSE, LICENSE-PAPER-DATA.md, NOTICE.md, and LICENSES/ for attribution and third-party terms.