UCSC-VLAA/openvision2-vit-huge-patch14-336-vision-only

OpenVision2 · H/14 @336 — vision encoder + caption decoder

This repo now contains the full OpenVision2 generative model: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image -> a descriptive caption. The encoder files are unchanged; only the decoder (and this card) were added.

Files

file	role
open_clip_pytorch_model.bin, open_clip_config.json	H/14 vision encoder (open_clip format)
caption_decoder.safetensors	caption text decoder weights
text_decoder_config.json	decoder architecture config
modeling_openvision2_decoder.py	standalone PyTorch decoder + generate()
bert_base_vocab_bos_eos.txt	tokenizer vocab ([PAD]=0 [bos]=1 [eos]=2)
caption_example.py	end-to-end image -> caption demo

Decoder architecture

A concat / prefix-LM autoregressive transformer (not a CoCa cross-attention decoder): ViT patch tokens are linearly projected and prepended as a bidirectional prefix, and text is generated causally while attending to all image tokens.

24 layers · width 1024 · 16 heads · mlp 4096 · vocab 32000 · pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream. It consumes the encoder's pre-final-norm patch tokens (open_clip output_tokens=True).

Usage

Needs the patched open_clip providing create_vision_encoder_and_transforms (https://github.com/UCSC-VLAA/OpenVision), plus torch, safetensors, pillow. See caption_example.py:

python caption_example.py --image your.jpg

Notes

• Encoder and decoder are a matched pair exported from the same training checkpoint.
• Captions are LLaVA-style dense descriptions (multi-sentence, detailed).