DarwinAnim8or/Trouper-v2-beta-12B-GGUF

Trouper-v2-12B — GGUF

Quantized versions of Trouper-v2-12B, a roleplay-focused 12B fine-tune with trained <think> block interiority.

For the prompt format, character card setup, sampler recommendations, and sample transcripts, see the main model repo. This page only covers the quants themselves.

Files

Quant	Size	Notes
IQ4_XS	~6.74 GB	Smallest recommended. Fits 8 GB VRAM with room for context.
Q4_K_M	~7.48 GB	Solid default for 8–12 GB cards.
Q5_K_M	~8.73 GB	Good quality/size balance.
Q6_K	~10.1 GB	Near-lossless in practice.
Q8_0	~12.5 GB	Effectively lossless.
BF16	~24.5 GB	Full-precision reference / requantization source.

All sub-8-bit quants were made with the custom importance matrix (imatrix.dat, included in this repo).

Why a custom imatrix?

An importance matrix guides quantization by recording which weights matter most over a set of calibration text — weights that fire hard on that text get more precision, the rest give some up. Most GGUF quants calibrate on generic data (wikitext or similar), which is a poor match for how an RP model is actually used.

These quants were calibrated on a mix of:

• ~40% general-purpose data (bartowski's calibration_datav3: prose, code, multilingual) to keep the model well-rounded, and
• ~60% in-domain roleplay data sampled from Trouper's own training distribution — real character cards, multi-turn RP exchanges, and <think> blocks, rendered through the exact chat template the model expects at inference, with special tokens parsed as tokens rather than literal text.

The result is that the token patterns Trouper actually lives in — persona adherence, think-block reasoning, the chat template scaffolding itself — are protected during quantization instead of being treated as statistical noise. The difference matters most at IQ4_XS and Q4_K_M; from Q6_K up the imatrix is largely along for the ride.

imatrix.dat is included, so you can requantize to other formats from the BF16 with the same calibration.

How they were made

llama.cpp (built from main with CUDA), convert_hf_to_gguf.py at bf16, imatrix computed fully GPU-offloaded over 250 chunks of the calibration mix, then quantized from the bf16 base with --imatrix.

Which one should I grab?

• 8 GB VRAM: IQ4_XS
• 12 GB VRAM: Q5_K_M or Q6_K
• 16 GB+ VRAM: Q6_K or Q8_0
• CPU / partial offload: any of the above, smaller = faster

If you find issues with a specific quant (repetition, template weirdness, degraded think blocks), please open a discussion and mention which file — that's exactly the kind of thing the custom imatrix is supposed to prevent, so reports help.