banner

Qwythos-9B-v2 โ€” full GGUF ladder ยท measured KLD fidelity + loop-rate test

This is NOT my model. Weights and training are by empero-ai (Qwythos-9B-v2, Apache-2.0). This is an independent GGUF repack whose value-add is what the other community GGUFs don't publish: an objective, measured fidelity sweep (KLD / PPL / Top-1 vs the Q8 reference) plus a real loop-rate benchmark on the quantized weights. No weights were modified.

What v2 fixes (per empero-ai): eliminated looping (6.7% โ†’ 0% at greedy decoding), restored the MTP head, cleaner identity, FTPO-trained. This repack is built from the official MTP-Q8_0 so the MTP head is preserved.

โœ… Recommended files

Use case File Size Top-1 vs Q8
Archival / best Qwythos-9B-v2-Q8_0.gguf 9.5 GB 100%
Near-lossless Qwythos-9B-v2-Q6_K.gguf 7.6 GB 98.2%
Balanced Qwythos-9B-v2-Q5_K_M.gguf 6.6 GB 97.2%
Best compact (imatrix) Qwythos-9B-v2-IQ4_XS.gguf 5.3 GB 96.2%
6 GB VRAM (imatrix 3-bit) Qwythos-9B-v2-IQ3_M.gguf 4.5 GB 93.0%
Smallest (lossy) Qwythos-9B-v2-Q3_K_M.gguf 4.7 GB 90.8%

The imatrix wins at every bit level. IQ4_XS dominates Q4_K_M (5.3 vs 5.8 GB, 96.2% vs 94.6% Top-1) and IQ3_M dominates Q3_K_M (4.5 vs 4.7 GB, 93.0% vs 90.8% Top-1, KLD 0.065 vs 0.102). For a 6 GB VRAM card, IQ3_M is the sweet spot โ€” smaller and more faithful than plain Q3_K_M, with room left for context. Prefer the IQ tiers over the plain Q tiers at 3- and 4-bit.

๐Ÿ“Š Measured fidelity โ€” vs the Q8_0 reference

llama-perplexity --kl-divergence over 24ร—512-token chunks. KLD (Kullbackโ€“Leibler divergence) is the gold-standard quant-fidelity metric โ€” it measures how far the quant's full next-token distribution drifts from the reference, catching damage that PPL and Top-1 alone miss. Top-1 match = how often the quant's argmax equals the reference's.

Quant Size PPL vs Q8 KLD mean KLD p50 Top-1 match
Q8_0 9.79 GB ref ref ref 100.0%
Q6_K 7.56 GB +0.41% 0.0070 0.0008 98.2%
Q5_K_M 6.64 GB +1.20% 0.0151 0.0020 97.2%
Q4_K_M 5.78 GB +1.60% 0.0377 0.0067 94.6%
IQ4_XS 5.33 GB +1.10% 0.0243 0.0045 96.2%
IQ3_M 4.52 GB +5.63% 0.0652 0.0210 93.0%
Q3_K_M 4.74 GB +6.90% 0.1017 0.0272 90.8%

fidelity vs size

Reference = Q8_0, not BF16. Qwythos-v2 is a hybrid qwen35 arch (3:1 Gated-DeltaNet linear-attention : full attention + MTP), which llama.cpp does not lower cleanly from safetensors โ€” so the ladder is requantized from empero-ai's official Q8_0 (near-lossless) with --allow-requantize + an imatrix, and fidelity is measured against that Q8_0.

๐Ÿงช Real loop-rate benchmark (measured on the quant)

v2's headline is "looping 6.7% โ†’ 0% at greedy decoding." We verified it on the quantized weights (Q5_K_M): greedy decode (temp 0), 10 diverse prompts ร— 220 tokens, flag any generation that degenerates into a repeated cycle or collapsed vocabulary.

Result: 0.0% loop rate (0/10 prompts) โ€” confirms the quantization preserves v2's loop-fix โ€” greedy (temp 0), 300 tokens each, mean repetition 0.038, every generation coherent (852โ€“1527 chars incl. a naturally-repetitive counting prompt). (Method + raw per-prompt output in scripts/loop_bench.py + reports/loop_bench.json.)

๐Ÿš€ Run it

ollama run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q5_K_M
llama-server -m Qwythos-9B-v2-Q5_K_M.gguf -c 32768

Sampling: qwen3.5-family reasoning models want temp 1.0 / top_p 0.95 / top_k 20. For Ollama use RENDERER qwen3.5 + PARSER qwen3.5 (recent build). v2 no longer loops at greedy, but temp 1.0 is still recommended for reasoning quality.

โš ๏ธ Honest scope

Two things are measured here: (1) KLD/PPL/Top-1 fidelity vs Q8, and (2) a loop-rate test. This is not an independent capability benchmark โ€” for agentic/reasoning scores (MMLU, GSM8K, HumanEval, GPQA), see empero-ai's card (their numbers, from their internal harness, not re-verified here).

๐Ÿ” Provenance

Requantized from the official MTP-Q8_0 with llama-quantize --allow-requantize + imatrix. Scripts in scripts/ (pipeline_v2.sh, parse_metrics.py, loop_bench.py), per-quant KLD logs in reports/, SHA-256 in reports/artifact-sha256sums.txt.

Importance matrix (qwythos-v2.imatrix, shipped in this repo): computed fresh on this model's own Q8 (not reused from other weights), over a general-purpose calibration corpus โ€” a ~250 KB mix of general English prose and code (standard for imatrix generation; not weight-specific data). Reuse it directly for your own quants. Note: the imatrix does not cover the MTP/nextn head (block 32) โ€” a normal forward pass doesn't activate it โ€” so IQ3_XS bails ("missing importance matrix โ€ฆ very low-bit"); IQ3_M quantizes cleanly (it tolerates the uncovered head) and is the recommended 3-bit tier.

Credit

Model ยฉ empero-ai (Apache-2.0). This repack + measured metrics: KikoCis.

Downloads last month
486
GGUF
Model size
9B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF

Finetuned
Qwen/Qwen3.5-9B
Quantized
(32)
this model