Instructions to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Use Docker
docker model run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with Ollama:
ollama run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with Docker Model Runner:
docker model run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
- Lemonade
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Qwythos-9B-v2-Full-Ladder-KLD-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwythos-9B-v2 โ full GGUF ladder ยท measured KLD fidelity + loop-rate test
This is NOT my model. Weights and training are by empero-ai (Qwythos-9B-v2, Apache-2.0). This is an independent GGUF repack whose value-add is what the other community GGUFs don't publish: an objective, measured fidelity sweep (KLD / PPL / Top-1 vs the Q8 reference) plus a real loop-rate benchmark on the quantized weights. No weights were modified.
What v2 fixes (per empero-ai): eliminated looping (6.7% โ 0% at greedy decoding), restored the MTP head, cleaner identity, FTPO-trained. This repack is built from the official MTP-Q8_0 so the MTP head is preserved.
โ Recommended files
| Use case | File | Size | Top-1 vs Q8 |
|---|---|---|---|
| Archival / best | Qwythos-9B-v2-Q8_0.gguf | 9.5 GB | 100% |
| Near-lossless | Qwythos-9B-v2-Q6_K.gguf | 7.6 GB | 98.2% |
| Balanced | Qwythos-9B-v2-Q5_K_M.gguf | 6.6 GB | 97.2% |
| Best compact (imatrix) | Qwythos-9B-v2-IQ4_XS.gguf | 5.3 GB | 96.2% |
| 6 GB VRAM (imatrix 3-bit) | Qwythos-9B-v2-IQ3_M.gguf | 4.5 GB | 93.0% |
| Smallest (lossy) | Qwythos-9B-v2-Q3_K_M.gguf | 4.7 GB | 90.8% |
The imatrix wins at every bit level. IQ4_XS dominates Q4_K_M (5.3 vs 5.8 GB, 96.2% vs 94.6% Top-1) and IQ3_M dominates Q3_K_M (4.5 vs 4.7 GB, 93.0% vs 90.8% Top-1, KLD 0.065 vs 0.102). For a 6 GB VRAM card, IQ3_M is the sweet spot โ smaller and more faithful than plain Q3_K_M, with room left for context. Prefer the IQ tiers over the plain Q tiers at 3- and 4-bit.
๐ Measured fidelity โ vs the Q8_0 reference
llama-perplexity --kl-divergence over 24ร512-token chunks. KLD (KullbackโLeibler divergence) is the gold-standard quant-fidelity metric โ it measures how far the quant's full next-token distribution drifts from the reference, catching damage that PPL and Top-1 alone miss. Top-1 match = how often the quant's argmax equals the reference's.
| Quant | Size | PPL vs Q8 | KLD mean | KLD p50 | Top-1 match |
|---|---|---|---|---|---|
| Q8_0 | 9.79 GB | ref | ref | ref | 100.0% |
| Q6_K | 7.56 GB | +0.41% | 0.0070 | 0.0008 | 98.2% |
| Q5_K_M | 6.64 GB | +1.20% | 0.0151 | 0.0020 | 97.2% |
| Q4_K_M | 5.78 GB | +1.60% | 0.0377 | 0.0067 | 94.6% |
| IQ4_XS | 5.33 GB | +1.10% | 0.0243 | 0.0045 | 96.2% |
| IQ3_M | 4.52 GB | +5.63% | 0.0652 | 0.0210 | 93.0% |
| Q3_K_M | 4.74 GB | +6.90% | 0.1017 | 0.0272 | 90.8% |
Reference = Q8_0, not BF16. Qwythos-v2 is a hybrid
qwen35arch (3:1 Gated-DeltaNet linear-attention : full attention + MTP), which llama.cpp does not lower cleanly from safetensors โ so the ladder is requantized from empero-ai's official Q8_0 (near-lossless) with--allow-requantize+ an imatrix, and fidelity is measured against that Q8_0.
๐งช Real loop-rate benchmark (measured on the quant)
v2's headline is "looping 6.7% โ 0% at greedy decoding." We verified it on the quantized weights (Q5_K_M): greedy decode (temp 0), 10 diverse prompts ร 220 tokens, flag any generation that degenerates into a repeated cycle or collapsed vocabulary.
Result: 0.0% loop rate (0/10 prompts) โ confirms the quantization preserves v2's loop-fix โ greedy (temp 0), 300 tokens each, mean repetition 0.038, every generation coherent (852โ1527 chars incl. a naturally-repetitive counting prompt). (Method + raw per-prompt output in scripts/loop_bench.py + reports/loop_bench.json.)
๐ Run it
ollama run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q5_K_M
llama-server -m Qwythos-9B-v2-Q5_K_M.gguf -c 32768
Sampling: qwen3.5-family reasoning models want temp 1.0 / top_p 0.95 / top_k 20. For Ollama use RENDERER qwen3.5 + PARSER qwen3.5 (recent build). v2 no longer loops at greedy, but temp 1.0 is still recommended for reasoning quality.
โ ๏ธ Honest scope
Two things are measured here: (1) KLD/PPL/Top-1 fidelity vs Q8, and (2) a loop-rate test. This is not an independent capability benchmark โ for agentic/reasoning scores (MMLU, GSM8K, HumanEval, GPQA), see empero-ai's card (their numbers, from their internal harness, not re-verified here).
๐ Provenance
Requantized from the official MTP-Q8_0 with llama-quantize --allow-requantize + imatrix. Scripts in scripts/ (pipeline_v2.sh, parse_metrics.py, loop_bench.py), per-quant KLD logs in reports/, SHA-256 in reports/artifact-sha256sums.txt.
Importance matrix (qwythos-v2.imatrix, shipped in this repo): computed fresh on this model's own Q8 (not reused from other weights), over a general-purpose calibration corpus โ a ~250 KB mix of general English prose and code (standard for imatrix generation; not weight-specific data). Reuse it directly for your own quants. Note: the imatrix does not cover the MTP/nextn head (block 32) โ a normal forward pass doesn't activate it โ so IQ3_XS bails ("missing importance matrix โฆ very low-bit"); IQ3_M quantizes cleanly (it tolerates the uncovered head) and is the recommended 3-bit tier.
Credit
Model ยฉ empero-ai (Apache-2.0). This repack + measured metrics: KikoCis.
- Downloads last month
- 486
3-bit
4-bit
5-bit
6-bit
8-bit
Model tree for KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF
Base model
Qwen/Qwen3.5-9B-Base
