| # Results: two files, one tensor apart |
|
|
| Date: 2026-09-04. Two runs of the same benchmarks against two model files |
| at temperature 0.0. Method: [`benchmarking.md`](./benchmarking.md) |
| (harness, settings, adaptations, extraction rules — everything needed |
| to reproduce). |
|
|
| ## Setup |
|
|
| - File A: `TheDrummer_Orion-26B-A4B-v1-IQ4_NL.gguf` (Gemma 4 26B MoE, |
| IQ4), served via llama-server, unmodified. |
| - File B: `lm_head-TheDrummer_Orion-26B-A4B-v1-IQ4_NL.gguf` — file A with |
| its `lm_head` tensor replaced via `voice cast`, Q8_0. Nothing else changed. |
| - Both files: temperature 0.0, seed 1234, identical prompts, identical token |
| budgets (ARC 1024, MMLU 2048, GSM8K 1024), identical answer extraction. |
| Unanswered items got one rescue pass at 4096 (MMLU) / 2048 (GSM8K), |
| applied to both files symmetrically. Rescue tie-break: on split pairs |
| (two verdicts for one doc) the decided verdict beats `unanswered`. |
| |
| ## Scores |
| |
| | Task | n | A correct | B correct | A acc | B acc | |
| |---|---|---|---|---|---| |
| | ARC-Challenge (generative) | 100 | 94 | 95 | 0.940 | 0.950 | |
| | MMLU (generative, 2 × 57 subjects) | 114 | 101 | 103 | 0.886 | 0.904 | |
| | GSM8K | 100 | 76 | 92 | 0.760 | 0.920 | |
| |
| ## Observations |
| |
| 1. **ARC (+1) and MMLU (+2) differences fall within slice noise** (±5 |
| points at n≈100). GSM8K (+16) exceeds it; see point 2. |
| 2. **The GSM8K difference sits in unanswered counts.** File A left 23 |
| items silent (empty content after thought exhausted the budget); file |
| B left 6. On items both files answered, accuracy is 0.987 (A) vs |
| 0.979 (B). The traces hold whatever else there is to see. |
| 3. **Thinking lengths roughly match.** No pairing script was archived, so no |
| median is claimed; per-file thinking lengths sit in the same range. |
| 4. **Error overlap is minimal.** One GSM8K item was wrong under both |
| files; remaining errors are different items per file. |
| |
| ## What this does not show |
| |
| - Small slices: ~100 items per task. Differences under ~5 points are noise. |
| - Generative scoring is not comparable to published logprob leaderboards; |
| compare only file A vs. file B within this table. |
| - One model pair, one replaced tensor, greedy decoding only. Other pairs |
| may differ — that is what the harness in `benchmarking/` is for. |
| - Writing voice is measured separately below — same two files, same |
| neutrality, no judge model. |
| |
| ## Evidence |
| |
| Per file, archived under `benchmarking/runs/file-a/` and |
| `benchmarking/runs/file-b/`: harness `results_*.json` and |
| `samples_*.jsonl` for every run, rescue files for the second passes, and |
| the thinking log (every prompt, answer, and reasoning trace). No personal |
| information in any of it — server file paths are redacted to run labels |
| at the log source. |
|
|
| ## Part II — writing voice: 32 prompts, two files |
|
|
| Method: [`creative-bench.md`](./creative-bench.md). 32 EQ-Bench v3 prompts, |
| one generation per prompt per file at temperature 0.7, 6000-token budget. |
| Measured, not judged: shared trigram vocabulary, cliche density, texture |
| stats. Blind A/B pairs archived for readers. |
|
|
| | Measure | File A | File B | |
| |---|---|---| |
| | Shared trigram vocab, mean (max) | 0.013 (0.027) across all 32 | same pair | |
| | Cliche hits per 100 words | 0.141 | 0.075 | |
| | Avg words per piece | 1018.0 | 878.6 | |
| | Lexical diversity (TTR) | 0.4439 | 0.5054 | |
| | Avg sentence words | 9.3 | 8.8 | |
|
|
| Per-prompt exhibits: `creative-bench/runs/file-a-creative.jsonl` and |
| `file-b-creative.jsonl` (prompt, full text, full reasoning trace each). |
|
|
