dongnhdev/Qwen3.8-Flash-Next-OrcaUncensored-IQ2-Light

Qwen3.8-Flash-Next OrcaUncensored — DS4 IQ2 (Light)

Qwen3.8-Flash-Next in the light DS4-IQ2 packaging of ivanfioravanti/Qwen3.8-Flash-Next-DS4-IQ2: an IQ2 main (IQ2_XXS gate/up, Q2_K down padded to 768, embedded MTP block) plus an external demand-paged PLE Q4_1 sidecar instead of a ~95 GiB resident BF16 n-gram. Runs resident and zero-swap on a 64 GiB Apple Silicon box (measured on M5 Pro), 8K→220K context.

Performance (M5 Pro 64 GiB, measured)

model	MTP-off	single-stream --mtp	notes
Q8 dense (exact)	~30–32 t/s	~38 t/s	most accurate
Q4_K imat dense	~36 t/s	~42–43 t/s	+~14% vs Q8, near-Q8 quality

• MTP (--mtp) adds ~+17% single-stream over MTP-off; draft acceptance ~67%.
• Adaptive draft depth (engine env DS4_QWEN4_MTP_DEPTH, default auto): drafts a 2nd token on deterministic/structured/code continuations for a further +5–6%, and falls back on free-form prose so it never regresses. Output stays autoregressive-exact (verify only commits argmax-matching drafts → zero quality change).
• Multi-session throughput (engine --batched-session N, concurrent requests, full quality): ~67 t/s @ 8 streams, ~80 t/s @ 16 streams aggregate (~2× a single stream). Decode is memory-bandwidth-bound on this hardware, so aggregate throughput plateaus around ~80 t/s rather than scaling linearly; a single stream cannot exceed ~43 t/s here (that ceiling needs a smaller model or more MTP heads, not tuning). Concurrency is bounded by ctx × sessions KV vs 64 GiB (ctx 4096 fits 16 streams; large ctx needs fewer).

Two dense-quant mains (pick one; both share the same PLE sidecar)

The IQ2 experts are identical across both; they differ only in how the per-layer dense projections are quantized:

main	dense	size	decode vs Q8	logit cosine vs Q8	pick when
...-Q2KDownPad768-MTP.gguf	Q8_0	41.73 GiB	baseline	1.000 (exact)	most accurate output
...-DenseQ4Kselimat-MTP.gguf	selective Q4_K + imatrix	40.68 GiB	+~14%	0.949 (argmax preserved)	faster decode, near-Q8 quality

The DenseQ4Kselimat variant requantizes the less-sensitive dense projections Q8_0→Q4_K (routed to DS4's fast dense Q4_K GEMV) while keeping the most quality-sensitive full-attention q/k/v/output projections at Q8_0, and weights each dense tensor's Q4_K quantization by an importance matrix (activation energy per input column). Net: ~+14% decode at logit cosine ~0.949 vs Q8 (argmax preserved, coherent). Encode-only change (same runtime kernel/bytes as plain Q4_K; the imatrix costs nothing at inference). imatrix source: the published llama.cpp calibration unsloth/Qwen3.8-Flash-Next-GGUF (same qwen4exp tensor decomposition).

Files

• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-MTP.gguf — Q8 dense main (exact). 44,806,612,448 bytes, sha256 e078c60abfdc9c5dd849eddb41660e5fc8f1b0da2f1200c643a8d3ec50324e8a.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-DenseQ4Kselimat-MTP.gguf — imatrix-weighted selective-Q4K dense main (faster, near-Q8). 43,677,591,168 bytes, sha256 b1dd08509231126b5f1596603fd2589ff8bdf8eae221420f3e4409110fee8bd4.
• Qwen3.8-Flash-Next-PLE-Q4_1.gguf — PLE Q4_1 n-gram sidecar, shared by both mains (required). Byte-identical between base Qwen and this fine-tune (abliteration does not touch the n-gram table), so it is Ivan's sidecar verbatim.
• Qwen3.8-Flash-Next-PLE-Q4_0.gguf — experimental PLE sidecar requantized Q4_1→Q4_0 with gguf-tools/ple_requant.py (−3 GiB). Not recommended: round-trip relL2 0.138 vs Q4_1, and measured no speed or context gain (PLE is CPU-mmap, not Metal-wired). Drop-in for --ple. 28,800,141,632 bytes, sha256 68920ba4887e7e463263ce0c771aacd1902b315c89145ab090e08e3c2d69d611.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-SelQ4Down18-48-DenseQ4Kselimat-MTP.gguf — "unc31" selective tier: 30 of 48 down layers (18–47) Q4_K. 51,201,124,096 bytes, sha256 7f45767af63a45b17971870d2288d8dbdcd92236f4961c69f531df165aaadef8.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownPad768-DenseQ4Kselimat-MTP.gguf — "unc48L" (Complete), strongest uncensor: all 48 trunk down layers Q4_K, MTP blk.48 down Q8_0. Needs SSD expert streaming on 64 GiB (see "Running unc48L at 256K"). 55,447,856,768 bytes, sha256 20b4c617a5e7a8d3976d328daf04c48c0f250f47c9a1e7ad91265e698616c41d.
• Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownTrim-DenseQ4Kselimat-MTP.gguf — "unc48L Trim": the same model as unc48L with the padding removed from its Q4_K down rows. A down row holds 640 real values, and unc48L pads it to 768 to fill whole 256-value Q4_K blocks. The Trim file keeps each row's two whole blocks plus the header and real nibbles of the third (368 bytes instead of 432). Every byte the kernels read is unchanged, and the output is byte-identical to unc48L. 3.75 GiB smaller. Needs ds4-metal develop 331cf3b2 or later; older engines refuse the file. 51,421,324,928 bytes, sha256 3a7c8430b25d3e1a3feeaa7c620e642d453f4d41f9a8c3d9d86d1e94afef22b9. The .gguf.json next to it lists every tensor's sha256.
• adapters/refusal-4-44.f32 — optional runtime refusal-projection adapter (480 KiB). See "Runtime refusal projection" below.
• Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt — optional MTP draft vocabulary (65,568 token ids, most frequent first; 390,414 bytes, sha256 f8b0dc7c28f173c4b071c986cf44d59ff4f169ad5a6442a88ea240e3b1fbe652). Shared by every main; see "MTP draft vocabulary" below.

Uncensor tiers: how strong each one is (2026-09-21; unc31/unc48L fit updated 2026-09-23)

Uncensor strength is set by how many trunk down-expert layers survive the abliteration delta — 2-bit (Q2_K) down experts erase most of it, Q4_K preserves it. The tiers:

tier	Q4_K down layers	uncensor strength (10-prompt probe, greedy)	fits on M5 Pro 64 GiB
Q2K main	0	weakest: only low-band relaxes (profanity)	220K–256K
unc8 (removed 2026-10-03)	7 (41–47)	≈ Q2K baseline: mid-band (phishing/hotwire) still refused	256K, ~42 t/s @4K
unc16 (removed 2026-10-03)	15 (33–47)	≈ Q2K baseline: mid-band still refused	256K, 40.1 t/s @4K
unc31	30 (18–47)	mid-band full: phishing/hotwire/lockpick comply; break-in + extreme (meth/IED) still refused	256K with SSD streaming (K=36): 46.5 t/s short prompt
unc48L (Complete)	48 (0–47)	strongest (extreme-band, disclaimer-led)	256K with SSD streaming (K=32): 42–43 t/s short prompt, **34 t/s prose at 256K full**, 49.3 GiB peak

Pick unc48L Trim (or unc48L on an older engine) for the strongest uncensor, unc31 for mid-band uncensor with a little more speed and faster prefill, or the Q2K mains for the lightest footprint. The unc8 and unc16 files were removed on 2026-10-03 because they refused like the Q2K mains. unc31 and unc48L at long context need the SSD-streaming engine below; the older engine OOMs them at 256K.

Running unc48L at 256K (SSD expert streaming)

The ds4-metal develop engine keeps the first K routed layers resident and streams the experts of the rest through a locked RAM cache; KV is 4-bit by default. On M5 Pro 64 GiB:

DS4_QWEN4_STREAM_FULL_LAYERS=32 DS4_QWEN4_PLE_PREFETCH_FULL=0 \
DS4_QWEN4_MTP_DRAFT_VOCAB=Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt \
ds4-server -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownPad768-DenseQ4Kselimat-MTP.gguf \
    --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf -c 262144 --prefill-chunk 2048 --mtp \
    --ssd-streaming --ssd-streaming-cache-experts 6GB

Measured (2026-09-23, greedy, MTP + 64K draft vocab, server warm; peak = system-wide wired):

unc48L config	short prompt VI/EN/code	full context, prose answer	full context, counting (MTP-friendly)	peak wired	expert cache
K=36, 6 GB	43.5 / 41.9 / 43.6 t/s	—	37.8 @226K	52.6 GiB	capped at 0.22 GiB (over the wire limit)
K=32, 6 GB	42.9 / 41.6 / 43.0 t/s	34.1 @256K	43.8 @226K, 43.1 @256K	48.9–49.3 GiB	fully locked
K=32, 8 GB	+2% vs 6 GB	34.3 @256K	43.7 @226K	50.9–51.4 GiB	fully locked
K=30, 6 GB	≈ K=32	—	38.4 @256K	49.0 GiB	fully locked

The counting column (answer + "count from 1 to 400") is what MTP predicts best, so it overstates real use. Decode speed is nearly flat with context length: the same reasoning-heavy prose task (summarize a C file, thinking on) runs 34.4 / 35.1 / 36.0 / 34.1 t/s at 37K / 71K / 135K / 256K (unc31: 35.9 at 135K). Simple short chats run ~42 t/s. What changes the speed is the kind of answer (how well MTP predicts it), not the context length.

unc48L Trim runs with the same command; only -m changes, and it needs ds4-metal develop 331cf3b2 or later. Measured on 2026-10-03 against unc48L (M5 Pro 64 GiB, the command above):

• Output: byte-identical (greedy with MTP, and perplexity on three texts).
• Decode: +3.5–5.2% (8K prompt, page cache warm, both run orders).
• Expert cache: the 6 GB cache holds 2581 experts instead of 2278.
• Memory: the resident model is 2.5 GiB smaller, and steady wired memory is 1.5 GiB lower.
• Cold start: the first reply is 18% faster (7.8–8.1 s vs 9.5–9.9 s).

Use K=32. Prefill at 256K runs ~310 t/s (about 14 min for a full 256K prompt); unc31 prefills ~30% faster. Needle retrieval passed in every full-context run, swap stayed flat.

MTP draft vocabulary (optional, ~+4% decode)

The MTP draft only needs its argmax, so it can be scored over the most frequent output rows instead of all 248K. DS4_QWEN4_MTP_DRAFT_VOCAB=<file> gathers those rows into a small draft head (~178 MiB); the verify pass still uses the full head, so every committed token is still checked by the full model. The Qwen3.8 tokenizer puts Vietnamese at high token ids, so a plain id prefix (DS4_QWEN4_MTP_DRAFT_ROWS) collapses Vietnamese draft acceptance (71% -> 36%); this list ranks ids by measured frequency over Vietnamese and English Wikipedia, code and Python instead, and covers 100% / 99.4% / 97.0% of the model's own Vietnamese / English / code output.

Measured on unc31 (M5 Pro 64 GiB, 8K, greedy, 500 tokens, two runs each):

prompt	full head	this list
Vietnamese	35.5 t/s (71.0% accept)	36.7 t/s (71.0%)
English prose	35.3 t/s (68.3%)	36.9 t/s (68.3%)
code	36.1 t/s (72.7%)	37.5 t/s (70.3%)

DS4_QWEN4_MTP_DRAFT_VOCAB=Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt ds4 -m ... --mtp ...

Built with gguf-tools/frspec_vocab.py (ds4-metal fork, branch develop).

Runtime refusal projection (optional adapter)

adapters/refusal-4-44.f32 removes the refusal direction from the residual stream at run time: h -= s·(h·v)·v after each FFN output, one unit direction v per layer, on layers 4–44. No weights change, so it also works on the stock (non-abliterated) weights.

• Format: 48 × 2560 little-endian f32, zero outside layers 4–44.
• Source: the direction is Cudecnik/Qwen3.8-Flash-Next-refusal-projection (revision 1886570b, Qwen3.8-Flash-Next-refusal-projection.gguf), converted to ds4's per-layer f32 format by dir-steering/tools/cvec_to_f32.py --layers 4-44. adapters/refusal-4-44.f32.json holds the provenance and both sha256 sums.
• License: like the source, qwen-community-1.0 (LICENSE).

ds4-server -m <main>.gguf --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf ... \
    --dir-steering-file adapters/refusal-4-44.f32 --dir-steering-ffn 0.5

Measured with ds4 on the stock Ivan DS4-IQ2 weights, at scale 0.5 (frozen eval harness, 2026-09-29/30):

• Harmful refusals: 0/50 at scale 0.5. Higher scales also reach 0/50 but change the model more.
• ifeval: 194/200, against stock's 193/200.
• Cost: the only measured cost is on requests for actions the model cannot perform, where correct declines drop from 7/15 to 3/15. The abliterated weights in this repo score 2/15 there.

The source repo reports 1/50 harmful refusals at KL 0.186 for the full-strength projection in llama.cpp.

Runtime (DS4 Metal engine)

Runs on the ds4-metal engine (llama.cpp-lineage Metal fork for qwen4exp). The build used here adds, over the stock light runtime: the external --ple sidecar loader combined with native multi-session batched decode, the fast Q4_K dense GEMV, adaptive MTP draft depth, and correctness fixes (accurate GDN-gate softplus, an isfinite guard in the expert reduction, ignore_eos honored under --mtp, and a clearer tool-call parse error). The --ple sidecar is CPU-mmap demand-paged (each token faults ~one page per hash head); it fits 64 GiB beside the resident model, whereas the embedded-BF16 n-gram lineage does not.

Usage

# exact (default)
ds4 -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-MTP.gguf \
    --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf --mtp -c 4096 -p "..."

# faster (imatrix-weighted selective dense Q4_K)
ds4 -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-DenseQ4Kselimat-MTP.gguf \
    --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf --mtp -c 4096 -p "..."

# serving many concurrent requests (ds4-server, ~2x aggregate)
ds4-server -m ...DenseQ4Kselimat-MTP.gguf --ple ...PLE-Q4_1.gguf --mtp \
    --batched-session 16 -c 4096 --host 127.0.0.1 --port 8000

Long contexts: add --prefill-chunk 2048 to stay zero-swap at 128K–220K on 64 GiB.

Lineage / method

Uncensored weights from orcarouter/Qwen3.8-Flash-Next-Uncensored (rev 8336e613ea508b13c2159bd0f68965d97a606b95), quantized to Ivan's DS4-IQ2 layout, then repackaged into this light form by stripping the embedded n-gram tensor (retained tensors byte-identical; external Q4_1 sidecar substituted). The DenseQ4Kselimat main is derived from the Q8 main by imatrix-weighted Q4_K requant of the dense projections (attn q/k/v/output kept Q8_0).

Behavior — uncensoring is PARTIAL and tier-dependent

This is an abliterated fine-tune; the 2-bit light-quant dilutes most of the abliteration, so the base Q2K mains behave much closer to the original base model than to a full abliteration (they relax mild refusals but keep most harmful refusals). The SelQ4 tiers progressively restore the abliteration delta on the down experts (table above): unc31 is the first tier where the mid-band requests comply; the extreme categories (drug/weapon synthesis, IED) and break-in stay refused on every tier except unc48L (Complete), which only reaches them with a disclaimer. Treat every tier as a research artifact, not an unrestricted model. Provided for research; you are responsible for lawful use.