aj9o9/GLM-5.3-Flash-GGUF

GLM-5.3-Flash, hand-mixed GGUF

Two quants, 87 GB and 112 GB, built by assigning every tensor class by hand instead of using a preset.

build	size	bpw	shards	runs on
AJ-IQ3_XXS	112.40 GB	2.80	3	128 GB unified memory, or a 96 GB card with a small spill
AJ-IQ2_XXS	87.35 GB	2.23	2	a 24 GB card with 64 GB of system RAM

The model

GLM-5.3-Flash is 320.6B total parameters with 17.3B active. Three things about its architecture drive every decision in these quants.

It is 94.9% routed experts. 304.4B of the 320.6B parameters sit in ffn_*_exps tensors across 42 sparse layers, 288 experts each, 8 active per token plus 1 shared. Everything else in the model, all the attention and embeddings and norms, is 8.9B parameters. That means the file size is set almost entirely by one decision: how many bits the experts get. Everything else can be kept generous for almost nothing.

Attention is hybrid, and mostly not attention. Of 45 layers, 34 use Kimi Delta Attention, a linear-attention variant that carries a fixed-size recurrent state rather than a growing KV cache. The other 11 use Multi-head Latent Attention with a DeepSeek sparse attention indexer, and MLA caches only a 512-wide latent. The practical result is that context is unusually cheap: 32K costs 0.44 GB, 131K costs 1.55 GB, 300K costs 3.45 GB at f16.

The KDA layers name their projections like normal attention. They use attn_q, attn_k, attn_v, not ssm_* as the recurrent naming might suggest. Getting this wrong is silent: unmatched tensors fall through to the base type with no error. An early version of this mix missed them and 3.42B parameters quietly dropped to 2 bits.

There is also one MTP/NextN block. Both builds drop it with --prune-layers 45, so the parameter count in these files is 313.33B rather than 320.6B, and speculative decoding through the MTP head is not available.

The two quants and what actually differs

The mix files are identical except for their routed-expert rules. Verified by md5 over everything else. So the entire difference in quality between these two files is attributable to the experts alone, with no other variable moving.

tensor class	AJ-IQ2_XXS	AJ-IQ3_XXS
norms, router, hc_	F32	F32
KDA gates and decay, DSA indexer	Q8_0	Q8_0
all attention, embeddings, lm_head, dense FFN, shared expert	Q6_K	Q6_K
routed experts, gate and up	IQ2_XXS	IQ2_S
routed experts, down	IQ2_XXS (IQ2_S on edge layers)	IQ3_XXS (IQ3_S on edge layers)

That is the whole delta. 25 GB, spent entirely on 304.4B expert weights.

Why down gets the higher tier. Gate and up together are two thirds of every expert layer's weights, so they set the price. Down is the remaining third and is the more sensitive of the three. Raising down by one tier costs 12.7 GB; raising gate and up costs 25.4 GB. Buying quality through down first is simply cheaper per unit of improvement, and the measurements below bear that out.

Quality

How this was measured, in full. Wikitext-2 wiki.test.raw, 30 chunks, n_ctx=512, llama-perplexity -fa on -b 512 -ub 512, llama.cpp PR #27752 at commit 24652c57a, on an RTX PRO 6000 Blackwell. The KLD base is BF16 logits generated in the same session, on the same build, with the same flags, from unsloth/GLM-5.3-Flash-GGUF BF16. All three rows are one build, one corpus, one chunk count. Only quantization differs.

model	size	bpw	PPL	vs BF16	KLD	RMS Δp	Same top-p
BF16	642 GB	16	2.2474 ± 0.0452	base	0 (base)	n/a	n/a
AJ-IQ3_XXS	112.40 GB	2.80	3.0147 ± 0.0687	1.341x	0.3557 ± 0.0089	23.80%	81.77%
AJ-IQ2_XXS	87.35 GB	2.23	4.2314 ± 0.1070	1.883x	0.7072 ± 0.0145	34.16%	73.20%

• The extra 25 GB halves the divergence. KLD 0.7072 to 0.3557, a factor of 1.99. Same-top-p gains 8.56 points. The PPL penalty falls from 1.88x to 1.34x. Error bars do not overlap on any column.
• The cost is 0.0140 KLD per extra GB. That is the number to beat before baking a further tier.
• 30 chunks, not 580. Wider error bars than a full run. BF16 costs 136 s per pass even on a 96 GB card, because 642 GB does not fit in a 367 GB container and streams from disk every pass. The quants run at about 1 s per pass, so the chunk count is a BF16 budget decision, not a methodology preference.

Why every run here uses -b 512

Every number above uses -b 512 -ub 512. The llama-perplexity default of -b 2048 gives n_seq=4, and that path is not reliable on this architecture. It returns nan from chunk 2 on quantized models. On BF16 it returns finite but divergent values:

chunk	BF16 at n_seq=4	BF16 at n_seq=1
1	1.9119	1.9139
2	7.2195	2.8699
3	6.2120	2.2662
4	4.8322	1.9707

Chunk 1 agrees to four significant figures across two different builds. Chunk 2 and everything after diverges by up to 2.5x, at exactly the chunk where quantized models produce nan. The multi-sequence path was corrupting results silently.

Earlier versions of this model card carried numbers from that path. They were wrong and have been replaced. Any perplexity figure for this architecture published without -b 512 should be treated with suspicion.

Against unsloth

unsloth/GLM-5.3-Flash-GGUF publishes top-1 accuracy and mean KLD. Those are Same top p and Mean KLD from the same llama-perplexity --kl-divergence tool, so they are the same quantities. The corpus and the base logit file differ, so this is indicative, not matched. A properly matched comparison would need their GGUFs scored against the same base, which has not been done.

quant	size	top-1	KLD
UD-IQ1_S	93.09 GB	70.89%	0.6697
UD-IQ1_M	97.58 GB	73.06%	0.5724
AJ-IQ2_XXS	87.35 GB	73.20%	0.7072
UD-IQ2_XXS	101.84 GB	76.30%	0.4501
UD-Q2_K_XL	108.72 GB	78.34%	0.3801
AJ-IQ3_XXS	112.40 GB	81.77%	0.3557
UD-IQ3_XXS	120.37 GB	81.63%	0.2838
UD-Q3_K_XL	147.54 GB	86.25%	0.1597

• AJ-IQ3_XXS matches UD-IQ3_XXS on top-1 while being 6.6% smaller, 81.77% against 81.63% at 112.40 GB against 120.37 GB. On a 128 GB machine that is 15.6 GB free instead of 7.6 GB, which decides whether the file runs with real context or only just loads.
• AJ-IQ2_XXS matches UD-IQ1_M on top-1 while being 10.5% smaller, 73.20% against 73.06% at 87.35 GB against 97.58 GB.
• UD wins on KLD at both sizes, 0.2838 against 0.3557 and 0.5724 against 0.7072. Same argmax agreement, closer full distribution.
• What that trade means in practice. Keeping routers at F32 and attention at Q6_K preserves which expert fires and which token wins. Pushing gate and up down to 2 bits coarsens the magnitudes behind that choice, so error lands in the tail rather than in the top-1 pick. These builds suit greedy and low-temperature decoding. For sampling at higher temperature, the unsloth builds at comparable size are the better choice.

Speed

llama-bench, -ctk q8_0 -ctv q8_0 -fa on, nothing else running.

AJ-IQ2_XXS, RTX PRO 6000 Blackwell 96 GB, fully resident

test	t/s
tg256	69.89 ± 0.15
tg256 @ d8192	52.74 ± 0.35
tg256 @ d16384	44.30 ± 0.15
tg256 @ d32768	31.70 ± 0.38
tg256 @ d65536	20.91 ± 0.05
tg256 @ d131072	12.24 ± 0.00

AJ-IQ2_XXS, single RTX 3090 24 GB + 64 GB DDR5, -ncmoe 40

test	t/s
pp2048	36.69 ± 6.23
tg128	11.55 ± 0.68

• Where the experts live is the only variable that matters. The same 87 GB file gives 69.89 tok/s resident on a 96 GB card and 11.55 tok/s with 40 of 42 expert layers on CPU. A 6x gap, entirely PCIe streaming.
• Depth roughly halves the rate every 32K. 69.89 at zero, 31.70 at 32K, 12.24 at 131K. Budget for that if you plan long-context work.
• AJ-IQ3_XXS is not benchmarked. At 104.68 GiB against 95.6 GiB of card it cannot go fully resident and needs a small spill, so it would not be comparable to the rows above anyway. Expect it to land between the two tables.

Run

Everything on GPU, 96 GB or more:

llama-server -m GLM-5.3-Flash-AJ-IQ2_XXS-00001-of-00002.gguf \
  -ngl 99 -fa on -c 131072 -ctk q8_0 -ctv q8_0 --jinja

AJ-IQ3_XXS on a 96 GB card, small spill:

llama-server -m GLM-5.3-Flash-AJ-IQ3_XXS-00001-of-00003.gguf \
  -ngl 99 -ncmoe 14 -fa on -c 131072 -ctk q8_0 -ctv q8_0 --jinja

24 GB card with system RAM, all experts on CPU:

llama-server -m GLM-5.3-Flash-AJ-IQ2_XXS-00001-of-00002.gguf \
  -ngl 99 -cmoe -fa on -c 32768 -ctk q8_0 -ctv q8_0 --jinja

Point at shard 1 and llama.cpp finds the rest. Prefer -ncmoe N over -cmoe whenever there is VRAM to spare; every layer kept on the card is a large win.

Use -np 1 unless you genuinely have concurrent requests. -np N divides -c across N slots, so -c 131072 -np 4 gives each request only 32,768 tokens.

Build requirement

Needs llama.cpp from PR #27752, which is unmerged. Specifically the commit llama : skip foreign-sequence cells when building the glm5next k-pools or later. Earlier commits return nan on the CPU expert path. The PR gets rebased, so pin by that commit message rather than by hash.

The mix, and why it is ordered

--tensor-type-file is first regex match wins, using substring search. A general rule placed too early silently swallows the ones after it. Order:

class	type	reason
norms, hc_, ffn_gate_inp, exp_probs_b	F32	the router picks 8 of 288; a wrong pick costs far more than any weight error
KDA gates and decay	Q8_0	recurrent state, error accumulates along the sequence
DSA indexer	Q8_0	must precede the attention rules, or indexer.attn_k.weight is caught by attn_k\.weight
attention, embeddings, lm_head, dense FFN, shared expert	Q6_K	8.9B params total, cheap to keep rich
routed experts	IQ2_XXS or IQ2_S / IQ3_XXS	the entire budget

Everything above the experts is 2.84% of the model and 7.5 GB. Raising all of it from Q6_K to Q8_0 costs 2.09 GB, which spent on experts instead would buy only 0.055 bpw across 304B weights. That is why it stays where it is, and why both builds share it unchanged.

Files

• AJ-IQ3_XXS/GLM-5.3-Flash-AJ-IQ3_XXS-0000{1,2,3}-of-00003.gguf
• AJ-IQ2_XXS/GLM-5.3-Flash-AJ-IQ2_XXS-0000{1,2}-of-00002.gguf
• mixes: glm53-flash-iq3xxs.txt, glm53-flash-iq2xxs.txt
• base: zai-org/GLM-5.3-Flash, MIT
• BF16 GGUF and imatrix: unsloth/GLM-5.3-Flash-GGUF. Using theirs saved a 328 GB FP8 download and an imatrix pass that is not practical at this scale.