Kimi-K3-pruned50

This is a calibration-free, safetensors-only expert-pruned checkpoint derived from moonshotai/Kimi-K3. It is intended to unblock vLLM serving and evaluation smoke tests while accurate calibration-based pruning, such as REAP support in llm-compressor, is under development.

This is an experimental prototype, not a REAP-pruned checkpoint. Expect a substantial quality regression relative to the source model.

Pruning

• Source experts per MoE layer: 896
• Retained experts per MoE layer: 448 (50%)
• MoE layers: 92
• Ranking: descending signed sum(router.weight[i]), independently per layer
• Removed data: pruned expert tensors and their corresponding router rows
• Router and expert indices were compacted, and the config was updated to 448 experts
• Calibration data: none
• Retained expert count is divisible by TP=8 (56 experts per rank)

pruning_manifest.json records the source hashes, exact per-layer expert remap, pruning method, and output statistics.

Validated vLLM run

The checkpoint loaded and served successfully with the default vLLM loader (no --load-format) on two nodes with four NVIDIA GB300 GPUs per node:

• vLLM: 0.26.1rc1.dev142+g30b4e7f47
• vLLM commit: 30b4e7f479674a9c4d8889d4857294d3bd5e6849
• FlashInfer: 0.6.16rc5
• Hardware: 8x NVIDIA GB300, 284208 MiB each
• Parallelism: TP=8 across 2 nodes
• Model memory: approximately 108.02 GiB per GPU
• Available KV cache after load: approximately 149.8 GiB per GPU

Run the following on both nodes, setting NODE_RANK=0 on the head node and NODE_RANK=1 on the worker. Add --headless on the worker:

vllm serve mgoin/Kimi-K3-pruned50 \
  --served-model-name kimi-k3-pruned-50pct \
  --host 0.0.0.0 \
  --port 18080 \
  --tensor-parallel-size 8 \
  --distributed-executor-backend mp \
  --nnodes 2 \
  --node-rank "${NODE_RANK}" \
  --master-addr "${MASTER_ADDR}" \
  --master-port 29501 \
  --trust-remote-code \
  --moe-backend auto \
  --gpu-memory-utilization 0.95 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --attention-backend FLASHINFER_MLA \
  --attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --max-cudagraph-capture-size 8

GSM8K

The evaluation used vLLM's standalone GSM8K harness against the /v1/completions endpoint with the full test split:

.venv/bin/python tests/evals/gsm8k/gsm8k_eval.py \
  --host http://${HEAD_NODE} \
  --port 18080 \
  --num-questions 1319 \
  --num-shots 5 \
  --max-tokens 256 \
  --temperature 0 \
  --seed 42

Checkpoint	Accuracy	Correct	Invalid	Latency	Questions/s
Kimi-K3-pruned50	68.2335%	900/1319	3/1319	481.863 s	2.737
Source Kimi-K3	95.5269%	1260/1319	1/1319	530.362 s	2.487

The pruned checkpoint regressed by 27.2934 percentage points on this calibration-free smoke test. The two runs used the same 5-shot prompt builder, 256-token limit, temperature, seed, vLLM checkout, and hardware.

License

This derivative is distributed under the upstream Kimi K3 License. See LICENSE in this repository.