File size: 13,571 Bytes
35d1eb0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32e1ea0
35d1eb0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7a81459
35d1eb0
13d1d69
 
 
 
 
 
 
35d1eb0
 
 
7a81459
35d1eb0
 
 
 
 
 
 
 
32e1ea0
35d1eb0
 
 
 
 
 
 
 
c27b664
35d1eb0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7a81459
35d1eb0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
---
pipeline_tag: text-generation
license: other
license_name: modified-mit
license_link: https://github.com/MiniMax-AI/MiniMax-M2.5/blob/main/LICENSE
library_name: llm-compressor
tags:
- fp8
- awq
- conversational
- vllm
- code
- devops
- software engineering
- engineer
- developer
- architect
- stem
- agent
datasets:
- HuggingFaceH4/ultrachat_200k
- databricks/databricks-dolly-15k
- neuralmagic/calibration
- HuggingFaceH4/no_robots
- nvidia/HelpSteer
- garage-bAInd/Open-Platypus
- PJMixers/grimulkan_physical-reasoning-ShareGPT
- PJMixers/grimulkan_theory-of-mind-ShareGPT
- HuggingFaceH4/Multilingual-Thinking
- ServiceNow-AI/M2Lingual
- droussis/euroblocks_sft_1sample_per_lang
- interstellarninja/hermes_reasoning_tool_use
- deepmind/code_contests
- dh02391735/stackoverflow-kubernetes-questions
- diversoailab/humaneval-rust
- ammarnasr/the-stack-rust-clean
- CSJianYang/CodeArena
- nvidia/OpenCodeInstruct
- nvidia/Llama-Nemotron-Post-Training-Dataset
- nvidia/Nemotron-Competitive-Programming-v1
- rombodawg/code_bagel_hermes-2.5
- MathArena/project_euler
- nvidia/Nemotron-Math-Proofs-v1
- nvidia/OpenMathInstruct-2
- nvidia/OpenScienceReasoning-2
- MegaScience/MegaScience
- OpenMed/Medical-Reasoning-SFT-GPT-OSS-120B
- ccdv/pubmed-summarization
- gbharti/finance-alpaca
- vladlen32230/summarization-yahoo-stock-finance-article-text
- fka/awesome-chatgpt-prompts
- theoldmandthesea/17k_business_book
- ruggsea/stanford-encyclopedia-of-philosophy_instruct
- mlfoundations-dev/stackexchange_philosophy
- FreedomIntelligence/SocraticChat
- Gryphe/Opus-WritingPrompts
- anthracite-org/nopm_claude_writing_fixed
- zerofata/Roleplay-Anime-Characters
- zerofata/Instruct-Anime
- zerofata/Instruct-Anime-CreativeWriting
- sam-paech/gutenberg3-generalfiction-scifi-fantasy-romance-adventure-dpo
- PocketDoc/Dans-Prosemaxx-Adventure
- anthracite-org/stheno-filtered-v1.1
- KaraKaraWitch/TvTroper-2025
- AquaV/US-Army-Survival-Sharegpt
- AquaV/Interrogation-Sharegpt
- AquaV/Multi-Environment-Operations-Sharegpt
- AquaV/Resistance-Sharegpt
- PocketDoc/Dans-Kinomaxx-VanillaBackrooms
base_model:
- MiniMaxAI/MiniMax-M2.5
---

# MiniMax M2.5 (Mixed-Precision BF16 + INT4 AWQ)

## Changelog

- 2026-02-15: Requant to ensure quality - `batch_size=1` ([thread](https://huggingface.co/mratsim/MiniMax-M2.5-BF16-INT4-AWQ/discussions/4)) and addition of Greek language and 34 multilingual dataset ([request](https://huggingface.co/mratsim/MiniMax-M2.1-FP8-INT4-AWQ/discussions/10#698f6b1dfbcffe20d0265976))
- 2026-02-14: Original quant, using LLMcompressor's new feature `batch_size=32`. `batch_size` may negatively impact calibration due to truncating or padding datasets, defeating the careful selection I made.

## Overview

This strives to be the highest quality quant that can run on 192GiB VRAM

> [!TIP]  
> 💡This is a sister model to [mratsim/MiniMax-M2.5-FP8-INT4-AWQ](https://huggingface.co/mratsim/MiniMax-M2.5-FP8-INT4-AWQ)
> with the original model FP8 weights pre-dequantized to BF16.
>
> This makes it compatible with 8x3090 systems (which don't have hardware FP8)
> and also compatible with SGLang for an extra 3 GiB in VRAM.

It features:
- That model has ensured that all experts are calibrated, not doing so is extremely detrimental, PR: https://github.com/vllm-project/llm-compressor/pull/2171
  <details>
  <summary>💡**[Click me!]** Visual showcase of why ensuring quantization of all MoE experts is important</summary>
  
  - Source: https://avtc.github.io/aquarium-side-by-side/
  - Context: https://github.com/ModelCloud/GPTQModel/pull/2235
  
  ![image](https://cdn-uploads.huggingface.co/production/uploads/67f26fd2c7b14380431d1f5a/BDc3-0m3_WLl3ZmbBMhmd.png)
  
  </details>
- Mixed precision with:
  - self-attention weights dequantized from the official version.
  - experts weights quantized using AWQ W4A16G32 scheme (4-bit weights, 16-bit activations, scaling factor per group of 32 weights)
- High-quality large and diverse dataset with programming and devops focus
  as well as domain-specific knowledge (math, sciences, medical, finance, business, humanities, philosophy, creative writing), general knowledge, pop culture and behavioral situations because we never code in a vacuum. And we want to make sure all experts are calibrated to the full range of their activations.
- Calibration explicitly tests multilingual capabilities:
  - Asia: Chinese, Hindi, Korean, Japanese
  - Europe: French, German, Portuguese, Russian, Spanish
  - Middle-East: Arabic, Hebrew, Turkish
- Calibration explicitly tests 60 programming languages and not just Python:
  - Imperative programming: C, C++, Go, Zig, ...
  - Functional programming: Haskell, F#, OCaml, Erlang, Lisp, Clojure ...
  - Web-focused: HTML/CSS, Typescript, PHP, ...
  - Mixed paradigm: D, Kotlin, Nim, Rust, Swift, ...
  - Theorem provers: Coq, Lean
  - Low-level: ARM64 assembly, x86-64 assembly, LLVM IR
  - GPU Programming: Cuda, Vulkan, Apple Metal
  - Game Programming: GDScript, GLSL
  - Domain-specific: MATLAB, Julia, Solidity, R
- Calibration tries to ensure coverage for a wide variety of experience (from explaining concepts to your grandmother to debugging Kubernetes logs)
- Built by a dev, for devs (and it looks very good for STEM as well)

It uses my new declarative quantization framework https://github.com/mratsim/quantizers which facilitates highly-tuned calibration sets: [calibrate_software_engineer.yaml](./calibrate_software_engineer.yaml)

<details>
<summary>This has taken several days and contribution and bug reports to the ecosystem, I hope you find it useful.</summary>
  
- https://github.com/vllm-project/llm-compressor/pull/2171
- https://github.com/vllm-project/llm-compressor/issues/2172
- https://github.com/vllm-project/vllm/issues/31623
- https://github.com/sgl-project/sglang/issues/16276
- https://github.com/sgl-project/sglang/issues/16295

</details>

## 📥 Usage & Running Instructions

The model was tested with SGLang + 2x RTX Pro 6000, here is a script suitable for such configuration with the maximum 196,608 context length. This uses 92.5GiB of VRAM with the flashinfer backend.

Please refer to [mratsim/MiniMax-M2.5-FP8-INT4-AWQ#running-script](https://huggingface.co/mratsim/MiniMax-M2.5-FP8-INT4-AWQ#running-script)
for running it in vLLM

### Running script

`--trust-remote-code` is necessary until the transformers team merges github.com/huggingface/transformers/pull/42028

You have 2 reasoning parsers;
- `minimax`, puts the reasoning content in a special field like DeepSeek models that is usually rendered in a specific manner in frontends.
- `minimax_append_think`, puts the reasoning into `<think>reasoning_content</think>` and that is sent as normal text. Few frontends properly render that, I'm aware of [Cherry Studio](https://github.com/CherryHQ/cherry-studio) on Desktop and [ChatterUI](https://github.com/Vali-98/ChatterUI) on Android.

The reason why `minimax_append_think` was introduced was Interleaved Thinking and having the model build upon it's previous thinking (usually frontends discard the thinking trace)

> [!TIP]  
> 💡In the sister model, I mentioned that with the recommended parameters the model tends to get stuck in repetition loops.\
> This does not seem to happen with SGLang hence "repetition_penalty: 1.10, frequency_penalty: 0.40" are not used. \
> There is no way to override such settings without editing generation_config.json anyway: https://github.com/sgl-project/sglang/issues/15487

```bash
# Model configuration (Mandatory)
MODEL="mratsim/MiniMax-M2.5-BF16-INT4-AWQ"
MODELNAME="MiniMax-M2.5"
GPU_UTIL=0.93
SGLANG_PORT=8000

# Prevent memory fragmentation
export PYTORCH_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512

python3 -m sglang.launch_server \
    --host 0.0.0.0 \
    --port "${SGLANG_PORT}" \
    --sleep-on-idle \
    --disable-custom-all-reduce \
    --max-running-requests 64 \
    --cuda-graph-max-bs 64 \
    --attention-backend flashinfer \
    --served-model-name "${MODELNAME}" \
    --model-path "${MODEL}" \
    --tool-call-parser minimax-m2 \
    --reasoning-parser minimax \
    --trust-remote-code \
    --tp 2 \
    --mem-fraction-static ${GPU_UTIL} \
    "$@"

    # --reasoning-parser minimax-append-think
```

## 🔬 Quantization method

The llmcompressor library was used with the following recipe:

```yaml
default_stage:
  default_modifiers:
    AWQModifier:
      config_groups:
        mlp_experts_projections:
          # Include only MLP expert weights for 4-bit quantization
          targets: ["re:.*block_sparse_moe\\.experts\\.\\d+\\.(w1|w2|w3)$"]
          weights:
            num_bits: 4
            type: int
            symmetric: true
            group_size: 32
            strategy: group
            dynamic: false
            # actorder: group
            observer: memoryless_minmax

      mappings:
        - smooth_layer: re:.*post_attention_layernorm$
          balance_layers: ["re:.*w1$", "re:.*w3$"]
        - smooth_layer: re:.*w3$
          balance_layers: ["re:.*w2$"]
      duo_scaling: true
```

The calibration set had 590 examples, 8192 sequence length, 60 programming languages, 12 spoken languages and is detailed at [calibrate_software_engineer.yaml](./calibrate_software_engineer.yaml)

## Quantization theory and heuristics for manual tuning

<details>
<summary>In-depth overview of quantization theory and heuristics for manual tuning</summary>

### Layers to quantize

Quantization should be focused on Linear layers (also called Dense or Fully-Connected layers i.e. MatMul+Bias)
In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1]
> LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the
> LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression.
> Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized.

This is also reported in Intel and Nvidia repo:
- https://github.com/intel/neural-compressor/issues/1963#issuecomment-2274873441
- https://github.com/NVIDIA/TensorRT/issues/4084#issuecomment-2294513950

### Tensors to up-quantize

If there is enough bits, down projections should be prioritized.

According to [4]
> Fig. 3: Maximum absolute value over layers for a LLaMA3-8B.
> Each color represent a different projection and we clearly see that down_proj has the biggest
> spikes in input and output. We also observe that RMSNorm propagate spikes through the entire model

According to [5]
> Figure 5(a) illustrates the extremal ratio across layers and modules in LLaMA2-7B, highlighting
> that weight outliers are concentrated in the down-projection matrices Wdown
> ℓ of the second layer and
> the last two layers. Figures 5(b) and 5(c) provide detailed visualizations of these outliers in the last
> two layers.

### Mixture-of-Experts quantization (MoE)

Mixture-of-Experts require specific quantization techniques.

#### Mixed-precision quantization

Some layers have a higher impact on LLM performance.
According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers.
According to [3] on 2-bit quantization:
- quantizing expert FFN layers do not seriously impact model quality
- quantizing cross-attention has some impact
- quantizing self-attention has a large impact
- quantizing dense FFN has a very significant impact

Hence to preserve model quality we should choose not to quantize dense FFN layers and self-attention layers.

We notice that:
- official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16:
  - https://huggingface.co/openai/gpt-oss-120b/blob/main/model.safetensors.index.json
- NVFP4 weights of DeepSeek-R1 quantized by Nvidia also keep self-attention in BF16:
  - https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4/blob/main/model.safetensors.index.json

#### Layers with high-impact

According to [2], giving more bits to the first `k` blocks have a significantly higher impact on model quality than for the same last `k` blocks.

#### Expert quantization

When quantizing MoE, quantizing activations is tricky as only a subset of experts are activated per request. You have to make sure all experts are calibrated.

<details>
<summary>Visual showcase of why ensuring quantization of all MoE experts is important</summary>

- Source: https://avtc.github.io/aquarium-side-by-side/
- Context: https://github.com/ModelCloud/GPTQModel/pull/2235

![image](https://cdn-uploads.huggingface.co/production/uploads/67f26fd2c7b14380431d1f5a/BDc3-0m3_WLl3ZmbBMhmd.png)

</details>

## References

1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)\
  Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia\
  https://arxiv.org/pdf/2506.12044

2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)\
  Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen\
  https://arxiv.org/pdf/2406.08155v1

3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)\
  Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla\
  https://arxiv.org/pdf/2310.02410

4. Precision Where It Matters: A Novel Spike\
   Aware Mixed-Precision Quantization Strategy for\
   LLaMA-based Language Models (2025)\
   Lucas Maisonnave, Cyril Moineau, Olivier Bichler, and Fabrice Rastello\
   https://arxiv.org/pdf/2504.21553

5. Systematic Outliers in Large Language Models (2025)\
   Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang\
   https://arxiv.org/pdf/2502.06415v2

</details>