MERaLiON-2-3B-TurboQuant

TurboQuant KV cache compression for aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B.

This is a documentation repository that explains how to combine MERaLiON-2-3B's weights with TurboQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply TurboQuant via the Python package or llama.cpp fork.

Hardware compatibility

Device	VRAM / RAM	Recommendation
Any host that runs the base model	baseline + runtime savings	RotorQuant/TurboQuant is a KV-cache runtime modifier; pair with any weight variant

What is this?

KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.

Technique	Where it's applied	Savings
Weight quantization (GGUF/MLX/AWQ)	Baked into model file	Reduces disk + weight memory
TurboQuant KV cache	At inference time	Reduces attention memory (critical for long context)

Both can be combined for maximum efficiency.

Quickstart

Option A — Python / transformers

Install the turboquant package:

pip install turboquant

Then use it with the base model:

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoTokenizer
from turboquant import TurboQuantCache

tokenizer = AutoTokenizer.from_pretrained("aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B", trust_remote_code=True)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Apply TurboQuant to the KV cache
cache = TurboQuantCache(bits=4)  # or bits=2 for more aggressive compression

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    past_key_values=cache,
    use_cache=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Model Specifications

Property	Value
Base Model	aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B
Architecture	Whisper-large-v3 + Gemma-2-2B-IT decoder
Parameters	~3B (audio encoder + text decoder)
Context Length	8K
BF16 Size	~6 GB
Modalities	Audio + Text
License	other

What is TurboQuant?

TurboQuant (ICLR 2026) applies random orthogonal rotations followed by optimal scalar quantization to the KV cache. Bit-identical prefill logits at 4-bit, up to 4-8× memory savings for long sequences.

Benchmarks (from the TurboQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):

• 4-bit KV cache: bit-identical prefill logits
• ~1.4-1.7× speedup on Apple Silicon
• Up to 8× KV memory savings

Benchmarks are from the TurboQuant repository using Llama 3.1 8B. Performance on MERaLiON-2-3B will differ. Please open a discussion if you have independent results.

Current Ecosystem Support

Runtime	TurboQuant Support	Notes
Python transformers + turboquant	✅ Full	Drop-in cache class
llama.cpp upstream	❌ Not merged	Use fork below
llama-cpp-turboquant fork	✅ planar3, iso3	GitHub
LM Studio	❌ Requested	Use q8_0 as alternative
Ollama	❌ Not supported	Use OLLAMA_KV_CACHE_TYPE=q8_0
vLLM	❌ Not supported	—
koboldcpp	❌ Not supported	—

Pre-quantized weight variants

If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:

• MLX (Apple Silicon)
• GGUF (llama.cpp / Ollama / LM Studio)

See Also

• RotorQuant GitHub
• TurboQuant paper (arXiv 2504.19874)
• llama-cpp-turboquant fork
• Base model: aisingapore/MERaLiON-AudioLLM-Whisper-SEA-LION-V3-3B

Variants in this family

(Showing 8 sibling variants under majentik/meralion2-3b-*. The current variant — TurboQuant — is bolded.)

Variant	Runtime	Approx size	Use case
RotorQuant	runtime modifier	n/a	KV-cache root (weight-agnostic)
RotorQuant-MLX-2bit	mlx-lm	~983 MB	Apple Silicon, smallest
RotorQuant-MLX-4bit	mlx-lm	~1.9 GB	Apple Silicon balanced
RotorQuant-MLX-8bit	mlx-lm	~3.5 GB	Apple Silicon reference
TurboQuant	runtime modifier	n/a	KV-cache root (weight-agnostic)
TurboQuant-MLX-2bit	mlx-lm	~983 MB	Apple Silicon, smallest
TurboQuant-MLX-4bit	mlx-lm	~1.9 GB	Apple Silicon balanced
TurboQuant-MLX-8bit	mlx-lm	~3.5 GB	Apple Silicon reference