RedHatAI/Qwen3-30B-A3B-speculator.dflash

This is a DFlash speculator model for Qwen/Qwen3-30B-A3B.

Training Details

This model was trained using the Speculators library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k.Responses were regenerated by Qwen/Qwen3-235B-A22B and stored at Qwen3_235B_base.

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3-30B-A3B
  --data ./regenerated_data.jsonl \
  --output ./output \
  --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
  --seq-length 16384

Launch vLLM

# In (separate) virutal environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3-30B-A3B \
  --target-layer-ids 1 12 23 34 45 \
  --max-model-len  32768 \
  --max-num-batched-tokens 32768\
  --tensor-parallel-size 2 \
  --no-enable-chunked-prefill

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=2,3 torchrun \
  --standalone \
  --nproc_per_node 2 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-30B-A3B \     
  --data-path ./output \    
  --on-missing generate \    
  --on-generate delete \    
  --scheduler-type cosine \    
  --draft-vocab-size 32000 \    
  --max-anchors 1024 \    
  --target-layer-ids 1 12 23 34 45 \
  --speculator-type dflash \    
  --num-layers 5 \    
  --logger trackio  \    
  --lr 0.0006 \    
  --epochs 5 \    
  --sliding-window 2048 \    
  --sliding-window-indices 0 1 2 3 4 \    
  --draft-hidden-act silu 

Model Specifications

Base Model	Qwen/Qwen3-30B-A3B
Chat Template	Qwen/Qwen3-30B-A3B (use /chat/completions endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia A100

Deployment

# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git     
                                                                                                                                                                                                                                                                                                          
# Deploy with speculative decoding                                                                                                                                                                                                                                                                        
vllm serve Qwen/Qwen3-30B-A3B \                                                                                                                                                                                                                                                                                
    --tensor-parallel-size 2 \                                                                                                                                                                                                                                                                            
    --max-num-batched-tokens 32768 \
    --attention-backend FLASH_ATTN \ 
    --speculative-config '{                                                                                                                                                                                                                                                                               
        "model": "RedHatAI/Qwen3-30B-A3B-speculator.dflash",                                                                                                                                                                                                                                                   
        "num_speculative_tokens": 15,                                                                                                                                                                                                                                                                      
        "method": "dflash"                                                                                                                                                                                                                                                                                
    }'

Acceptance Rates

Per-position token acceptance rates across datasets:
(with reasoning enabled)

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	81.4%	58.5%	40.1%	26.9%	17.9%	11.7%	7.6%	3.44
math_reasoning	83.1%	62.9%	46.3%	33.7%	24.0%	16.6%	10.9%	3.77
qa	68.9%	41.3%	23.1%	12.8%	6.9%	3.6%	1.8%	2.58
question	73.4%	47.1%	29.3%	18.3%	11.5%	7.2%	4.4%	2.91
rag	73.5%	47.2%	28.7%	16.9%	9.5%	5.1%	2.6%	2.84
summarization	67.5%	38.9%	21.0%	10.7%	5.1%	2.2%	0.9%	2.46
tool_call	72.9%	46.5%	27.5%	15.9%	9.2%	5.3%	3.1%	2.80
translation	68.5%	43.2%	25.1%	13.3%	7.2%	3.9%	1.9%	2.63
writing	73.4%	47.1%	29.3%	18.3%	11.5%	7.2%	4.4%	2.91

Latency Speedup

Speedup comparisons of DFlash speculative decoding vs. baseline (no speculation) at varying request rates on Nvidia A100:

References

Paper: DFlash: Block Diffusion for Flash Speculative Decoding