Qwen

/

Qwen3.8-Flash-Next

Like 5.54k

Follow

Qwen 106k

Sharing our SGLang serving config and measured numbers for the NVFP4 build of this model on two Blackwell (SM120) cards. Everything below is measured on our live engine, not copied from release notes.

Checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4 (local copy), NVFP4 W4A4, model_type=qwen4_exp, TP=2, 262,144 ctx.
Hardware: 2× NVIDIA RTX PRO 6000 Blackwell (97,887 MiB each, CC 12.0), driver 595.84, Threadripper PRO 5975WX, 251 GiB RAM.
Engine: SGLang build carrying the Flash-Next support work (squash commit 73a2552, reported as 0.0.0.dev1+g73a255206) plus a local SM120 QSA/XQA patch in qwen_sparse_attn_backend.py, with FP8 KV cache from PR #36644 applied in a separate worktree pulled in via PYTHONPATH.

Effective launch command

Read from /proc/<pid>/cmdline on the running process, not from our script:

export SGLANG_ENABLE_SPEC_V2=True
python -m sglang.launch_server \
  --model-path RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --served-model-name qwen38-flash-next \
  --tp 2 \
  --quantization modelopt_fp4 \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_cutlass \
  --linear-attn-prefill-backend flashinfer \
  --ple-offload-embedding \
  --reasoning-parser auto --tool-call-parser auto --trust-remote-code \
  --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --chunked-prefill-size 4096 \
  --context-length 262144 --allow-auto-truncate \
  --enable-cache-report --enable-metrics \
  --kv-cache-dtype fp8_e4m3 \
  --enable-linear-replayssm-spec \
  --mem-fraction-static 0.88 \
  --max-running-requests 45 \
  --host 0.0.0.0 --port 8001

Two flags are load-bearing and easy to get wrong:

• SGLANG_ENABLE_SPEC_V2=True is mandatory. Without it SGLang converts NEXTN to EAGLE and loads a second full copy of the model as the draft — OOM.
• The last duplicate wins. Our unit passes --mem-fraction-static and --max-running-requests twice (base script 0.80/10, a systemd drop-in appending 0.88/45). Argparse keeps the last value, so 45 wins today — but a drop-in reorder silently returns the engine to admission 10, which is the old throughput choke. Deduplicate if you copy this layout.

Throughput (output tok/s, ISL 1024 / OSL 512, quiet engine)

* contaminated — another tenant arrived mid-run; the CuTe row's 886 is the honest read of that point.

The earlier admission step 10 → 32 was the bigger one:

At admission 10, conc 64 is slower than conc 16 — the queue is holding KV. Raising admission removes it. Admission, not memory, was the binding constraint.

Judge these changes at concurrency 1 first. That is where agent workloads live; one config (adaptive spec) looked fine at c64 while being 33 % worse single-stream.

Capacity

FP8 E4M3 KV nearly doubled the pool for free. ReplaySSM is neither a latency nor a KV win on its own — it removes the spec-decode intermediate mamba buffer (which scales with admission, ~72 MB per slot), and that is what makes high admission affordable. The two changes must stay together.

What the quantization cost in quality

• GSM8K, 200 questions, temperature 0, identical probe on both configs: BF16 KV 0.9650 (193/200) vs FP8 KV 0.9550 (191/200). A 2-question gap against a ±2-question standard error — no measurable loss.
• Long-context recall: 16 cold runs (unique salt per run, so nothing is served from the radix cache) at 115k, 144k and 201k prompt tokens, three needles each, temperature 0 — 48 of 48 needles recalled, and every temperature-0 repeat came back byte-identical.
• Why it can be free: this checkpoint ships no KV calibration, so the descale factor falls back to 1.0 and E4M3 is used as a plain 3-mantissa-bit store. Measured, not assumed.
• Spec decoding stayed alive throughout: live spec_accept_length 2.2–3.6. Read it correctly — it counts the bonus token, so strict per-draft-token acceptance is (accept_length − 1) / 3; at 2.56 that is 52 %, matching the live spec_accept_rate of 0.519.

Rejected on measurement

• --fp4-gemm-backend flashinfer_cutedsl — inside noise on SM120, single-stream median ITL 3.05–3.08 ms either way. Not worth an unproven backend.
• --speculative-adaptive — single-stream 211 → 141 tok/s. The journal shows why: Adaptive spec params updated: steps 3 -> 1 (ema_accept_len=0.14) while the engine's actual accept length was 2.5–3, then it oscillated back, and it disables index_share_for_mtp_iteration. Its cold long-context runs were clean, so this is a speed failure, not a correctness one.
• DSpark — the genuinely faster draft path upstream, but it exists for other models, not Flash-Next, and brings its own weights and KV. Not feasible with ~126 GB of weights on two cards.

Operating note

Admission 45 is a server ceiling, not a safe agent fan-out target. One coding agent carries ~90–120k tokens of context, so a handful of concurrent agents already occupy most of the 2.66M pool. Above roughly 16 concurrent the queue forms, KV fills, prefix-cache hits collapse from ~77 % to ~5 %, every turn re-prefills, and per-agent speed has been measured falling from ~190 tok/s to 78. Six to ten live agents is the sweet spot on an otherwise quiet engine.

Total single-stream cost of the whole stack: median TPOT 4.18–4.21 ms BF16 vs 4.35–4.55 ms with FP8 KV + ReplaySSM + higher admission. About 4 %, in exchange for 2.3× throughput under load.

Happy to answer questions or run a config others are curious about — the bench harness is just sglang.bench_serving with load sampled during the run, which caught us out twice when we forgot to do it.