SERVING_FUSED.md

Serving GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 on the fused MoE path

Reference numbers below were measured on 4x RTX PRO 6000 Blackwell (SM120a, 96 GB each, PCIe Gen5, no NVLink), TP4 + DCP4, MTP 3.

Docker image

verdictai/glm52-exl3-sparkinfer:v39-r28-r7fused-broadcast-cu132-sm120a

docker pull verdictai/glm52-exl3-sparkinfer:v39-r28-r7fused-broadcast-cu132-sm120a

Built for sm_120a (Blackwell). It will not run on other architectures without a rebuild. It already contains every patch in patches/ -- that directory is only needed if you are building your own image.

Measured

	value	configuration
KV cache	355,328 tokens	max_model_len 262144, util 0.955, batched 2048, seqs 4, nvfp4_ds_mla, bf16 rope, DCP4
Max concurrency	1.36x	at 262,144 tokens per request
KLD vs reference logits	0.069527	5-run gate, nvfp4_ds_mla + outer scales, bf16 rope, fused path
Prefill @ 8K	1,725 tok/s	8,039-token prompt, unique prefix (prefix cache defeated)
Decode	75.6 tok/s	256 tokens, ignore_eos, single stream

Non-fused reference on the identical rig: 497,408 tokens, 1,757 tok/s prefill, 73.5 tok/s decode. The fused path trades KV capacity for decode throughput.

The outer scales file is not optional

nvfp4_mla_outer_scales.json ships in this repo. It must be mounted and pointed at by VLLM_NVFP4_MLA_SCALES_FILE. Measured on the identical build:

nvfp4 KV	with outer scales	without
bf16 rope	0.069527	0.099717
fp8 rope	0.075542	0.114359

Omitting it costs ~30% KLD for zero memory and zero throughput benefit. It is a per-layer outer-scale calibration (wikitext-2-raw-v1, 2048 context).

Settings that matter, and why

setting	value	rationale
VLLM_EXL3_R7_FUSED	1	SparkInfer fused MoE. Requires the FC2-bound fix below.
VLLM_EXL3_R7_FUSED_LAYERS	48	Loader default. 6 layers carry a third trellis K value and correctly fall back to exl3_moe_r7_fused.
VLLM_NVFP4_MLA_SCALES_FILE	(path)	See above. Largest single quality lever.
KV_FP8_ROPE	0	fp8 rope yields ~16% more KV tokens but measured +8.7% KLD (0.075542 vs 0.069527). Not worth it for citation-sensitive work.
VLLM_DCP_INDEXER_SHARDS	4	Auto-selection picks 2 under DCP4, holding each DSA index-cache entry on two ranks. 4 is pure placement -- identical arithmetic, +8.5% KV.
VLLM_EXL3_PREFILL_CAPACITY	1024	Halves the 574 MiB fused prefill arena. 512 and 256 give more KV but cost 17% and 34% of prefill; 1024 is the knee (1,725 vs 1,739 tok/s at the full 2048).
VLLM_EXL3_PREFILL_BLOCK_M	64	Route block size.

If you build your own image

The fused path needs patches that are not upstream. They are in patches/ in this repo; each is idempotent, uses exact anchors and fails closed.

Required for correctness:

• patch_sparkinfer_projection_tiers.py — the FC2 dispatch bound. SparkInfer bounded the FC2 (down-projection) tier-local expert index with the FC1 slot count. With per-(expert,projection) trellis bit widths, FC2 holds more experts than FC1 (tier1: 231 down vs 77 gate/up), so most down tiles were rejected and their output silently dropped — 54.14% of down-projection expert slots across the 48 fused layers. Because zero_fc2_output=False and rotation_gate is aliased as the FC2 buffer, the rejected tiles left gate-rotated hidden states in place, which were then rotated, scaled and accumulated: fluent but wrong output, measured at KLD 2.36.

Required for the memory numbers above:

• patch_r7_broadcast_rotations.py — requires an r28-or-later SparkInfer base. R7 stores one shared rotation row per layer (r7_shared.gate_up_suh, r7_shared.down_svh, both [6144]). Earlier bases lacked broadcast rows, so the loader expanded each to 256 identical copies. r28's ABI-v6 broadcast_suh/broadcast_svh takes the single row. ~0.5 GiB/rank.
• patch_r7_ballast_dealias.py — the loader handed prepare a view into the never-read ballast pool, which every frozen tier then retained (192 live aliases per rank), pinning ~576 MiB. Hands it a standalone scalar instead.
• patch_r7_release_ballast.py — releases the pool once, after all layers load. Must not be done per layer: that refragments the heap and OOMs the load.
• patch_r7_pad_before_prepare.py — the rotation tables were padded after the prepare loop, but prepare stores the [:1]/[:n] slices it is handed as views, so the unpadded originals stayed pinned alongside the padded copies. ~448 MiB.

Optional (occupancy, no measured end-to-end effect on this rig):

• patch_moe_deadscale_2cta.py, patch_moe_stages3_only.py — raise the fused MoE from 8 to 15.4 warps/SM. Kernel-level improvement; end-to-end it measured at parity here.

Apply order matters: the broadcast patch must run after the R7 loader patch.

Verifying your deployment

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.2-EXL3-TR3v4-3.5bpw",
  "messages": [{"role":"user","content":"Explain the holding of Marbury v. Madison."}],
  "max_tokens": 1200, "temperature": 0}'

A broken fused path is obvious: hallucinated case law, degenerate verbatim repetition, or empty completions. A healthy one returns an accurate ~2,000 character answer. Note this model uses a reasoning channel — with max_tokens below ~300 the content field can come back empty simply because the budget was consumed by reasoning. That is not a fault.