# Serving GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 on the fused MoE path

Reference numbers below were measured on 4x RTX PRO 6000 Blackwell (SM120a, 96 GB
each, PCIe Gen5, no NVLink), TP4 + DCP4, MTP 3.

## Docker image

```
verdictai/glm52-exl3-sparkinfer:v39-r28-r7fused-broadcast-cu132-sm120a
```

```bash
docker pull verdictai/glm52-exl3-sparkinfer:v39-r28-r7fused-broadcast-cu132-sm120a
```

Built for `sm_120a` (Blackwell). It will not run on other architectures without
a rebuild. It already contains every patch in `patches/` -- that directory is
only needed if you are building your own image.

## Measured

| | value | configuration |
|---|---:|---|
| KV cache | **355,328 tokens** | `max_model_len` 262144, util 0.955, batched 2048, seqs 4, `nvfp4_ds_mla`, bf16 rope, DCP4 |
| Max concurrency | 1.36x | at 262,144 tokens per request |
| KLD vs reference logits | **0.069527** | 5-run gate, `nvfp4_ds_mla` + outer scales, bf16 rope, fused path |
| Prefill @ 8K | 1,725 tok/s | 8,039-token prompt, unique prefix (prefix cache defeated) |
| Decode | 75.6 tok/s | 256 tokens, `ignore_eos`, single stream |

Non-fused reference on the identical rig: 497,408 tokens, 1,757 tok/s prefill,
73.5 tok/s decode. The fused path trades KV capacity for decode throughput.

## The outer scales file is not optional

`nvfp4_mla_outer_scales.json` ships in this repo. It must be mounted and pointed
at by `VLLM_NVFP4_MLA_SCALES_FILE`. Measured on the identical build:

| nvfp4 KV | with outer scales | without |
|---|---:|---:|
| bf16 rope | **0.069527** | 0.099717 |
| fp8 rope | 0.075542 | 0.114359 |

**Omitting it costs ~30% KLD for zero memory and zero throughput benefit.** It is
a per-layer outer-scale calibration (wikitext-2-raw-v1, 2048 context).

## Settings that matter, and why

| setting | value | rationale |
|---|---|---|
| `VLLM_EXL3_R7_FUSED` | `1` | SparkInfer fused MoE. Requires the FC2-bound fix below. |
| `VLLM_EXL3_R7_FUSED_LAYERS` | `48` | Loader default. 6 layers carry a third trellis K value and correctly fall back to `exl3_moe_r7_fused`. |
| `VLLM_NVFP4_MLA_SCALES_FILE` | *(path)* | See above. Largest single quality lever. |
| `KV_FP8_ROPE` | `0` | fp8 rope yields ~16% more KV tokens but measured +8.7% KLD (0.075542 vs 0.069527). Not worth it for citation-sensitive work. |
| `VLLM_DCP_INDEXER_SHARDS` | `4` | Auto-selection picks 2 under DCP4, holding each DSA index-cache entry on two ranks. 4 is pure placement -- identical arithmetic, +8.5% KV. |
| `VLLM_EXL3_PREFILL_CAPACITY` | `1024` | Halves the 574 MiB fused prefill arena. 512 and 256 give more KV but cost 17% and 34% of prefill; 1024 is the knee (1,725 vs 1,739 tok/s at the full 2048). |
| `VLLM_EXL3_PREFILL_BLOCK_M` | `64` | Route block size. |

## If you build your own image

The fused path needs patches that are not upstream. They are in `patches/` in
this repo; each is idempotent, uses exact anchors and fails closed.

**Required for correctness:**

- `patch_sparkinfer_projection_tiers.py` — the FC2 dispatch bound. SparkInfer
  bounded the FC2 (down-projection) tier-local expert index with the **FC1**
  slot count. With per-(expert,projection) trellis bit widths, FC2 holds more
  experts than FC1 (tier1: 231 down vs 77 gate/up), so most down tiles were
  rejected and their output silently dropped — **54.14% of down-projection
  expert slots across the 48 fused layers**. Because `zero_fc2_output=False` and
  `rotation_gate` is aliased as the FC2 buffer, the rejected tiles left
  gate-rotated hidden states in place, which were then rotated, scaled and
  accumulated: fluent but wrong output, measured at **KLD 2.36**.

**Required for the memory numbers above:**

- `patch_r7_broadcast_rotations.py` — requires an r28-or-later SparkInfer base.
  R7 stores one shared rotation row per layer (`r7_shared.gate_up_suh`,
  `r7_shared.down_svh`, both `[6144]`). Earlier bases lacked broadcast rows, so
  the loader expanded each to 256 identical copies. r28's ABI-v6
  `broadcast_suh`/`broadcast_svh` takes the single row. ~0.5 GiB/rank.
- `patch_r7_ballast_dealias.py` — the loader handed `prepare` a *view into* the
  never-read ballast pool, which every frozen tier then retained (192 live
  aliases per rank), pinning ~576 MiB. Hands it a standalone scalar instead.
- `patch_r7_release_ballast.py` — releases the pool once, after all layers load.
  Must not be done per layer: that refragments the heap and OOMs the load.
- `patch_r7_pad_before_prepare.py` — the rotation tables were padded *after* the
  prepare loop, but `prepare` stores the `[:1]`/`[:n]` slices it is handed as
  views, so the unpadded originals stayed pinned alongside the padded copies.
  ~448 MiB.

**Optional (occupancy, no measured end-to-end effect on this rig):**

- `patch_moe_deadscale_2cta.py`, `patch_moe_stages3_only.py` — raise the fused
  MoE from 8 to 15.4 warps/SM. Kernel-level improvement; end-to-end it measured
  at parity here.

Apply order matters: the broadcast patch must run after the R7 loader patch.

## Verifying your deployment

```bash
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.2-EXL3-TR3v4-3.5bpw",
  "messages": [{"role":"user","content":"Explain the holding of Marbury v. Madison."}],
  "max_tokens": 1200, "temperature": 0}'
```

A broken fused path is obvious: hallucinated case law, degenerate verbatim
repetition, or empty completions. A healthy one returns an accurate ~2,000
character answer. Note this model uses a reasoning channel — with `max_tokens`
below ~300 the content field can come back empty simply because the budget was
consumed by reasoning. That is not a fault.