# Measured results — GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78

All numbers below were measured on the reference rig described in
[SERVING.md](SERVING.md): 4x RTX PRO 6000 Blackwell (SM120a, 96 GB, PCIe, no
NVLink), TP4 + DCP4, MTP-3.

Two different KV cache dtypes appear here. The KLD evaluation was run with
**fp8** KV and BF16 RoPE; the throughput benchmark was run with the production
**`nvfp4_ds_mla`** KV. They are reported separately and are not interchangeable.

## KLD vs BF16 reference

Prefill KLD against pinned BF16 reference logits, 2,047 positions, five
independent runs.

| Run | KLD | Elapsed (s) |
|---:|---:|---:|
| 1 | 0.0616431846 | 57.6349 |
| 2 | 0.0623442891 | 56.4631 |
| 3 | 0.0626262729 | 59.0992 |
| 4 | 0.0604832642 | 60.2972 |
| 5 | 0.0593142137 | 57.3964 |
| **Mean** | **0.0612822449** | |
| Sample SD | 0.0013762398 | |

**KLD = 0.061282 ± 0.001376.**

Configuration for the KLD run:

```
KV cache                fp8
RoPE                    BF16
tensor parallel         4
decode context parallel 4
DCP backend             a2a
DCP KV interleave       64
reference SHA-256       87f992a689c054a0548a4b3863da6c809f9239beacd5786d0401e45904fec063
```

Reproduce with the runner preserved in this repo:

```bash
MODEL_OVERRIDE=/path/to/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 \
KV_DTYPE=fp8 \
DCP_SIZE=4 \
DCP_COMM_BACKEND=a2a \
DCP_KV_CACHE_INTERLEAVE_SIZE=64 \
RUNS=5 \
bash reproducibility/local-corrected-v1/eval/run_kld_corrected.sh
```

The runner verifies the whole-file SHA-256 of the BF16 reference logits before
it starts, so the number cannot be produced against a different reference. Raw
logs, parsed JSONL, aggregate JSON, image inspection and evaluation-code hashes
are under `reproducibility/local-corrected-v1/results/kld-fp8-dcp4-x5/`.

## Throughput

Measured with `llm_decode_bench` on the production configuration.

**Decode (tok/s, aggregate):**

| Concurrency | @ ctx 0 | @ ctx 32k |
|---|---:|---:|
| c1 | 60.5 | 56.5 |
| c2 | 81.5 | — |
| c4 | 115.5 | 103.4 |

**Prefill (tok/s):**

| Context | 8k | 16k | 32k |
|---|---:|---:|---:|
| tok/s | 1595 | 1546 | 1527 |

Configuration for the throughput run:

```
KV cache dtype          nvfp4_ds_mla
max num batched tokens  3072
tensor parallel         4
decode context parallel 4
MTP                     3
gpu memory utilization  0.95
KV capacity             340,224 tokens
MoE kernel              exl3_moe_r7_fused
```

At `gpu_memory_utilization=0.96` and `max_model_len=524288` the same build
profiles to roughly **1,132,544 KV tokens** (2.16x concurrency at 524K context).

## Notes on interpretation

- `max_num_batched_tokens=3072` was the only configuration lever that moved
  throughput materially (+9–15% prefill, +8–19% decode over the prior default).
  Several other knobs were measured and found flat or negative: A1 re-tile
  −14.7% prefill, route_block 256 −2%, fused all-reduce −1%, and `MOE_SH_STAGES`
  3/4/5 within ±0.6% of each other.
- The routed-expert MoE kernel is the dominant remaining cost — roughly 50% of a
  decode step and 31% of a prefill chunk on these shapes — and plateaus near 490
  GB/s, about 27% of HBM bandwidth. That is a kernel-side ceiling, not a
  configuration one.
- The 3.5 bpw figure is the average over routed experts, allocated per tensor
  from measured router mass times measured per-tensor loss drop. Individual
  tensors are 3, 4 or 5 bits; `tier_bitmap.json` carries the map.