RESULTS.md

Measured results — GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78

All numbers below were measured on the reference rig described in SERVING.md: 4x RTX PRO 6000 Blackwell (SM120a, 96 GB, PCIe, no NVLink), TP4 + DCP4, MTP-3.

Two different KV cache dtypes appear here. The KLD evaluation was run with fp8 KV and BF16 RoPE; the throughput benchmark was run with the production nvfp4_ds_mla KV. They are reported separately and are not interchangeable.

KLD vs BF16 reference

Prefill KLD against pinned BF16 reference logits, 2,047 positions, five independent runs.

Run	KLD	Elapsed (s)
1	0.0616431846	57.6349
2	0.0623442891	56.4631
3	0.0626262729	59.0992
4	0.0604832642	60.2972
5	0.0593142137	57.3964
Mean	0.0612822449
Sample SD	0.0013762398

KLD = 0.061282 ± 0.001376.

Configuration for the KLD run:

KV cache                fp8
RoPE                    BF16
tensor parallel         4
decode context parallel 4
DCP backend             a2a
DCP KV interleave       64
reference SHA-256       87f992a689c054a0548a4b3863da6c809f9239beacd5786d0401e45904fec063

Reproduce with the runner preserved in this repo:

MODEL_OVERRIDE=/path/to/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 \
KV_DTYPE=fp8 \
DCP_SIZE=4 \
DCP_COMM_BACKEND=a2a \
DCP_KV_CACHE_INTERLEAVE_SIZE=64 \
RUNS=5 \
bash reproducibility/local-corrected-v1/eval/run_kld_corrected.sh

The runner verifies the whole-file SHA-256 of the BF16 reference logits before it starts, so the number cannot be produced against a different reference. Raw logs, parsed JSONL, aggregate JSON, image inspection and evaluation-code hashes are under reproducibility/local-corrected-v1/results/kld-fp8-dcp4-x5/.

Throughput

Measured with llm_decode_bench on the production configuration.

Decode (tok/s, aggregate):

Concurrency	@ ctx 0	@ ctx 32k
c1	60.5	56.5
c2	81.5	—
c4	115.5	103.4

Prefill (tok/s):

Context	8k	16k	32k
tok/s	1595	1546	1527

Configuration for the throughput run:

KV cache dtype          nvfp4_ds_mla
max num batched tokens  3072
tensor parallel         4
decode context parallel 4
MTP                     3
gpu memory utilization  0.95
KV capacity             340,224 tokens
MoE kernel              exl3_moe_r7_fused

At gpu_memory_utilization=0.96 and max_model_len=524288 the same build profiles to roughly 1,132,544 KV tokens (2.16x concurrency at 524K context).

Notes on interpretation

• max_num_batched_tokens=3072 was the only configuration lever that moved throughput materially (+9–15% prefill, +8–19% decode over the prior default). Several other knobs were measured and found flat or negative: A1 re-tile −14.7% prefill, route_block 256 −2%, fused all-reduce −1%, and MOE_SH_STAGES 3/4/5 within ±0.6% of each other.
• The routed-expert MoE kernel is the dominant remaining cost — roughly 50% of a decode step and 31% of a prefill chunk on these shapes — and plateaus near 490 GB/s, about 27% of HBM bandwidth. That is a kernel-side ceiling, not a configuration one.
• The 3.5 bpw figure is the average over routed experts, allocated per tensor from measured router mass times measured per-tensor loss drop. Individual tensors are 3, 4 or 5 bits; tier_bitmap.json carries the map.