--- base_model: zai-org/GLM-5.2 library_name: trellis tags: - exl3 - trellis - quantization - glm - moe - mixed-precision --- # GLM-5.2 EXL3 TR3 v4 — 3.5 bpw mixed experts, MTP-78 ## Checkpoint layout update (2026-08-10): BF16 shared experts, merged online K6 The shared-expert MLP tensors are now stored as the original BF16 weights instead of pre-encoded EXL3 payloads, and the serving stack encodes them as one merged K6 payload at load. Routed R7 expert payloads and rotations are byte-identical to the previous revision; nothing else about the quantization changed. > If you pulled this repo before 2026-08-10, re-download > `model.safetensors.index.json`, `config.json`, `quantization_config.json`, > the `model-layer-003` .. `model-layer-078` carrier shards, and the new > `model-sharedbf16.safetensors`. `hf download` picks up the delta on its own. ### What changed and why **Festr** isolated a decode-throughput loss to the previous shared-expert layout. `mlp.shared_experts.{gate_proj,up_proj,down_proj}` were stored as three separately encoded K6 EXL3 trellis payloads per MoE layer (layers 3-77; the layer-78 MTP draft shared experts were already BF16). Shared gate and up as two separate payloads force **two small-M GEMM launches per routed layer per decode step** where a merged payload needs one -- 75 extra kernel launches per decode step at MTP0. His diagnostic, TP4 / DCP1 / MTP0, split vs merged shared gate+up: | decode tok/s | C1 | C4 | C8 | |---|---:|---:|---:| | split gate/up (previous layout) | 50.62 | 157.34 | 258.81 | | merged gate+up | **53.86** | **169.07** | **281.72** | This revision therefore stores the shared experts unencoded: - A new shard `model-sharedbf16.safetensors` (~5.7 GB) carries the 228 shared-expert BF16 tensors: gate/up/down for layers 3-78, including the MTP-78 draft layer. - The 76 carrier shards `model-layer-003` .. `model-layer-078` are rewritten without their shared-expert entries. All other shards are unchanged. - `tensor_storage` in `quantization_config.json` (and the copy embedded in `config.json`) drops its 225 shared-expert module entries, so a loader sees the shared experts as ordinary BF16 modules. - Routed experts (`r7-experts-layer-*.safetensors`) and all rotations: byte-identical, untouched. ### Serving Set `ONLINE_QUANT=exl3-b6`. At load the runtime concatenates gate and up while still BF16 and encodes one merged K6 payload per layer; the encoded result lands in the JIT/weight cache, so the cost is paid once, on first load. One payload, one launch. Without `ONLINE_QUANT` the checkpoint still serves, with the shared experts running in plain BF16: correct output, roughly 3.5 GB more weight memory model-wide than the online-encoded path (5.74 GB BF16 versus 2.20 GB encoded), and none of the merged-launch decode gain. ### Quality: single pass from source The BF16 tensors are the original shared-expert weights -- verified bit-identical across the willfalco/GLM-5.2-EXL3-TR3-3.42bpw and -3.25bpw lineage checkpoints by independent ranged-read sha256 sampling. The online merged K6 encode is one quantization pass from that source, exactly as the previous layout's offline K6 encode was. Nothing is re-quantized from an already-quantized representation. KLD, 5-run gate against the same BF16 reference logits: | layout | mean | sd | |---|---:|---:| | previous (offline split K6 shared) | 0.062450 | 0.001533 | | this (BF16 shared, `ONLINE_QUANT=exl3-b6`) | 0.064250 | 0.000383 | Verdict: PASS with documented delta -- 0.064250 +/- 0.000383 (5 runs, full exl3-b6 serving policy) vs 0.062450 +/- 0.001533 for the previous layout (5-run gate). Delta +0.0018; the new mean sits below the previous layout's own worst run (0.064910). Traded for +9.5% mean decode throughput and +20.5% KV-cache capacity. Decode at MTP-3 on the reference rig (4x RTX PRO 6000, TP4 + DCP4): | config | previous layout (tok/s) | this layout (tok/s) | delta | |---|---:|---:|---:| | C1 decode, MTP-3 (ctx 0-32k avg) | 61.3 tok/s | 69.8 tok/s | +13.9% | | C2 decode, MTP-3 (ctx 0-32k avg) | 90.7 tok/s | 99.1 tok/s | +9.3% | | C4 decode, MTP-3 (ctx 0-32k avg) | 127.2 tok/s | 136.8 tok/s | +7.5% | | KV cache @ 262,144 MML (nvfp4_ds_mla) | 432,384 tokens (1.65x) | 520,960 tokens (1.99x) | +20.5% | | Prefill 8k-32k | 1309-1365 tok/s | 1301-1345 tok/s | -0.6 to -1.5% (within run variance) | ### Previous layout The pre-update revision remains available at [`c55c1cd4ca42`](https://huggingface.co/brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78/tree/c55c1cd4ca4216c27b0dddb51a4c6757c1765d1e) if you need the old offline-encoded shared payloads. The exact source lineage, independent TP4 validation, conversion tool, and focused tests are recorded in [BF16_SHARED_ONLINE_K6_VALIDATION.md](./BF16_SHARED_ONLINE_K6_VALIDATION.md). ## Run it Docker image (public on Docker Hub): ``` verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a ``` ```bash docker pull verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a hf download brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 --local-dir ./glm52-tr3v4 cd ./glm52-tr3v4 && ./server.sh start ``` | File | What | |---|---| | [SERVING.md](./SERVING.md) | Image, compose, serve parameters, gotchas | | [RESULTS.md](./RESULTS.md) | Measured KLD and throughput | | [docker-compose.yml](./docker-compose.yml) | Reference deployment | | [server.sh](./server.sh) | start / stop / logs / status wrapper | Reference rig: 4x RTX PRO 6000 Blackwell (SM120a, 96 GB, PCIe), TP4 + DCP4, MTP-3. **KLD 0.061282 +/- 0.001376** (fp8 KV, BF16 RoPE, 5 runs). Decode 60.5 / 81.5 / 115.5 tok/s at c1/c2/c4; prefill 1595 tok/s at 8K. > The image is built for `sm_120a` (Blackwell). It will not run elsewhere > without a rebuild, and the checkpoint needs a mixed-bit loader -- see the > loader compatibility warning below. ## Fused MoE path (long context) An alternative serving path that runs the routed experts on SparkInfer's fused mixed-Trellis MoE kernel. It is **opt-in** (`VLLM_EXL3_R7_FUSED=1`) and targets long-context work: more KV capacity per GB than the numbers in "Run it" above, at a slightly higher KLD because it pairs with the 4-bit `nvfp4_ds_mla` cache rather than fp8. Measured on 4x RTX PRO 6000 Blackwell (SM120a, 96 GB, PCIe Gen5, no NVLink), TP4 + DCP4, MTP-3: | | value | |---|---:| | KV cache | **355,328 tokens** | | Max concurrency | 1.36x at 262,144 tokens/request | | KLD vs BF16 reference logits | **0.069527** (5-run gate) | | Prefill @ 8K | 1,725 tok/s | | Decode | 75.6 tok/s | Full configuration for every number above: `max_model_len` 262144, `gpu_memory_utilization` 0.955, `max_num_batched_tokens` 2048, `max_num_seqs` 4, CUDA-graph size 32, `kv_cache_dtype=nvfp4_ds_mla` **with outer scales**, BF16 RoPE, `VLLM_EXL3_PREFILL_BLOCK_M=64`, `VLLM_EXL3_PREFILL_CAPACITY=1024`, `VLLM_DCP_INDEXER_SHARDS=4`, 48 fused layers. Prefill measured with a unique prompt prefix so the prefix cache cannot serve it; decode with `ignore_eos` over 256 tokens. Non-fused reference on the identical rig and settings: 497,408 tokens KV, 1,757 tok/s prefill, 73.5 tok/s decode. The fused path trades KV capacity for decode throughput. > The **0.061282** figure in "Run it" is **fp8** KV cache; the 0.069527 here is > **nvfp4** KV. They are different cache formats measured against the same > reference logits, so compare them with that in mind rather than as a > regression. ### The outer scales file is required [`nvfp4_mla_outer_scales.json`](./nvfp4_mla_outer_scales.json) now ships in this repo. Mount it and point `VLLM_NVFP4_MLA_SCALES_FILE` at it. Measured on the identical build, changing nothing else: | nvfp4 KV | with outer scales | without | |---|---:|---:| | BF16 RoPE | **0.069527** | 0.099717 | | FP8 RoPE | 0.075542 | 0.114359 | **Omitting it costs about 30% KLD for no memory or throughput benefit.** It is a per-layer outer-scale calibration (wikitext-2-raw-v1, 2,048 context). The table also shows why BF16 RoPE is the default here: FP8 RoPE yields roughly 16% more KV tokens but measured +8.7% KLD with scales applied. ### What had to be fixed The fused path was previously unusable on this checkpoint and would have produced **garbage output**, not a mild regression. SparkInfer bounded the FC2 (down-projection) tier-local expert index using the **FC1** slot count. Because this checkpoint chooses the trellis bit width per (expert, projection), FC2 holds more experts than FC1 -- tier1 carries 231 down-projection experts against 77 gate/up -- so most down tiles were rejected and their output silently dropped: **6,653 of 12,288 down-projection expert slots, 54.14%**, across the 48 fused layers. The failure was fluent rather than obviously broken. `zero_fc2_output=False` and the FC2 buffer aliases `rotation_gate`, so a rejected tile left gate-rotated hidden states in place, which were then Hadamard-rotated, scaled by `down_svh`, router-weighted and accumulated. Measured **KLD 2.36** with output that hallucinated case law and degenerated into verbatim repetition. The fix is `patches/patch_sparkinfer_projection_tiers.py` (`fc1_bound_ok`). ### Files | File | What | |---|---| | [SERVING_FUSED.md](./SERVING_FUSED.md) | Full fused-path guide, every knob and its measured justification | | [nvfp4_mla_outer_scales.json](./nvfp4_mla_outer_scales.json) | Required outer-scale calibration | | [serve.sh](./serve.sh) | Reference serve script for the fused path | | [docker-compose.yaml](./docker-compose.yaml) | Reference compose deployment | | [patches/](./patches) | Patches needed to build your own fused image | ### Docker image ``` verdictai/glm52-exl3-sparkinfer:v39-r28-r7fused-broadcast-cu132-sm120a ``` ```bash docker pull verdictai/glm52-exl3-sparkinfer:v39-r28-r7fused-broadcast-cu132-sm120a ``` Built for `sm_120a` (Blackwell); it will not run on other architectures without a rebuild. Every patch in `patches/` is already applied inside it -- that directory is only needed if you are building your own image from an r28-or-later SparkInfer base. An r28-or-later base is **required**, not merely preferred: `patch_r7_broadcast_rotations.py` depends on SparkInfer ABI-v6 `broadcast_suh`/`broadcast_svh`, which earlier bases do not expose. Without it the loader expands one shared rotation row per layer into 256 identical copies, costing about 0.5 GiB per rank. ### Caveats - KLD 0.069527 is a 4-run mean; the runner was interrupted before a fifth. The separation from the unscaled 0.099717 is far larger than the run-to-run spread, but treat the third decimal as provisional. - Prefill and decode figures are single probes on a rig that has shown double-digit container-to-container variance. Treat them as indicative. - The occupancy patches (`patch_moe_deadscale_2cta.py`, `patch_moe_stages3_only.py`) raise the fused kernel from 8 to 15.4 warps/SM, but measured **at parity end-to-end** on this rig. They are included for completeness, not as a speed claim. ## What is different about this quantization ### Full-width down-projection encoding Each expert down projection is encoded jointly across its full 2048-channel input dimension. It is not independently quantized as four serving-specific 512-channel slices. Error correction can therefore compensate across the whole tensor before a loader slices it for tensor parallelism. ### Down calibration uses reconstructed gate/up outputs Gate and up are encoded at candidate bit widths and reconstructed through the same quantized representation that will be stored. Their reconstructed SwiGLU output supplies the conditional calibration input for down. Down is therefore optimized for the quantized gate/up tensors it will actually follow, not for unquantized BF16 gate/up outputs. ### Router-mass-weighted exact bit budgeting Each layer contains 768 routed-expert tensors. Starting all tensors at 3 bits uses 2,304 bit units; the exact 3.5 bpw target is 2,688 units, leaving exactly 384 one-bit upgrades. Candidate 3-, 4-, and 5-bit losses are weighted by the captured float32 routing mass. A dynamic program spends all 384 upgrades, and a tensor may receive a 4→5 upgrade only after its 3→4 upgrade is selected. This is why an equal-average “barbell” split of very high and very low bit widths was rejected: quantization error falls smoothly with bit width, so the high end wastes marginal bits while the low end crosses a steep error cliff. ### Expert-private intermediate reordering Five intermediate-channel orderings are considered for each expert. The chosen ordering is baked consistently into gate output, up output, and down input. SwiGLU is elementwise, so a consistent permutation is functionally free at serving time while giving the error-correcting walk a better row order. ### Shared residual rotations, private intermediate rotations The 6144-dimensional residual/model-space sides are shared per layer: - `gate_up_suh` is shared by all gate/up inputs in the layer. - `down_svh` is shared by all down outputs in the layer. The 2048-dimensional intermediate sides remain private to each expert: - gate output (`gate_svh`) - up output (`up_svh`) - down input (`down_suh`) The residual space must be common because tokens enter all routed experts in one coordinate system and expert outputs are routing-weighted and summed back into that system. The intermediate space exists only inside one expert, so each expert can choose the sign-vector draw that best conditions its own weights and activations without imposing a layer-wide compromise. Twelve candidate draws are searched. ### Per-128-channel scales folded into the stored representation Scales are searched independently on a 128-channel grid and folded into the existing per-element sign representation. This gives finer conditioning without a separate runtime scale tensor. ### Topology-neutral schema v2 Routed-expert tensors are stored whole with a per-tensor bit map. Tensor- or expert-parallel slicing happens at load time on 128-channel boundaries; no four-GPU topology is baked into the files. ## Authoritative corrected local build The corrected model did not rerun the expensive bit/permutation search. It recovered and froze all 75 layer decisions from R10, corrected the absolute normalization/global-scaling math, and rebuilt layers 3–77 causally on four local SM120 GPUs. Layer `L+1` was calibrated from the corrected installed output of layer `L`. For every layer, the successful supervisor performed: 1. a four-GPU attention/router-only flat capture; 2. one streamed absolute-normalization/GSS fit; 3. four pinned GPU workers consuming a dynamic 256-expert queue; 4. a corrected successor forward to create the next layer's input state; 5. atomic promotion and a durable layer seal; and 6. reclamation of reproducible capture, predecessor-state, expert-mini-shard, and no-longer-needed BF16 source-window data. The complete code, tests, decisions, receipts, and storage runbook are in [`reproducibility/local-corrected-v1`](./reproducibility/local-corrected-v1). The 75 routed shards total 317,347,848,944 bytes and their sidecar manifests total 350,486,725 bytes. All 75 layer seals are included. ## Historical rental-box R10 run The files under [`reproducibility/r10`](./reproducibility/r10) record the B300 capture/search and first encoding attempt. They remain important provenance for the corpus, deterministic prompt plan, inventories, and frozen allocation decisions, but the old encoder is superseded for reproducing corrected routed payload bytes. ### 1. Seal inputs and runtime The BF16 source, carrier checkpoint, numeric core, compiled extension, package versions, and runtime Python files were inventoried before timed work. One scalar SafeTensors serializer defect was repaired after that launch-time seal; the published `runtime_inventory.r10.json` is retained as the literal launch record, while `DEPLOYED_CODE.sha256` binds the exact post-repair code that produced the completed layer shards. ### 2. Flat capture on NVMe `r10_capture.py` performs a BF16 source forward and writes one flat capture per MoE layer through `LayerCalibRAM`/memory-mapped storage. The 75 captures total about 970 GB, so they live on NVMe rather than `/dev/shm`. The exact corpus file is [`calibration/reap_recall_calib.jsonl`](./calibration/reap_recall_calib.jsonl): - 12,228 JSONL records - 34,002,059 bytes - SHA-256 `cf247acc7c5da9f0600c7d6ab3b7c2fcfc54ec30b794e3b6047559285fa44df4` The deterministic capture plan selected 1,773 prompts totaling 1,049,589 tokens. The exact selection and the complete 75-layer capture summary are in [`reproducibility/r10/capture`](./reproducibility/r10/capture). Capture and encoding were operationally pipelined: GPU 0 published captures in layer order while the other GPUs consumed already-sealed captures. After all 75 captures completed, GPU 0 joined the encoder pool. ### 3. Dynamic 12-worker encoding Six B300 GPUs run two pinned workers each. Workers pull completed layer captures from a SQLite dynamic queue rather than receiving static layer ranges. The deployed settings are: ```text layers 3-77 workers 12 (two per GPU) GPUs 6 rotation draws 12 held-out rows 4096 minimum fit rows 1024 row chunk 4096 factor cache 512 MiB per worker sigma regularizer 0.025 CPU threads 36 per worker ``` Each completed layer is emitted atomically as one `r7-experts-layer-NNN.safetensors` shard plus its JSON manifest. A layer does not enter the completed queue state until both artifacts exist. ### 4. Assemble, upload, and verify The final assembler hard-links complete expert shards, hard-links clean carrier shards, rewrites only carrier shards that mix retained and replaced tensors, and creates the final tensor index and manifest. Progressive uploads use the same final expert-shard basenames, so the authoritative final folder upload reuses the already-present Hub objects. ## What was removed for wall-clock speed The accuracy design above was not weakened. Work that did not change emitted bytes was removed from the timed encoder: payload/runtime re-hashing, routing audits, fixed-point successor passes, install-and-forward checks, pack/unpack repeat decodes, functional oracles, and assembly/conversion. Search, allocation, reconstruction-based down calibration, rotations, permutations, and final encoding remain. The result is sealed by construction and written atomically. Full checkpoint assembly and structural verification occur after all 75 layers complete. There is deliberately no claim that expensive functional evaluation ran during quantization. ## Historical rental-box performance The live machine is: - 6× NVIDIA B300 SXM6 AC, 275,040 MiB each, compute capability 10.3 - Python 3.12.3 - PyTorch 2.12.1+cu132 - Transformers 5.12.1 - CUDA 13.2 - exllamav3 numeric extension built from v0.0.43 for `sm_103` - 4.295 TB NVMe filesystem and a 1.509 TB memory cgroup Uncached completed layers have taken roughly 130–138 minutes per worker, including search, all 256 expert probes, exact allocation, final encoding, and atomic emission. Twelve concurrent slots turn that into waves; it is not a 75× serial runtime. The uploader and memory-cache reclaimer run at reduced priority so GPU encoding remains dominant. ## Corrected KLD result The corrected local checkpoint was measured five times against the same BF16 reference logits, with standard FP8 KV cache, BF16 RoPE, TP4/DCP4, one 2,048-token context, and 2,047 scored positions per run: ```text run 1 0.0616431846 run 2 0.0623442891 run 3 0.0626262729 run 4 0.0604832642 run 5 0.0593142137 mean 0.0612822449 sd 0.0013762398 ``` The reference-logits SHA-256 is `87f992a689c054a0548a4b3863da6c809f9239beacd5786d0401e45904fec063`. The exact result JSON, raw run logs, evaluation-code hashes, runner, and scoring script are published with the corrected bundle. ## Reproducing the corrected run Start with the exact guide in [`reproducibility/local-corrected-v1/README.md`](./reproducibility/local-corrected-v1/README.md). It explains the required checkpoint identities, decision recovery, local inventory rebinding, layer-3 causal anchor, rolling execution, seal adoption, NVMe free-space frontier, BF16 shard paging, safe cleanup order, assembly, and KLD command. The old [`reproducibility/r10/README.md`](./reproducibility/r10/README.md) is historical lineage, not the corrected byte-production runbook. ## Tuning: prefill route block size The single largest configuration win found on this checkpoint. The default `VLLM_EXL3_PREFILL_BLOCK_M=64` leaves substantial performance on the table. Measured end-to-end on the reference rig (4x RTX PRO 6000, TP4 + DCP4, MTP-3, `nvfp4_ds_mla` KV, `max_model_len` 262144, `max_num_batched_tokens` 2048, util 0.95). Output verified coherent and unchanged at every setting: | `VLLM_EXL3_PREFILL_BLOCK_M` | 64 (default) | 32 | 16 | **8** | |---|---:|---:|---:|---:| | Prefill 8K (tok/s) | 1157 | 1403 | 1471 | **1674** | | Prefill 64K (tok/s) | 1134 | 1320 | 1397 | **1600** | | Prefill 128K (tok/s) | 1112 | 1211 | 1308 | **1514** | | Decode (tok/s) | 57.84 | 59.55 | 63.44 | **70.43** | | KV capacity (tokens) | 441,344 | 446,720 | 448,512 | **448,512** | At `block_m=8`: **prefill +44.7% / +41.1% / +36.2%**, **decode +21.8%**, **KV +1.6%**. Prefill, decode and KV all improve together -- there is no trade-off to balance. ### Why Nsight Compute on the isolated MoE kernel (single GPU, synthetic weights, so no TP collectives to deadlock the profiler) at the stock `block_m=64`: ``` DRAM Throughput 24.03% <- memory not saturated Compute (SM) Throughput 35.96% <- compute not saturated either Registers Per Thread 244 Dynamic Shared Memory 101.38 KB of 102.40 KB configured Block Limit Registers 1 Block Limit Shared Mem 1 Theoretical / Achieved Occupancy 16.67% ``` Dropping to `block_m=16` moves **DRAM throughput 24.03% -> 64.22%** and halves kernel duration (1530 -> 575 us), with registers 244 -> 144 and shared memory 101.38 -> 52.22 KB. Occupancy does **not** change (16.67% either way). The kernel is a cooperative one-grid launch -- grid size equals the SM count, so exactly one block per SM by construction. The gain is instead the zero padding a 64-row route block carries when it holds far fewer live rows; smaller blocks waste less. `block_m` below 32 additionally requires a register-count table entry for the `cta_m_blocks=1` specialization (`(256, 1, 16, 4, False)`), absent upstream; without it the launch model raises `missing W4A16 register count for NVFP4 BF16 specialization`. ### Levers that did NOT help Measured and rejected, so nobody repeats them: | Lever | Result | |---|---| | FC1/FC2 tile config | `tile_n=128` already optimal (`tn=64` was 18% slower at m=128; `tn=256` unsupported) | | `_STAGES` pipeline depth | 4 already optimal; 3 within noise; **2 was 27% slower** | | `VLLM_EXL3_TRELLIS_MAX_M` 32 -> 64 | within noise (+0.7-1.0% prefill, decode flat) | | `max_num_batched_tokens` 2048 -> 3072 | trade-off, not a win: prefill +2.7-8.2% but **KV -18%** (448,512 -> 366,336) | ## KV cache The serving stack uses **MLA** (`kv_lora_rank=512`, `qk_rope_head_dim=64`), so the KV cache is one latent per token per layer rather than per-head K and V, and DCP4 shards it across the four GPUs (`--dcp-kv-cache-interleave-size 64`). **Supported `--kv-cache-dtype`** (the bundled launcher rejects anything else): `nvfp4_ds_mla` (production default, 4-bit MLA latent), `fp8_ds_mla`, `fp8`. ### Measured capacity Every row below is a number printed by vLLM at startup on the reference rig (4x RTX PRO 6000, 96 GB, TP4 + DCP4, MTP-3, CUDA graphs on, `nvfp4_ds_mla` KV). A KV number is meaningless without its utilization, KV dtype, context length and batch budget, so all of them are listed. | util | KV dtype | max_model_len | max_num_batched_tokens | MTP | KV capacity | |---:|---|---:|---:|---:|---:| | 0.95 | nvfp4_ds_mla | 8,192 | 2,048 | 3 | **457,728 tokens** | | 0.95 | nvfp4_ds_mla | not recorded | 3,072 | 3 | 340,224 tokens | The first row is the production MoE path (`exl3_moe_r7_fused`), measured 2026-08-05. The second is the configuration the throughput numbers in [RESULTS.md](RESULTS.md) were taken under; its `max_model_len` was not recorded alongside the figure, so treat it as indicative rather than reproducible. Capacity moves inversely with `max_num_batched_tokens`: dropping 3,072 -> 2,048 returned roughly 117k tokens of KV, because the prefill scratch arena and profiling peak shrink with the batch budget. ### About the 1.13M figure `server.sh` carries a comment claiming ~1,132,544 tokens (2.16x at 524K context) at `GPU_MEMORY_UTILIZATION=0.96`. **Do not read that as usable serving capacity.** It is an auto-profile ceiling measured without the speculative decode path engaged, and a configuration that reaches it is not one you would serve from. It is retained in the script's comment for provenance only and is not reproduced as a headline number here. ### Pinning a smaller cache Capacity is auto-profiled at startup. Leave `NUM_GPU_BLOCKS_OVERRIDE` empty to take the maximum the utilization allows, or set a positive integer to pin a smaller cache and leave memory for other work on the same GPUs. Note the KLD figure in [RESULTS.md](RESULTS.md) was measured with **fp8** KV and BF16 RoPE, while throughput and the capacities above use **`nvfp4_ds_mla`**. They are not interchangeable. ## Repository map ```text calibration/ reap_recall_calib.jsonl exact live corpus reproducibility/r10/ README.md superseded B300 run and provenance RUN_METADATA.json human-readable run parameters capture/ exact prompt plan and capture summary inventories/ source, carrier, numeric, runtime seals DEPLOYED_CODE.sha256 exact post-repair deployed-code hashes lineage/encode_tr3_v31.py proven numeric core r7_encoder/ exact deployed encoder package run/ exact launch, supervisor, guard, assembly, finisher, and progressive-upload scripts reproducibility/local-corrected-v1/ README.md authoritative corrected reproduction guide STORAGE_BOUNDED_RUNBOOK.md NVMe paging, seals, and reclamation order code/ exact executed local correction code decisions/ frozen decisions for all 75 routed layers receipts/ preflight, source windows, and 75 seals tests/ local correction regression tests results/ corrected five-run KLD record r7-experts-layer-*.safetensors progressively uploaded expert weights r7-experts-layer-*.json per-layer bit maps and provenance ``` ## Loader compatibility warning This schema-v2 checkpoint is specification-complete but is **not compatible with the current production loader**. Serving mixed 3/4/5-bit experts requires the loader extension described in `R7_SERVING_SPEC.md`: group tensors into a 3-bit stack and a 4/5-bit stack and launch the fused kernel for each stack. The included converter refuses to label an unsupported conversion as ready. The quantized weights inherit the license and use restrictions of `zai-org/GLM-5.2`. The vendored exllamav3 numeric core is from turboderp's v0.0.43 release (MIT).