# BF16 shared experts with runtime-merged K6

Date: 2026-08-10

## Canonical checkpoint layout

Revision `9ab9579774cc432df91567a36f6e9e863e0d4c9f` keeps the routed R7
K3/K4/K5 expert payload and rotations unchanged. It removes the separate
shared-expert K6 payloads from layers 3-77 and moves all original BF16 shared
experts from layers 3-78 into one `model-sharedbf16.safetensors` shard.

Header-only safetensors inspection of that revision found:

| Check | Result |
|---|---:|
| BF16 shared tensors | 228 |
| Covered layers | 3-78 |
| Non-BF16 or unexpected tensors | 0 |
| Tensor payload bytes | 5,737,807,872 |
| Central shard file bytes | 5,737,837,256 |
| Central shard SHA-256 | `ee1e7d9b2adb5d49c0895dc2f4b7d6d424b108cdb796879eee4c55d040408c6a` |

The canonical central shard was also downloaded at the pinned revision and
all 228 tensors were compared element-for-element with the pinned BF16 donor.
Every tensor matched.

The runtime concatenates each BF16 gate/up pair before online K6 encoding, so
it executes one merged `gate_up_proj` instead of two separately encoded
matrices. The first load writes a content-addressed cache; warm loads reuse it.

## Independent runtime validation

The independent POC used the same 228 BF16 tensors and the same runtime tensor
map. Its only physical difference was storing layers 3-77 back in their layer
carrier shards; physical shard placement does not change loaded tensors or
kernel selection.

All 228 BF16 tensors in the independent POC were also compared exactly with
`willfalco/GLM-5.2-EXL3-TR3-3.42bpw@a350292cb2038f2c31732569a711a89e5d72fd46`
and an independent 3.25 bpw lineage checkpoint. The conversion replaced 225
prequantized tensors in layers 3-77 and validated the three existing BF16
tensors in layer 78.

A clean TP4 load encoded exactly 300 merged `gate_up_proj` and 300 `down_proj`
matrices: 75 routed layers on four ranks. Warm restarts performed zero encodes.
MTP0 and MTP3 both produced coherent output.

### MTP0 decode

Four RTX PRO 6000 Blackwell GPUs on independent PCIe root ports, TP4/DCP1,
A16, online K6, NVFP4 DS-MLA KV, 20-second sustained runs:

| Concurrency | Separate shared K6 | BF16 + merged K6 | Delta |
|---:|---:|---:|---:|
| C1 | 50.62 tok/s | 53.67 tok/s | +6.0% |
| C4 | 157.34 tok/s | 170.04 tok/s | +8.1% |
| C8 | 258.81 tok/s | 279.75 tok/s | +8.1% |

A separate 40-second C1 run measured 54.03 tok/s. This controlled MTP0 A/B
isolates the merged shared-expert change.

### Prefill

| Prompt | Separate shared K6 | BF16 + merged K6 | Delta |
|---:|---:|---:|---:|
| 8.2k | 3,280 tok/s | 3,265 tok/s | -0.5% |
| 64.4k | 3,034 tok/s | 3,254 tok/s | +7.3% |

The 8k result is parity. Each 64k value is one long sample, so that uplift is
indicative rather than a standalone release claim.

### Quality

The paired 2,047-position full-vocabulary test used the same BF16 teacher, FP8
KV cache, and three repeats per candidate.

| Candidate | Mean KLD | Run SD |
|---|---:|---:|
| Original R7 + online K6 | 0.064467 | 0.001973 |
| BF16 source + online K6 | 0.065339 | 0.001285 |

The absolute mean difference is 0.000873, smaller than observed run-to-run
variation, and the run distributions overlap. This independent gate found no
measurable quality regression from the BF16-source merge. The repository README
also records the canonical revision's separate five-run KLD gate.

### CUDA graphs and memory

- MTP0 target steps were captured in full CUDA graphs.
- MTP3 target verification and all three draft forwards were captured in full
  CUDA graphs, with no eager fallback in the trace.
- No new host synchronization or K6 bottleneck appeared.
- At a 65,536-token model limit, MTP0 at GMU 0.97 exposed 79,104 KV tokens.
- MTP3 at GMU 0.98 exposed 75,072 KV tokens.

## Metadata correction

The first BF16-shared commit wrote `metadata.total_size=346218639128` in the
safetensors index. Parsing all 157 safetensors headers gives exactly
`346196508248` tensor payload bytes, a difference of 22,130,880 bytes. The
follow-up uses the computed tensor-byte value.

The old `MANIFEST.json` describes the parent R7 assembly: 187,580 tensors and
75 rewritten carriers ending at layer 77. The BF16-shared revision has 186,905
tensors and rewrites 76 carriers through layer 78. Rather than silently mutate
the old schema, the follow-up preserves it as `SOURCE_R7_MANIFEST.json` and
removes its no-longer-current verification seals.

## Reproduction

The converter and focused tests are under `reproducibility/bf16-shared/`:

```bash
python reproducibility/bf16-shared/build_shared_bf16_revision.py \
  --source /path/to/parent-c55c1cd \
  --source-repo-id brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 \
  --source-revision c55c1cd4ca4216c27b0dddb51a4c6757c1765d1e \
  --donor /path/to/GLM-5.2-EXL3-TR3-3.42bpw \
  --donor-repo-id willfalco/GLM-5.2-EXL3-TR3-3.42bpw \
  --donor-revision a350292cb2038f2c31732569a711a89e5d72fd46 \
  --output /path/to/output
```

Serve through the compatible Gilded Gnosis R7 helper with
`MODEL_FAMILY=glm52-exl3`, `QUANTIZATION=exl3`, and
`ONLINE_QUANT=exl3-b6`. Keep `JIT_CACHE` persistent.