# Round 7 — Rewrite the GLM-5.2 routed-expert encoder (TR3 v4, sequential calibrated)

Design-and-rewrite task, run as a sequential multi-agent pipeline. The owner has
locked a set of encoder changes; your job is to rewrite the capture+encode
pipeline to implement them, research further KLD levers, and iterate the draft
through analysis and critique until it converges. Draft code only — nothing is
executed against real weights this round.

## Owner-locked decisions (not up for debate; implement them)

1. **Target model**: new checkpoint derived from
   `/home/brandonmusic/models/GLM-5.2-EXL3-DENSE6-MTP78`. Everything that is
   6 bpw EXL3 dense or BF16 there is **carried byte-exact** (sha256-gated).
   Only the routed experts (layers 3..77, 256 experts, gate/up/down) are
   re-encoded — **from the BF16 originals**, NOT from NVFP4, NOT from the
   existing trellis (no quant-of-quant).
2. **Routed average exactly 3.5 bpw**, integer bits per tensor, **hard floor
   3 bpw per tensor** (nothing below 3 anywhere).
3. **Allocation by cumulative routed mass, per layer** — the 4-bpw (or higher)
   set is chosen by measured routed-token mass at that layer, not by expert
   count. Plus **within-expert allocation**: asymmetric gate/up/down bits are
   allowed, driven by measured per-tensor sensitivity (probe it; do not assume
   which projection is most sensitive — verify on sample layers and cite the
   probe results).
4. **Gap 1 (mandatory)**: down_proj Hessians must be computed through the
   TRELLIS ROUND-TRIP gate/up (encode gate/up first, reconstruct, then build
   I_e = silu(X Wg_rt^T) * (X Wu_rt^T)), never through unquantized weights.
5. **Gap 2 (mandatory)**: cross-layer error propagation. The prior pipeline was
   one-shot capture from the source model; the new one must calibrate each
   layer against activations produced by the ALREADY-QUANTIZED predecessors
   (including the carried 6 bpw dense reconstructions and the newly encoded
   experts of earlier layers).
6. **Cross-slice joint down encode**: down_proj is encoded as ONE
   [K=2048, N=6144] tensor with the full 2048x2048 Hessian and one LDLQ walk
   over all K. No diagonal-block slicing. (The v31 header's justification at
   encode_tr3_v31.py:57-65 — "cross-slice H terms have no consumer" — is
   circular and wrong: the TP all-reduce sums the four rank partials, so
   cross-slice error covariance is exactly what the diag-block encode drops.)
7. **Per-expert intermediate-channel permutation** (act-order through SwiGLU):
   permute the 2048 intermediate channels consistently in gate-N, up-N, and
   down-K, baked into the stored weights. Exact at serve (permutations commute
   with elementwise silu/mul); zero runtime cost; enables descending-diag
   act-order for down's LDLQ. Choose the permutation criterion with research
   (act-order by H diag is the baseline; consider Hadamard-block grouping
   effects — a permutation changes which channels share a 128-block).
8. **Layer-shared residual-side rotation vectors**: gate/up input suh (6144)
   and down output svh (6144) are shared across all 256 experts of a layer,
   selected by layer-level multi-draw. The intermediate-side vectors (gate/up
   svh at 2048, down suh at 2048) stay per-expert with per-expert multi-draw.
   Rationale: the shared pair enables hoisting the residual-side rotations out
   of the per-expert loop at serve (rotate once per token instead of 8x); the
   intermediate pair brackets the SwiGLU nonlinearity and can never hoist.
9. **Multi-draw su/sv selection** (D=8-16 seeded candidates, proxy-scored,
   deterministic given seed) at the granularities in (8).
10. **Per-128-block g_scale** folded into suh/svh (element-wise vectors absorb
    any per-element scale exactly).
11. **Topology-neutral schema-v2 emission**: full tensors, no rank{r}
    pre-slicing. One trellis [K/16, N/16, 16*bits] + suh[K] + svh[N] (+ shared
    layer vectors stored once). TP slicing becomes a LOAD-time decision on
    128-element (8-tile) boundaries; EP gets whole experts. Emit a metadata map
    of per-expert per-tensor bits. Also provide a v2->v1 converter script
    (replicating shared vectors into the rank{r} schema) so the result can be
    served on the current r13 loader before the v2 loader lands.
12. **All v31 bit-exactness guarantees are preserved** (sha256 on carried
    tensors; pack->unpack equality; ext.reconstruct bit-equality; MANIFEST +
    index audit), extended with new oracles: joint-encode-vs-sliced-emission
    reconstruction equality; permutation round-trip correctness; allocation
    sums to exactly the 3.5 target; shared-vector consistency.

## Ground truth and corrections

- The ~922 GB Pass-A capture archive NO LONGER EXISTS. Do not design around
  it. The sequential-walk architecture (below) removes the need for it.
- The prior production encode source was the BF16 originals (owner statement).
  v31's DEF_SRC/luke-nvfp4 header text describes an earlier hybrid variant —
  do not infer the source from it. Parameterize --src; the owner supplies the
  BF16 location at run time. Design for streaming layer-by-layer reads
  (~1.4 TB total; never resident at once).
- Existing scripts to read (cite file:line from them):
  - `/home/brandonmusic/klc-linux/glm52_hybrid_opt/capture_hessians.py`
  - `/home/brandonmusic/klc-linux/glm52_hybrid_opt/encode_tr3_v31.py` (and v3/v2
    lineage in the same dir; v31 header documents schema, seeds, lockstep,
    g_scale GSS, bit-exactness gates)
  - exllamav3 dev checkout:
    `/home/brandonmusic/klc-linux/glm52_hybrid_opt/four_stage_20260713/exllamav3-dev`
    — the checkout is ~117 commits past v0.0.43 (`v0.0.43-117-g6b0bd4a`).
    CONCRETE RESEARCH TASK: diff its quantize/convert pipeline against the
    v0.0.43-vendored code in encode_tr3_v31.py for encoder-quality changes we
    get for free (format-compatible improvements only; the serve kernel and
    trellis format are pinned).
  - Serving reference (read-only, short-lived containers):
    image `glm52-r13-pf:v1`, vLLM exl3 loader
    (`vllm/model_executor/layers/quantization/exl3.py`, `_configure_rank_sliced`,
    `hybrid_tr3_tail`), b12x fused MoE kernels. Do NOT start a serving
    container; do NOT touch running services, checkpoints, or production.
- Model geometry (verified from config): 256 routed experts, top-8,
  hidden 6144, moe_intermediate 2048, layers 3..77 MoE, layer 78 MTP carried.
  HAD_K = HAD_N = 128; trellis tiles 16x16; MCG codebook 0xCBAC1FED;
  SIGMA_REG default 0.025; per-tensor integer bits (trellis last dim = 16*bits).
- Size arithmetic anchor: routed at flat 3.0 = 259.40 GiB; at 3.5 avg =
  ~302.6 GiB; carried non-expert tensors = 20.2 GiB; total ~322.8 GiB.
- Quality anchors (same harness, same rig): DENSE6 flat-3.0 experts KLD
  0.114391 +/- 0.00192 (n=5); willfalco 3.25 bpw mixed = 0.100; remichu
  ~4-bit experts = 0.042. The eval gate for the new model is the SAME harness
  (KLD 5x + estonia + lavd + alpha), apples-to-apples vs 0.114391.

## Presumptive architecture (validate or refute with reasons)

Single integrated sequential walk replacing the detached Pass-A/Pass-B split,
modeled on exllamav3's own convert loop:

```
for L in 3..77:
    acts_L = forward(corpus, layers 0..L-1 AS QUANTIZED)   # rolling buffer,
             # ~1M tokens x 6144 x 2B ~= 12.3 GB, disk-backed, resumable
    routing_L = router(acts_L)          -> per-expert mass curve (allocation input)
    allocate bits for layer L            (mass x sensitivity, floor 3, avg 3.5)
    encode gate/up (joint, full-K 6144 H per expert; shared layer suh;
                    per-expert svh; permutation applied)
    I_e via trellis-RT gate/up          (Gap 1)
    encode down (joint [2048,6144], full 2048^2 H, act-order permutation,
                 per-expert suh, shared layer svh)
    replace layer L weights with reconstructions; continue walk   (Gap 2)
```

Open design points you must settle and justify: per-layer 3.5 average vs a
global budget with per-layer flexibility (and how to keep it exactly 3.5
overall); where allocation probes run (per-tensor RT-error at candidate bits);
walk hardware shape (single-GPU layer streaming vs small TP) and forward cost;
resumability granularity (rental economics — the walk must survive preemption);
whether allocation may use bits {3,4} only or {3,4,5} where the mass curve is
extreme (justify with the rate-distortion arithmetic — the owner has already
rejected a 6/2.58 barbell on convexity + sub-3-cliff + hot-path-bandwidth
grounds; do not re-litigate sub-3).

## Research mandate

After the first draft exists, research and evaluate additional KLD levers that
compose with the pinned trellis format and serve kernel. For each: mechanism,
expected magnitude, cost, and a falsification test. Reject freely with
arithmetic. At minimum evaluate:
- encoder improvements in the exllamav3 dev checkout (the 117-commit delta)
- post-encode least-squares output rescale folded into svh (free at serve)
- GuidedQuant-style end-loss-weighted Hessians vs plain activation covariance
- EoRA-class low-rank error correction sidecars (note the serving cost honestly
  — extra per-expert GEMM; likely spec-only)
- permutation criteria beyond act-order (Hadamard-block grouping, energy
  balancing across future TP shards)
- sigma_reg tuning per tensor class; damping structure
- anything else you find — labeled by whether it changes the serve kernel

## Agent pipeline (sequential; each stage writes its artifact BEFORE the next starts)

1. **Main agent — research + draft.** Read the scripts and the dev checkout,
   then write `R7_RESEARCH.md` and the full rewritten pipeline under
   `r7_encoder/` (capture/walk + encode + assemble + v2->v1 converter +
   oracles; runnable draft with tests where testable without real weights).
2. **Sub-agent `analyzer`** — deep analysis of the draft against this brief:
   does it implement every locked decision correctly? Is the math right
   (Hessian construction, LDLQ ordering, permutation consistency, shared-vector
   emission, allocation arithmetic)? Writes `R7_ANALYSIS.md`.
3. **Sub-agent `critic`** — adversarial critique and correction: attack
   correctness, resumability, bit-exactness gates, edge cases (cold experts,
   degenerate H, layer 77 boundary, MTP carry), rental-failure modes, and any
   silent deviation from v31's proven guarantees. Writes `R7_CRITIQUE.md`
   including concrete corrections.
4. **Main agent — synthesis + correction.** Reconcile, apply corrections to
   `r7_encoder/`, write `R7_SYNTHESIS.md` with an iteration ledger,
   the final runbook (rental shape, BF16 staging, expected wall-clock and
   cost, resumability points), and residual risks. Iterate stages 2-4 until
   analyzer and critic both pass, or state precisely why convergence is not
   reachable.

Also produce `R7_SERVING_SPEC.md` (spec only, no vLLM changes this round):
schema-v2 loader (load-time slicing on 8-tile boundaries), two-stack mixed-bits
MoE launch (per-bit-group), optional hoisted shared-rotation variant (kernel
change; quantify the expected decode saving), and the v1-compat path.

## Rules

- Read before asserting; cite file:line for every claim about existing code.
- Mark anything unverified as **UNVERIFIED**.
- Do not execute the encoder against real weights; do not start serving
  containers; do not modify production, checkpoints, or any running service.
- Write every artifact to disk as it is produced — a truncated run must still
  leave everything learned on disk.
- Preserve determinism: all draws seeded; same inputs -> same bytes.
- A well-argued rejection of any lever, with arithmetic, is a successful
  outcome. An unfounded yes is not.