# Round 7 research record — CODEX_ROUND7 Status: design evidence and draft decisions. No model weights were loaded and no encoder or serving process was run. Results which require BF16 weights or the owner's calibration/evaluation corpus are marked **UNVERIFIED**. ## Evidence scope The shorthand paths used below are: - `capture_hessians.py` = `/home/brandonmusic/klc-linux/glm52_hybrid_opt/capture_hessians.py` - `v31` = `/home/brandonmusic/klc-linux/glm52_hybrid_opt/encode_tr3_v31.py` - `dev/convert_model.py` and `dev/quantize.py` are respectively `exllamav3/conversion/convert_model.py` and `exllamav3/modules/quant/exl3_lib/quantize.py` under the requested dev checkout. - `r13/exl3.py` and `r13/_impl.py` were extracted read-only from image `glm52-r13-pf:v1` without starting it. Stable SHA identities and the exact cited excerpts are preserved in `R7_EXISTING_CODE_EVIDENCE.md:7-144`. The old capture is structurally unsuitable for Round 7. It writes the inputs and IDs for all 75 MoE layers at once and describes about 922 GiB of activation storage (`capture_hessians.py:3-17`, `capture_hessians.py:40-45`, `capture_hessians.py:192-205`). Its routing artifact stores selected expert IDs and integer counts, not normalized routed mass (`capture_hessians.py:220-250`, `capture_hessians.py:423-442`). Its retry behavior is whole-pass and deletes an incomplete directory when explicitly restarted (`capture_hessians.py:514-537`). Round 7 therefore uses a rolling, per-layer state buffer and a journal. The v31 encoder is a valuable numerical and audit reference, but not a Round 7 base. It encodes rank-local gate/up N slices and down K slices (`v31:43-69`), emits rank-sliced schema v1 (`v31:107-117`), hard-codes 3 bits and TP=4 (`v31:151-165`), and includes the now-rejected diagonal-block down construction (`v31:1728-1783`). It nevertheless supplies the invariants to retain: carried payload hashes, pack/unpack equality, extension reconstruction equality, and index/manifest audit (`v31:77-86`). Its atomic raw tensor writer uses a temporary file plus `os.replace`, a pattern retained here (`v31:269-305`). The prior capture computes down inputs through dequantized source gate/up weights (`capture_hessians.py:14-17`), and v31 loads those weights before constructing the full down covariance and then selects a diagonal slice (`v31:1728-1783`). Round 7 instead reconstructs the newly encoded gate/up weights and builds the down covariance from their actual TRELLIS round trip. ## Dev-checkout delta The requested comparison found no format-compatible encoder-quality change to adopt for free. The current dev `quantize.py` and the file at tag v0.0.43 have the same SHA-256: `34d87d473984963efcd3bf15bcb9f297850b5a7ccccb8859f658aaab13f47e6a`. The relevant current implementation still finalizes the covariance with the same randomized Hadamard-side vector and default `sigma_reg=0.025` (`dev/quantize.py:604-679`), performs the same regularization/G-scale stages (`dev/quantize.py:704-715`, `dev/quantize.py:835-941`), and enters the same EXL3 quantization path (`dev/quantize.py:944-1030`). The conversion driver *does* demonstrate the architecture Round 7 needs: it captures a module on the current rolling state, quantizes/reloads that module, then advances the state through the quantized module before continuing (`dev/convert_model.py:582-710`). Its state file is resumable (`dev/convert_model.py:253-264`, `dev/convert_model.py:733-748`). The observed 117-commit delta in the two inspected files was CLI behavior and a default-codebook change, not quantizer math. The exact commits, hashes, and three-line diff are preserved at `R7_EXISTING_CODE_EVIDENCE.md:193-214`. Round 7 pins the MCG marker and does not accept a new default silently. This conclusion is about the inspected files only; any uninspected downstream CUDA change affecting numerical output is **UNVERIFIED** and is covered by the extension reconstruction oracle. ## Settled encoder design ### Exact rate allocation Allocation is per layer, not global. There are 256 experts x 3 projections = 768 equal-sized matrices per MoE layer: each has 6144 x 2048 weights. Starting at the 3-bit floor costs 2,304 tensor-bit units; exact 3.5 bpw costs 2,688, so the deterministic optimizer must spend exactly 384 one-bit upgrades per layer. With candidate bits `{3,4,5}`, if `n4` tensors end at 4 and `n5` end at 5, the hard equality is: ```text n4 + 2*n5 = 384 sum(bits) = 3*768 + 384 = 2688 mean(bits) = 3.5 exactly ``` The dynamic program maximizes measured mass-weighted marginal benefit. For projection `p` of expert `e`, it consumes two ordered increments: `mass[e] * (loss3-loss4)` and `mass[e] * (loss4-loss5)`. A 5-bit second increment cannot be selected unless its first increment is selected. Ties follow canonical tensor order, low-bit choices first, and strict replacement only for a greater integerized gain; the emitted manifest records the integer proof above. There is no sub-3 candidate and no floating-point budget comparison. `mass[e]` is the sum of the actual float32 normalized top-k router weights for expert `e`. Each finite nonnegative float32 is converted to integer units of `2^-149`, so the serialized sum is exact and invariant to batch boundaries; IDs/counts are insufficient. The historical router recomputation is sigmoid(logits)+bias followed by top-k (`capture_hessians.py:220-249`); it does not persist normalized weights. The Round 7 backend contract must return the exact model-runtime IDs and weights, including normalization and routed scaling, and the walk fails if the per-token mass audit disagrees with the backend-declared expected sum. Per-projection sensitivities are measured for all tensors in each owner-run layer on deterministic fit/holdout row plans. Each probe performs the actual candidate TRELLIS encode/reconstruct and scores held-out output error; down candidates use the final-round-trip gate/up candidate selected for that fixed-point iteration. The draft emits one immutable `probe-iter-NN.json` per fixed-point iteration, with all three losses, the full context-map hash, all three reconstruction hashes, row-plan hashes, and source/numeric bindings. It refuses allocation when the artifact is missing or incomplete. **UNVERIFIED probe result:** no real BF16 weights or calibration activations were allowed to run in this round, so there is no defensible ranking of gate/up/down sensitivity yet. The implementation does not assume one. The owner run must fill and hash those probe ledgers; a report that labels a projection most sensitive without that artifact is invalid. ### Sequential calibrated walk The rolling state is stored as one preserved prompt per BF16 shard plus token/position metadata and the I32 cross-layer DSA `prev_topk_indices` tensor, not as a flattened token matrix. Official GLM-MoE-DSA either computes or reuses that sparse-attention state (`R7_EXISTING_CODE_EVIDENCE.md:146-159`), and its post-attention norm is the precise routed-MoE capture boundary (`R7_EXISTING_CODE_EVIDENCE.md:175-187`). Initialization forwards carried layers 0..2 from the target reconstruction. At layer `L`: 1. Audit the sealed predecessor state for `L`. 2. Route it and accumulate weighted expert mass. 3. Accumulate full 6144 x 6144 input covariances for gate/up. 4. Probe/allocate, choose one layer-shared residual input vector, and encode full gate/up matrices for each expert. 5. Reconstruct gate/up through the same pack/unpack/extension path that serving uses. 6. Form `I_e = silu(X_e Wg_rt^T) * (X_e Wu_rt^T)` and accumulate a full 2048 x 2048 down covariance. 7. Encode each full `[K=2048,N=6144]` down tensor with one LDLQ walk. 8. Reopen each sealed packed expert mini-shard, verify its reconstruction hash, install that packed-bound reconstruction for layer `L`, forward every prompt shard once, atomically seal the layer output state, and discard the predecessor shard only after the successor seal is audited. Layer 77 is encoded and sealed without forwarding into layer 78. Layer 78/MTP is carried. No layer-78 expert name may appear in the replacement set. The previous v31 source loader is explicitly NVFP4-oriented (`v31:1792-1846`); Round 7's source adapter accepts BF16 only and rejects quantization metadata or integer/packed expert payloads. The target carrier adapter separately reads the DENSE6/MTP78 checkpoint. This prevents accidental quant-of-quant. ### Full down Hessian and cold experts The down encoder receives exactly one 2048 x 2048 finalized covariance and one full `[2048,6144]` EXL3 convention matrix. The encoder API has no K-slice option; the topology split exists only in emission/conversion code. The block LDL/LDLQ reference operates bottom-up with error feedback (`v31:814-945`), so a channel permutation changes ordering materially and is selected before finalizing the down covariance. For a cold expert, the fallback uses a deterministic mass-stratified subset of layer inputs and forwards those rows through *that expert's reconstructed TRELLIS gate/up*. It may shrink the expert covariance toward a layer covariance, but it may never substitute BF16 gate/up. The shrinkage coefficient, sample indices, and reason are journaled. Degenerate covariance regularization either passes the same finite/positive checks as the normal path or fails closed; it does not silently use identity H. The historical v31 fallback used a layer covariance below a routed-token threshold (`v31:66-69`, `v31:151-158`), which motivates but does not define this safer fallback. ### Intermediate permutation For Hugging Face `[out,in]` matrices and a permutation matrix `P` on the 2048 intermediate coordinates: ```text Wg' = P Wg Wu' = P Wu Wd' = Wd P^T silu(x Wg'^T) * (x Wu'^T) = (silu(x Wg^T) * (x Wu^T)) P^T ``` Therefore the final product is unchanged before quantization. The permutation is baked into all three encoded weights and never served as metadata requiring a gather. The mandatory pilot compares: - identity; - stored order whose bottom-up LDLQ visit is descending by down-input diagonal (the act-order baseline); - stored descending diagonal (a separately named reverse-visit candidate); - energy-balanced assignment across sixteen 128-channel Hadamard blocks, with descending energy round-robin distributed among blocks; - energy-balanced contiguous-block variant. All policies are selected by the same deterministic weight/H-diagonal sketch; the mass-stratified pilot sample is then checked with held-out reconstructed expert output loss. Every expert is checked again by its actual candidate probes and final encode. Block balance is relevant because v31 fixes Hadamard blocks to 128 (`v31:169-173`) and LDLQ walks in reverse K order (`v31:873-945`). The direction and policy winner are **UNVERIFIED** until the pilot artifact exists. The permutation oracle checks bijection, byte-exact weight round trip, and relative functional error at float64 tolerance for each final expert. ### Rotations, multi-draw, and G-scale For every layer, 8–16 deterministic candidates jointly proxy-score one 6144-vector gate/up `suh` and one 6144-vector down `svh` on a mass-stratified expert sample. The winning pair is stored once under the layer. Each expert independently selects its gate `svh[2048]`, up `svh[2048]`, and down `suh[2048]` jointly over the full reconstructed SwiGLU path. Shared decisions promote two candidates to full TRELLIS scores on the mass-stratified sample. Per-expert candidates are selected by the deterministic sketch; the same sample receives a full chosen- bundle validation, while every expert later receives actual 3/4/5 probes and a final held-out encode. Every shared promoted full-round-trip is checkpointed per sampled expert before its aggregate is accepted (`r7_encoder/search.py:329-382`). Seeds are a hash of recipe version, layer, expert or `shared`, role, and draw index; scheduling cannot alter draws (`r7_encoder/determinism.py:68-77`). G-scale is constrained to separable 128-coordinate block factors so it commutes with the pinned block Hadamard transform. K-side factors repeat into `suh`; N-side factors repeat into `svh`. This is an exact parameter folding, not an extra stored tensor. A generic per-element scale placed on the wrong side of a Hadamard transform would not commute and is forbidden. The prior encoder's quantization stages contain a G-scale step (`v31:1116-1184`, `v31:1270-1384`); Round 7 extends the recorded scale proof to every block. The replacement regularization is explicit. For stored vectors `su,sv` and weight `W[K,N]`, the codec forms `R = Had_K((W / sv) / su) Had_N`, with the exact stored FP16 vectors used on both encode and decode. Input-side `su` additionally includes the inherited `1/(-CODEBOOK_SCALE)` MCG normalization; `sigma_reg * mean(diag(H))` is added once before the same block-LDL primitive. The selected per-128 K/N factors are folded reciprocally into `su/sv`. Unlike v31's regularizer, this draft does not silently derive unrestricted per-channel input/output RMS factors because the residual-side vectors are owner-locked layer-shared. Whether proxy-selected block factors recover that rate-distortion benefit is **UNVERIFIED** and must be compared against v31 on an owner-authorized sample layer before production. ### Fixed point for asymmetric bits Down sensitivity depends on gate/up round trips, while gate/up bit allocation depends on the layer budget that also includes down. The implementation uses a deterministic fixed point: 1. Encode/reconstruct gate/up probes at 3, 4, and 5 bits. 2. Initialize the down path with 4-bit gate/up reconstructions. 3. Measure down candidates, solve the exact layer allocation, then rebuild the down inputs through the newly selected gate/up round trips. 4. Repeat until the entire 768-entry gate/up/down allocation map is unchanged; detect cycles, cap iterations, and fail closed on non-convergence. The final down Hessian and final down encode are always rebuilt after convergence. No provisional down tensor is emitted. ### Determinism, installed-path evidence, and corpus closure The draft pins its process environment before Torch CUDA or Transformers imports, requires deterministic algorithms with `warn_only=False`, disables TF32 and hub kernels, pins eager expert dispatch, and inventories both the wrapper and original official expert callables (`r7_encoder/determinism.py:21-43`, `r7_encoder/transformers_runtime.py:197-350`). If CUDA was initialized before that contract, or a required deterministic implementation is unavailable, the owner run fails before accepting state bytes. The first prompt of every successor layer is forwarded twice and must produce identical BF16 hidden and int32 DSA-index bytes (`r7_encoder/transformers_runtime.py:618-631`, `r7_encoder/state.py:463-488`). Actual CUDA repeat results are **UNVERIFIED**. The TRELLIS extension is loaded from the exact inventoried canonical path and hash; an ambient same-named module is rejected. CUDA LDL OOM aborts rather than falling back to CPU, and extension reconstruction is repeated byte-for-byte inside each completed encode (`r7_encoder/trellis.py:79-164`, `r7_encoder/trellis.py:342-400`). These are draft gates; no full-size extension encode was executed in Round 7. Per-expert installation is a packed-decode receipt, not a self-comparison. After all 256 experts are installed, the runtime constructs the official eager expert module, uses deterministic fixed routes with repeated expert hits, runs that module twice, and compares against a separately accumulated reference that reads the packed-decoded matrices rather than the official parameters (`r7_encoder/transformers_runtime.py:1018-1253`, `r7_encoder/glm52_backend.py:902-985`). The aggregate audit is included in the layer install artifact before completion (`r7_encoder/layer.py:1466-1485`). Real-weight arithmetic remains **UNVERIFIED**. Tokenizer assets are part of the carrier inventory, and the entire resolved `tokenizers` package is part of the runtime inventory. Before prompt zero, the adapter seals `CORPUS_PLAN.json` with exact input IDs, global offsets, and the complete expected prompt domain (`r7_encoder/inventory.py:183-264`, `r7_encoder/transformers_runtime.py:379-489`). Both carried-prefix progress and normal state partials bind that plan. Carried layers 0..2 checkpoint every layer/prompt intermediate; successor layers yield one prompt before starting the next (`r7_encoder/transformers_runtime.py:671-905`, `r7_encoder/transformers_runtime.py:1304-1369`, `r7_encoder/state.py:291-452`). ## Additional KLD levers All magnitude statements below are **UNVERIFIED** until measured by the same KLD-5x + Estonia + LAVD + alpha harness. ### 1. Post-encode least-squares output rescale — pilot, serve-free For each output coordinate, fit `a_j = sum(yhat_j*y_j) / max(sum(yhat_j^2), eps)` on a fit split and multiply the existing `svh_j` by `a_j`. Gate/up may fit per expert before the final down Hessian. Because down `svh[6144]` is layer-shared, down fits one weighted layer-level vector aggregated across routed experts; it cannot silently create per-expert residual vectors. Expected magnitude: **UNVERIFIED**, likely small. Cost: one extra statistics pass, zero stored tensors and zero serving ops. Falsification: reject if held-out expert-output MSE or any KLD-5x seed regresses, or if fitted scale clipping exceeds a declared bound. This lever is **specified but not enabled in the production draft**. The helper implements the statistic only; no owner-run path applies it before the required sample-layer falsification test. ### 2. GuidedQuant-style loss-weighted covariance — opt-in pilot [GuidedQuant](https://arxiv.org/abs/2505.07004) weights layer reconstruction with downstream loss gradients and uses a Fisher-like covariance rather than plain activation covariance. It composes with LDLQ by accumulating `X^T diag(mean(g^2)) X`. Expected magnitude: **UNVERIFIED**. Cost is high: a backward calibration pass, gradient storage/recomputation, and loss-target choice. Falsification: on a fixed sample-layer pilot, require a held-out end-loss and KLD improvement over plain H at identical bits and seeds; reject if the gain disappears across two corpus halves. This is **not implemented in the production path**: the covariance accumulator exposes the weighted primitive, but the layer processor deliberately uses unweighted routed membership until an owner-authorized experiment supplies gradients and passes the gate. ### 3. EoRA-class low-rank error sidecars — rejected for base checkpoint [EoRA](https://arxiv.org/abs/2410.21271) compresses quantization error in an activation-aware eigenspace. A rank-`r` sidecar for every projection adds two matrices and an extra low-rank path per selected expert. Even at rank 8, each token's top-8 path would add many small memory-bound GEMMs unless fused into the MoE kernel. Expected KLD gain: **UNVERIFIED**. This violates the pinned serving path and is therefore spec-only, with a future fused-kernel experiment as its falsification test. No sidecar appears in schema v2. ### 4. Rotation/permutation pilots — accepted as mandatory search [SpinQuant](https://arxiv.org/abs/2405.16406) and [QuaRot](https://arxiv.org/abs/2404.00456) support the general premise that orthogonal rotations can suppress outliers, while GPTQ's Hessian-aware error feedback provides the LDLQ basis ([GPTQ](https://arxiv.org/abs/2210.17323)). Round 7 does not import their formats. It searches only transformations that fold into existing `suh/svh` vectors or baked channel order. Expected magnitude: **UNVERIFIED**. Falsification: fixed train/holdout split, identical trellis seeds, and exact serving reconstruction oracle; reject a criterion that wins proxy error but loses held-out full-expert error. ### 5. `sigma_reg` and structured damping — accepted as a pilot Search `{0.01, 0.025, 0.05, 0.10}` separately for residual-input gate/up and intermediate down covariances. The incumbent default is 0.025 (`v31:151-156`; `dev/quantize.py:604-679`). Also test diagonal shrinkage toward the class median for cold experts. Expected magnitude: **UNVERIFIED**. Cost: four finalizations on a sample, not four full-model runs. Falsification: reject unless held-out RT loss improves in both sample-layer halves and factorization health does not worsen. This search is **spec-only and UNVERIFIED** in Round 7. The runnable draft pins `sigma_reg=0.025` in its recipe; changing it creates a different recipe and is not silently selected per tensor. ### 6. Activation-aware weight scaling — constrained pilot [AWQ](https://arxiv.org/abs/2306.00978) motivates protecting salient channels, but arbitrary rescaling around SwiGLU can alter the function or add serving work. Only algebraically exact reciprocal scalings that are absorbed into the existing vectors and validated through the full expert path are admissible. Expected magnitude: **UNVERIFIED**. Reject any form that needs an additional runtime tensor or does not survive the synthetic algebra oracle. ### 7. Mixed precision rate-distortion allocation — accepted [HAWQ-V2](https://arxiv.org/abs/1911.03852) motivates sensitivity-aware mixed precision, but Round 7 uses measured TRELLIS round-trip marginal loss rather than a trace-only proxy. Bits `{3,4,5}` allow a genuinely hot tensor to consume two of the 384 upgrades only when its measured second marginal beats other first marginals. Bits 6 and sub-3 are outside this design. Expected improvement over count-ranked `{3,4}`: **UNVERIFIED**. Falsification: the exact allocation oracle plus layer-level held-out output error at the same 2,688 bit-unit budget. ## Serving compatibility finding The unmodified inspected r13 loader requires scalar rank-sliced metadata and a single `bits` value (`R7_EXISTING_CODE_EVIDENCE.md:21-40`). Its rank-sliced fused preparation checks one integral bit width across gate/up/down slabs and passes one scalar width to the plan (`R7_EXISTING_CODE_EVIDENCE.md:42-87`). The existing mixed-K patch partitions whole experts and explicitly rejects different gate/up/down widths inside an expert; its tiers run sequentially and accumulate outputs (`R7_EXISTING_CODE_EVIDENCE.md:117-144`). Therefore an arbitrary asymmetric schema-v2 checkpoint is **not** loadable by unmodified r13 merely by renaming tensors. The supplied complete-checkpoint v2-to-v1 converter replicates shared vectors, rank-slices tensors, rebuilds the index/config/manifests, and preserves carried payloads, but the current image also needs the loader-only two-stack mixed-bit dispatch described in `R7_SERVING_SPEC.md`. The converter fails closed when asked to declare unmodified-r13 compatibility for a non-uniform projection map. This is an honest compatibility boundary, not a reason to violate the owner-locked asymmetric allocation. ## Cost, quality, and unresolved measurements - The owner-provided size anchors imply about 302.6 GiB routed and 322.8 GiB total at exact 3.5 bpw. They are requirements, not independently verified measurements in this draft. - Rolling BF16 hidden activations are about 12 GiB for one million x 6144 elements. The required DSA index state can add up to about 8 GiB at `index_topk=2048` (`R7_EXISTING_CODE_EVIDENCE.md:189-191`), before JSON, checksums, and atomic predecessor/successor overlap. All storage figures are **UNVERIFIED** until the deterministic corpus plan is materialized. - Full 6144 covariances and repeated full-weight probes dominate GPU memory and time. Exact wall-clock and dollar cost are **UNVERIFIED** until one owner-authorized sample layer is timed. The draft supplies `run --pilot-stop-after-layer 3`, which stops only after a sealed layer and writes a content-bound timing/storage report (`r7_encoder/walk.py:616-727`); that mode was not executed against real weights in this round. - The static successful-two-iteration schedule has 1,008 search tensor encodes, 3,328 first-context probe encodes, `13*C` follow-up probe encodes where `C` is the number of experts whose local `(Kg,Ku,Kd)` context changed, and at most 1,536 final/interaction encodes per layer. Thus the planning range is 7,536–9,200 tensor encodes per layer for `C=128..256`; these are structural counts, not a wall-clock estimate. A nonconvergent run may spend more and then fail closed. - KLD improvement is **UNVERIFIED**. The draft defines gates; it makes no quality claim. ## Acceptance gates for the eventual owner run 1. BF16-source audit rejects quantized expert inputs; carrier manifest hash is sealed before replacement. 2. Probe artifact is complete and demonstrates measured per-projection losses; no assumed sensitivity order. 3. Every layer allocation has 768 tensors, minimum 3, integer bits in 3..5, sum 2,688, mean exactly 3.5, and a mass checksum. 4. Final down-H provenance names the exact gate/up round-trip hashes. 5. State provenance at layer `L` names the sealed quantized predecessor state; layer 77 never advances into layer 78. 6. Pack/unpack and repeated extension reconstruction are bit-exact; the sealed extension path is the path executed; joint encode then load-time slice reconstructs identically; permutations and shared vectors pass their oracles. 7. Carried tensor payloads are SHA-256 identical and the final index/manifest is bijective. 8. The official installed-layer audit and repeated successor-forward audit pass under the sealed deterministic runtime. 9. Same-harness KLD-5x plus Estonia/LAVD/alpha gates pass before acceptance.