Field notes + I/O fix: RAM-resident calibration state, per-expert row memo, selected-rows-only split hashing, vectorized sketch

FIELD_NOTES.md

@@ -0,0 +1,86 @@
+# Field notes: making this run on real hardware
+
+This file records every problem hit while running the encoder on rented GPUs
+for the first time, what caused it, and what fixed it. All fixes are already
+applied in the code and scripts in this repository. If you are reproducing
+this work, you should not have to rediscover any of these.
+
+Two runs were used: 8x B200 (vast.ai) and 6x B300 SXM6 (runpod).
+
+## Environment problems
+
+**Torch 2.11 is gone from the CUDA 13.2 index.** The original build used
+torch 2.11.0 with a prebuilt exllamav3 0.0.43 wheel. Those wheels are no
+longer downloadable, and the prebuilt extension will not load against a newer
+torch (it fails with an undefined symbol on at::cuda::getCurrentCUDABlasHandle).
+Fix: build the extension from the v0.0.43 source tag against whatever torch you
+are on. The source archive is in run_scripts/. Set TORCH_CUDA_ARCH_LIST to your
+architecture: 10.0 for B200, 10.3 for B300.
+
+**Transformers must be pinned to 5.12.1.** The runtime adapter imports
+GlmMoeDsaNaiveMoe from transformers.models.glm_moe_dsa.modeling_glm_moe_dsa.
+That class was refactored away in later 5.x releases, so 5.14.1 fails at
+import. setup_box.sh pins the correct version.
+
+**Remove torchvision and torchaudio before running.** Cloud images commonly
+ship them built against an older torch. Transformers imports torchvision if it
+is present, and the ABI mismatch raises "operator torchvision::nms does not
+exist" during model import. setup_box.sh uninstalls both and asserts they are
+gone. On Ubuntu you need --break-system-packages or the uninstall silently
+does nothing.
+
+**NVIDIA apt repo conflicts.** Some images already carry the CUDA repository.
+Adding the keyring package again produces a Signed-By conflict that breaks
+apt entirely. Check for an existing entry before adding one.
+
+## Code problems found by running it
+
+**The provenance collector initialized CUDA.** Reading the cuDNN version
+through torch.backends.cudnn.version() initializes a CUDA context. The runtime
+then refuses to start, correctly, because its determinism contract requires a
+clean CUDA state. Both behaviors were intentional and they collided. Fix: the
+collector now parses the cuDNN build string from torch.__config__.show(),
+which touches no device.
+
+**Startup verification was single threaded.** Every launch re-hashes every
+shard of both checkpoints before doing any work. On 1.7 TB that took about 26
+minutes each time, including every failed attempt. Fix: the shard hashing now
+runs 16 ways in parallel, which brings it to roughly 3 minutes with identical
+failure semantics.
+
+**The search phase re-read the entire calibration state from disk.** This was
+the expensive one. Layer 3 sat for over 90 minutes with all six GPUs idle and
+about 70 CPU cores busy. The process had read 607 GB against a 13 GB working
+set, meaning the same state files were being re-read roughly 47 times: once
+per row request, and the search phase makes dozens of those per layer. On top
+of that, the fit/holdout split was computed by hashing every row in the corpus
+for every expert, when only the rows that expert actually selected matter
+(about 3 percent of them). Fix: bound batches are loaded once per layer and
+held in RAM, per-expert row selections are memoized, and the split hash runs
+only on selected rows. The diagnosis tool that found it was reading
+/proc/PID/io on the live process; if a phase looks slow with idle GPUs, check
+read_bytes before assuming the math is slow.
+
+## Things that are slow by design, not bugs
+
+The carried-prefix pass at the start of the walk runs the corpus through the
+three carried dense layers on one GPU. It is bounded, one time, and writes
+crash-consistent state for every prompt, so it is dominated by hashing and
+file writes rather than GPU math.
+
+The inventory hashing is single threaded per file stream. It is the price of
+the provenance guarantee that the walk encoded exactly the bytes it claims.
+
+## Performance measured on real hardware
+
+Encode cost per tensor is about 0.85 s on B300 and 0.74 s on a clean B200, and
+it is flat across 3, 4, and 5 bits. The trellis dynamic program is integer and
+occupancy bound, so B300 gives no per-GPU advantage over B200 on this workload
+despite its much faster FP4 and FP8 tensor cores. Plan capacity by GPU count,
+not by generation.
+
+One GPU on a contended host was measured at 0.84 s/tensor while a foreign
+workload held it at 100 percent utilization, versus 0.74 s clean. If you find
+GPUs busy that you did not start, they are probably another tenant's; you
+cannot clear them from inside a container, and the cost is smaller than you
+would expect, but you should still open a support ticket.

r7_encoder/glm52_backend.py

@@ -661,7 +661,29 @@ class GLM52Backend(Round7Backend):
             else "fit"
         )
 
+    def _bound_batches_cached(self, capture: LayerCapture, shards):
+        """Load the layer's bound batches ONCE into RAM and reuse.
+
+        The prior design re-read 1,773 state files plus 1,773 routing
+        sidecars from disk on every row request; the search phase issues
+        dozens of such requests per layer. One resident copy (~13 GB
+        against 2 TB of RAM) serves them all. Keyed by capture digest so
+        layer N+1 replaces layer N; contents are treated as read-only.
+        """
+        key = (capture.digest, tuple(shard.shard_id for shard in shards))
+        cached = getattr(self, "_bound_cache", None)
+        if cached is not None and cached[0] == key:
+            return cached[1]
+        batches = list(self._iter_bound_batches_raw(capture, shards))
+        self._bound_cache = (key, batches)
+        # a new layer invalidates every per-expert row memo as well
+        self._expert_rows_memo = {}
+        return batches
+
     def _iter_bound_batches(self, capture: LayerCapture, shards: Iterable[StateShard]):
+        yield from self._bound_batches_cached(capture, tuple(shards))
+
+    def _iter_bound_batches_raw(self, capture: LayerCapture, shards: Iterable[StateShard]):
         manifest = read_json(capture.routing_dir / "CAPTURE.json")
         records = manifest["sidecars"]
         for shard in tuple(shards):
@@ -696,22 +718,45 @@ class GLM52Backend(Round7Backend):
 
         if split not in ("fit", "holdout"):
             raise ValueError("row split must be fit or holdout")
+        memo = getattr(self, "_expert_rows_memo", None)
+        if memo is None:
+            memo = self._expert_rows_memo = {}
+        memo_key = (capture.digest, expert, split)
+        cached_rows = memo.get(memo_key)
+        if cached_rows is not None:
+            yield from cached_rows
+            return
+        collected = []
         for _, batch, ids, weights, row_ids in self._iter_bound_batches(
             capture, shards
         ):
             chosen = ids == expert
             selected = chosen.any(dim=1)
-            split_mask = torch.tensor(
+            if not selected.any():
+                continue
+            # The split only matters where the expert selected the row, so
+            # compute the per-row hash on the ~selected subset instead of the
+            # full batch. Identical outcome: mask == selected & split_mask.
+            sel_index = selected.nonzero(as_tuple=True)[0]
+            keep = torch.tensor(
                 [
-                    self._split(capture.layer, expert, int(value)) == split
-                    for value in row_ids
+                    self._split(capture.layer, expert, int(row_ids[int(i)]))
+                    == split
+                    for i in sel_index
                 ],
                 dtype=torch.bool,
             )
-            mask = selected & split_mask
-            if mask.any():
+            final_index = sel_index[keep]
+            if final_index.numel():
                 route_weight = (weights * chosen).sum(dim=1)
-                yield ExpertRows(batch.hidden[mask], route_weight[mask], row_ids[mask])
+                rows = ExpertRows(
+                    batch.hidden[final_index],
+                    route_weight[final_index],
+                    row_ids[final_index],
+                )
+                collected.append(rows)
+                yield rows
+        memo[memo_key] = tuple(collected)
 
     def _fallback_locations(
         self,

r7_encoder/search.py

@@ -270,17 +270,30 @@ class SearchRunner:
 
         matrix = torch.as_tensor(weight).detach().to("cpu", dtype=torch.float32)
         n, k = matrix.shape
-        buckets = [0.0] * 32
-        absolute = 0.0
+        # Same 512 seeded coordinates and same arithmetic as the original
+        # scalar loop, gathered in ONE batched read. The scalar version issued
+        # 512 individual element reads per call (a device sync each when the
+        # weight lived on CUDA), which dominated the entire search phase.
+        rows = []
+        cols = []
         for index in range(512):
             coordinate = derive_seed(seed, index, bits=64)
-            row = coordinate % n
-            column = (coordinate // n) % k
-            value = float(matrix[row, column]) * float(suh[column]) * float(svh[row])
-            buckets[index % len(buckets)] += value
-            absolute += abs(value)
+            rows.append(coordinate % n)
+            cols.append((coordinate // n) % k)
+        row_t = torch.tensor(rows, dtype=torch.long)
+        col_t = torch.tensor(cols, dtype=torch.long)
+        suh_t = torch.as_tensor(list(suh), dtype=torch.float32)
+        svh_t = torch.as_tensor(list(svh), dtype=torch.float32)
+        values = (
+            matrix[row_t, col_t].double()
+            * suh_t[col_t].double()
+            * svh_t[row_t].double()
+        )
+        buckets = torch.zeros(32, dtype=torch.float64)
+        buckets.index_add_(0, torch.arange(512, dtype=torch.long) % 32, values)
+        absolute = float(values.abs().sum())
         denominator = max(absolute * absolute, 1e-30)
-        return sum(value * value for value in buckets) / denominator
+        return float(buckets.square().sum()) / denominator
 
     def _proxy_expert(
         self, search: LayerSearch, expert: int, diagonal: Sequence[float] | None = None