Results & verification

All local runs use the MLX 4-bit model (fft_mlx4, ~~1 GB) + the fp32 pooler, on an 8 GB MacBook at ~36 tok/s. The PyTorch/MPS fp16 path also works but swaps on 8 GB (~~2–5 s/tok) — MLX 4-bit is ~180× faster here. Sampling is temp=0.6 unless noted.

1. The MLX port is faithful

pooler_mlx.py is a manual MLX re-implementation of the PyTorch AttnPoolSP (packed-QKV attention, head-averaged mass for eviction). Verified against the original on random input: SP max abs diff = 2.4e-7, eviction-mass correlation = 1.0.
The base LLM is converted to MLX 4-bit (4.5 bits/weight) and generates coherently — the FFT weights survive quantisation.

2. Single-turn reasoning battery (`sp_mlx.py`)

Six word problems, auto-graded on the boxed final answer, bounded SP-evict path:

problem	answer	result
30 animals / 74 legs	23 chickens, 7 cows	✅
Natalia clips (48 + half)	72	✅
train 60 km / 1.5 h → 4 h	160 km	✅
17 × 24	408	✅
20% discount → $20	$25	✅
taller-than logic	Sam	✅

Accuracy is high but stochastic, not perfect. Re-sampling the chickens problem 8× gave 7/8 with a correct \boxed{23}\boxed{7}, plus separately one clearly wrong sample (28/2, 64 legs) and one that rambled to the length cap without concluding. So treat single-turn accuracy as ~80–90%, sampled, not deterministic. top_p=0.95 and chat-template wrapping did not improve termination or accuracy.

Termination (EOS) is stochastic and length-cap sensitive — with enough room (gen_len=8000) a run EOS'd cleanly at 740 tokens; other runs over-think past 1500. EOS reliability would need more training.

3. Compression is active; eviction is a separate (untested-at-normal-length) regime

Instrumented generate_sp:

With rw=128, every step compresses the distant transcript (everything older than the last 128 tokens) into 32 SP vectors. On an 8000-token budget run the distant buffer reached max_kept=576 → ~18× compression actually fed to the model. Compression is genuinely exercised in every test.
Eviction (dropping lowest-mass tokens) only fires once the distant buffer exceeds maxD=4096, i.e. only for very long generations. Normal runs (max_kept 576) never reach it.
Forcing it with maxD=256 fired eviction 40× (mechanism works) but the tight bound degraded the reasoning into a non-converging trial-and-error search. So aggressive eviction has a real quality cost.

4. Multi-turn "long-CoT rally" (`mt_mlx.py`) — and the `rw` fix

A 5-turn dependent-arithmetic chat where each answer feeds the next: 3+5=8 → −2=6 → ×2=12 → −4=8 → ÷2=4.

recent window	outcome
`rw=128`	turns 1–2 correct (state in the raw window); turn 3 fails — "no number given" (the running `6` fell into the lossy SP and was lost)
`rw=512`	all 5 turns correct — e.g. turn 3: "starts with 6 after giving away 2 from 8 … 6×2=12"; turn 5: "after eating 4, 8 left, split by 2 → 4 each"

This is the cleanest demonstration of the core finding: specific values must live in the raw window, not the SP. Widening rw keeps more of the recent state verbatim → multi-turn carry works. Cost is a larger (still O(1)) KV constant and less actual compression — rw trades compression for retention, and on the retention/quality axis larger rw is monotonically better (the SP only ever loses info vs raw).

Caveat: the model was trained at rw=128, so rw=512 at inference is slightly out-of-distribution; ideally one would train at the target window. The multi-turn result above is a single sample.

5. The compression fidelity floor (training)

Measured during training, deep-position cross-entropy (deepCE, positions ≥ rw+512) vs a full-attention reference computed by full_kv_ce.py on the same held-out split:

full-attention floor: deepCE ≈ 0.56 (ppl ≈ 1.76)
QLoRA r=64: plateaued ≈ 0.67–0.68
Full fine-tune (FFT): plateaued ≈ 0.65 (val 0.66)

The ~0.09 gap to full attention does not close with more LLM capacity (LoRA→FFT gained ~0.02). This is the inherent cost of squeezing the distant context into 32 vectors — consistent with the gist-vs- verbatim behaviour seen in generation.

Honest summary

Works: bounded-memory long single-turn CoT (coherent, mostly-correct, often EOS); multi-turn state carry with an adequate raw window; genuine soft-prompt compression; fast laptop deployment.
Limits: ~0.09 deepCE gap to full attention that capacity can't close; verbatim facts/proper nouns lost to the SP; stochastic accuracy and termination; tight eviction degrades reasoning.
Knobs: rw = compression ↔ retention; maxD = bounded-buffer cap (eviction); n_sp=32 SP budget.