manjunathshiva/Laguna-S-2.1-tqTe-g64

Laguna-S-2.1 — TurboQuant ternary (1.58-bit experts, MLX)

Data-free ternary (1.58-bit) expert quantization of poolside/Laguna-S-2.1 for Apple Silicon via TurboQuant-MLX. Laguna S-2.1 is a 118B-parameter Mixture-of-Experts (256 experts, top-10, +1 shared, moe_intermediate 1024) built by Poolside for agentic coding and long-horizon work.

The headline: a 118B model that runs resident on a 64 GB Mac.

• 27 GB on disk, ~30 GB peak resident on a 64 GB machine — leaving ~25 GB for the OS. It would fit a 40 GB Mac. For comparison, affine 4-bit of this model is ~64 GB and will not load resident on 64 GB at all (it exceeds Metal's ~56 GB working-set limit and crashes the machine); even TurboQuant's own 3-bit build (48 GB) peaks at 52.7 GB and starves the OS. Ternary is the only build that fits resident with real headroom.
• Data-free: no calibration set. Experts are quantized to a ternary {−c, 0, +c} codebook (base-3 trit packing, ~1.6 bits/weight); attention stays at 3-bit. Randomized Hadamard rotation → per-group RMS scale → nearest centroid, decoded inline by fused Metal kernels.
• Why it holds at ~1.6 bits where XS.2 didn't: sub-2-bit quality on MoE needs wide experts, not just many. S-2.1's experts are 1024-wide (2× XS.2's 512); at ternary, XS.2 perseverated and S-2.1 stays coherent across every stress test below.

How to run

pip install "turboquant-mlx-full>=0.18.1"  # Laguna support; 0.17.0/0.18.0 are yanked
# generate (resident on a 64 GB Mac)
python -m turboquant_mlx.generate --model <this-repo> \
    --prompt "Write a mergesort in Python." --temp 0.7 --top-p 0.9 \
    --max-tokens 1024 --no-think
# OpenAI-compatible server
turboquant-serve --model <this-repo> --port 8080 \
    --temp 0.7 --top-p 0.9 --prompt-concurrency 1 \
    --chat-template-args '{"enable_thinking": false}'

Pass --no-think unless you want the reasoning trace. Laguna thinks out loud by default, so a plain generate call can spend its whole token budget deliberating and never reach the answer. The server equivalent is --chat-template-args '{"enable_thinking": false}' (already in the command above).

Always pass --top-p 0.9. Laguna's generation_config ships top_p: 1.0; the untruncated 100k-vocab tail injects rare junk tokens.

Do not pass --kv-bits. 36 of 48 layers are sliding-window(512), so KV stays tiny; KV-quant saves nothing here and costs decode speed.

The transformers tokenizer warning is harmless — ignore it. Recent transformers prints "incorrect regex pattern … this will lead to incorrect tokenization" and suggests fix_mistral_regex=True. Don't. This repo's tokenizer.json is structurally identical to upstream poolside/Laguna-S-2.1 (same vocab, merges and pre-tokenizer), so the default is how the model was trained and what every other runtime uses. Setting the flag changes case-aware splitting (apoA, GeV, iPhone) on ~1% of real prompts and would make results diverge from every other Laguna deployment. All benchmarks below were run without it.

64 GB is the practical floor for resident use. At ~30 GB peak it also runs on 40–48 GB Macs with headroom. On a 16 GB Mac, use expert streaming instead — verified on an M4 mini at a 7.3 GB peak (see Expert streaming below).

Will it fit my Mac?

Don't guess, and don't download 28.7 GB to find out — ask first (needs turboquant-mlx-full >= 0.18.1):

turboquant-plan --model manjunathshiva/Laguna-S-2.1-tqTe-g64

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use. On a 64 GB M4 Max:

Model
  type                 laguna  (turboquant 3-bit g64)
  weights (exact)      28.70 GB

Machine
  Metal working set    55.66 GB   ← the real ceiling
  system RAM           68.72 GB

Projection at 16,384 tokens of context
  weights              28.70 GB
  KV cache             0.88 GB  (52.5 KB/token, hybrid: 12/48 full-attention, 36 sliding (window 512))
  prefill workspace    3.22 GB  (estimate, at --prefill-step-size 2048)
  runtime reserve      1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 33.80 GB of 61.85 GB usable   28.05 GB headroom

Verdict: ✅ RESIDENT — fits fully in memory

Note the two ceilings it separates. The Metal working set (55.66 GB here, not the 68.72 GB of installed RAM) is what actually decides the verdict — which is exactly why the affine 4-bit and 3-bit builds in the table below fail on this machine while this one has 28 GB to spare.

On a machine too small to hold it, the verdict changes rather than just failing. This is the real output from a 16 GB M4 Mac mini, where the model does not fit by a wide margin:

Model
  type                 laguna  (turboquant 3-bit g64)
  MoE                  256 experts, top-10, 48 layers
  weights (exact)      28.70 GB
    experts            26.42 GB  (streamable)
    resident           2.28 GB  (attention, embeddings, routers)

Machine
  Metal working set    12.71 GB   ← the real ceiling
  system RAM           17.18 GB

Projection at 16,384 tokens of context
  weights              28.70 GB
  KV cache             0.88 GB  (52.5 KB/token, hybrid: 12/48 full-attention, 36 sliding (window 512))
  prefill workspace    0.20 GB  (estimate, at --prefill-step-size 128)
  runtime reserve      1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 30.78 GB of 15.46 GB usable   OVER by 15.32 GB

Verdict: ⚠️  STREAMING — too big to hold; experts page from disk

The line that matters is the split above: only 2.28 GB of this model has to stay resident — the attention layers, embeddings and routers. The other 26.42 GB is expert weights, and in a sparse MoE those are read on demand. That is why the streaming peak in Expert streaming below is a fraction of 28.7 GB, and why a 16 GB mini gets a ⚠️ rather than a ❌.

To plan for a machine you're not sitting at, pass its numbers directly: --wired-gb 12.7 --ram-gb 16.

Needs turboquant-mlx-full >= 0.18.1 for the streaming verdict on Laguna specifically. Earlier releases matched expert tensors by the switch_mlp name only, so Laguna's mlp.experts container counted as resident and the planner reported ❌ on small Macs. Resident projections were always correct.

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Quality (6-prompt stress battery, temp 0.7 / top-p 0.9, resident)

test	result
1500-word essay	✅ coherent, no repetition
multi-step arithmetic	⚠️ meeting-time correct, distance signal missed
code + unit tests	✅ function + asserts
long-context needle recall	✅ retrieved
strict formatting	✅ exactly 5 lines, exited thinking
repetition trap	✅ no loop

5.5 / 6. The one soft spot — arithmetic distance — is the known ternary weakness; adding bits (2-bit or down-proj-4-bit builds) did not fix it, so it is a quantization-floor effect, not a build bug. Everything else, including the long-form coherence that ternary XS.2 failed, is clean.

Standard accuracy (resident, no-think, temp 0.7; harness in scripts/benchmarks/):

benchmark	this build (ternary)	samples
MMLU-Redux 2.0	79.5% (95% CI 75.3–83.2)	396 (7 × 57 subjects)
GSM8K	82.5% (95% CI 76.6–87.1)	200

At 1.58-bit experts this 118B still edges the 33B Laguna-XS.2 tq3 (3-bit) on both (MMLU 79.5 vs 76.9, GSM8K 82.5 vs 79.0) — the larger, wider expert pool absorbs the deeper quantization.

Why ternary, and what the other builds cost

The full low-bit sweep on this model (all resident-tested on a 64 GB Mac, same 6-prompt battery):

build	size	peak	quality	verdict
affine 4-bit	~64 GB	—	—	won't load resident (exceeds Metal cap)
TurboQuant 3-bit	48 GB	52.7 GB	—	starves OS / crashes; streaming-only
★ ternary (this)	27 GB	30 GB	5.5/6	smallest + best-fitting; ships
2-bit experts	32 GB	35 GB	5.5/6	ties quality, +5 GB — no reason over ternary
ternary + 4-bit down_proj	37 GB	41 GB	5/6	larger and worse math

Ternary is both the smallest and the best-quality resident build — the sweep has a clean winner.

Agentic coding (Opencode) — passes at 1.58-bit

On the standard planted-bug task (run tests → locate the off-by-one → minimal edit → re-run), this ternary 118B completes the full loop cleanly in ~2 minutes: ran the exact pytest command, read both files, made the correct one-line fix (len(values) + 1 → len(values)), re-ran to green (3 passed), and explained the bug. No perseveration, no hallucinated tool args.

This is notable because the 35B ternary (a narrower 512-wide expert build) fails the same harness (across-turn looping + ungrounded tool calls). S-2.1's 1024-wide experts appear to restore agentic grounding, not just coherence — the same width effect that lets it sustain 1.58-bit at all. Serve with --top-p 0.9 --prompt-concurrency 1 and {"enable_thinking": false}; tool calls use Laguna's native GLM-XML (TurboQuant-MLX ships the glm47 trailing-newline parser fix).

Speed

Resident on an M4 Max (64 GB): decode ~12.5 tok/s. As with all TurboQuant codebook builds, this is ~2× slower per token than MLX-native affine 4-bit — the cost is compute (online Hadamard rotation + codebook decode each forward pass), not bandwidth. Here that tradeoff is not really a choice: affine 4-bit cannot load resident on this machine class at all, so ternary is what makes a 118B model runnable resident in the first place.

Expert streaming — bounded RAM on small Macs

Streaming is wired up for Laguna, so the resident 27 GB is not the floor. Experts page from disk on demand and only the router-selected ones are ever held:

python -m turboquant_mlx.stream.stream_generate --model <this-repo> \
    --cache-budget-gb 4 --max-active-experts 0 --prompt "..." --max-tokens 256

Verified on a 16 GB M4 Mac mini (256 GB internal SSD, Metal working set 12.71 GB), 141 of 144 expert projections streamed (layer 0 is dense). Peak memory is the cache budget plus a constant 3.33 GB, which held to within 40 MB across every run:

--cache-budget-gb	peak	tok/s	hit rate	disk read / token
2	5.31 GB	1.15	39.8%	0.65 GB
4	7.32 GB	1.26	56.2%	0.48 GB
auto (5.89)	9.23 GB	1.36	66.1%	0.37 GB
8	11.35 GB	1.58	75.6%	0.27 GB

Peak at the 4 GB setting matches the 7.3 GB measured on a 64 GB M4 Max — the footprint is set by the cache budget and the model's sparsity, not by the host.

8 GB is the fastest setting that fits a 16 GB mini, and auto is conservative here: it picks 5.89 GB and leaves ~15% of the speed unused. Going 2 GB → 8 GB cuts disk reads 60% but gains only 37% speed, because most of the time is not disk — on a 10-core GPU the ternary expert decode dominates, and this machine tops out near 1.6 tok/s regardless of cache. A faster SSD will not move that; more GPU cores will.

Use --max-active-experts 0 to keep native top-10 routing; lowering it reads less from disk but changes which experts run (and breaks agentic tool-calling).

Needs turboquant-mlx-full >= 0.18.1. Earlier versions keyed the expert container on mlp.switch_mlp only, so on Laguna's mlp.experts (SwitchGLU) the loader swapped 0 projections and silently fell back to loading all 27 GB resident — fine on a 64 GB Mac, an out-of-memory crash on a 16 GB one. Check the loader's [stream] swapped N expert projections line: swapped 0 means streaming did not engage. 0.17.0 and 0.18.0 are yanked for an unrelated packaging fault that breaks every command on install.

Provenance

• Base: poolside/Laguna-S-2.1, released by Poolside under the OpenMDW License v1.1 — a permissive model licence (unrestricted use, modification and distribution, with a patent-litigation termination clause and no restrictions on generated outputs). Its one distribution obligation is to carry the agreement along, so a verbatim copy of Poolside's LICENSE.md ships in this repo and applies to these derived weights.
• Converter: python -m turboquant_mlx.convert --hf-path poolside/Laguna-S-2.1 --mlx-path out --attn-bits 3 --ternary-experts --group-size 64 --streaming.
• MLX Laguna support (model port + loader) is provided by TurboQuant-MLX's compat shim; mlx-lm has no native laguna model class as of 0.31.3.

Copyright 2026 Manjunath Janardhan. Model weights derive from poolside/Laguna-S-2.1 and remain subject to the OpenMDW-1.1 licence in LICENSE.md; Poolside's copyright and notices of origin are retained per that agreement.