manjunathshiva/Qwen3.8-27B-tq3-mini-g64

Qwen3.8-27B — TurboQuant 3-bit, 16 GB build (MLX)

Data-free 3-bit quantization of Qwen/Qwen3.8-27B sized to fit a 16 GB Apple Silicon Mac, via TurboQuant-MLX.

Qwen3.8-27B is a dense 27.8B multimodal model with a hybrid attention stack: 64 decoder layers, of which 48 are Gated DeltaNet (linear attention) and 16 are full attention. It carries a 248,320-token vocabulary and a 27-block vision tower.

• 11.55 GiB on disk, from 55.6 GB at bf16 — a 4.8× reduction.
• Verified on a real 16 GB M4 Mac mini — every prompt length from 220 to 5,017 tokens plus vision, peaking 12.95–13.61 GiB, not projected from a bigger machine.
• Passes the Opencode agentic loop from 24 GB up — observe → read → edit → verify. It does not work agentically on 16 GB: 0 for 4 there, against 4 for 4 with the same weights on a larger Mac. See Agentic.
• 4/4 on the vision battery (with an image-size budget, see below).
• Decode is 3.7 tok/s on the 16 GB mini, 12–13.6 tok/s on an M4 Max. Read that number before you commit to this build — see Speed.

If you have more than 16 GB, take the 4-bit build instead. This one trades +10.4% perplexity for 3.6 GiB. That is a real quality cost, worth paying only if 4-bit does not fit your machine.

Will it fit my Mac?

Real output from the shipped planner for this model on a 16 GB machine:

Model
  type                 qwen3_5  (turboquant 3-bit g64)
  weights (exact)      12.40 GB

Machine  (assumed, not this one)
  Metal working set    11.34 GB   ← the real ceiling
  system RAM           17.18 GB

Projection at 8,192 tokens of context
  weights              12.40 GB
  KV cache             0.54 GB  (64.0 KB/token, hybrid: 16/64 full-attention)
  prefill workspace    0.10 GB  (estimate, at --prefill-step-size 256)
  runtime reserve      1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 14.04 GB of 15.46 GB usable   1.42 GB headroom

Verdict: ⚠️  RESIDENT — fits, but only after raising the Metal wired cap

Install:

pip install "turboquant-mlx-full[vlm]>=0.25.0"

Two settings are required on a 16 GB machine:

sudo sysctl -w iogpu.wired_limit_mb=14336   # Metal's default cap is the binding limit,
                                            # not your RAM. Resets on reboot.
python -m turboquant_mlx.generate_vlm --model <this-repo> \
    --prompt "..." --prefill-step-size 256

Use 14336, not the 13863 the planner suggests above. The planner projects the workspace at 8K context; the measured peak on a 5,043-token prompt is 13.81 GiB, which is 0.27 GiB over a 13863 MiB (13.54 GiB) cap and would fail on long prompts. 14336 MiB (14.00 GiB) clears every length tested, with 0.19 GiB to spare at 5K — the same margin Muse Glimmer runs on.

--prefill-step-size 256 is not optional here. The default (2048) needs 3.30 GB of transient prefill workspace at 8K context instead of 0.10 GB, and 256 also keeps every TurboQuant layer on the fused Metal kernel, which materializes nothing.

Measured peak with those settings, which is what has to stay under the cap. The mini column is a real 16 GB M4 Mac mini, not a projection:

prompt	peak, 16 GB M4 mini	peak, 64 GB M4 Max	prefill (mini)	decode (mini)
220 tok	12.95 GiB	13.22 GiB	20.7 tok/s	3.9 tok/s
818 tok	12.96 GiB	13.05 GiB	19.0 tok/s	3.9 tok/s
2,014 tok	13.14 GiB	13.30 GiB	21.0 tok/s	3.6 tok/s
5,017 tok	13.61 GiB	13.81 GiB	20.6 tok/s	3.7 tok/s

At the longest prompt that leaves 0.39 GiB of headroom under the 14.00 GiB cap. Peaks on the mini run consistently 0.09–0.27 GiB below the same prompts on the M4 Max, so the larger machine is, if anything, the pessimistic estimate.

Those peaks are repeatable to the hundredth of a GiB — two independent sweeps returned 12.96, 13.14 and 13.61 identically. Decode matched to 0.01 tok/s; only prefill moved, by about 7%.

Close other apps before a long prompt. At 0.39 GiB of headroom they are not background noise, and the very first load after boot is the tightest moment of all — it faults 11.55 GiB off disk while Metal wires the same pages. If a first run is killed, simply run it again; the second starts from a warm page cache.

Speed

Fitting is not the same as being fast, and on a 16 GB mini this model is not fast:

	16 GB M4 mini	64 GB M4 Max	ratio
prefill	19–21 tok/s	~81 tok/s	3.9×
decode	3.6–3.9 tok/s	~12.8 tok/s	3.4×

That gap is hardware, not a quantization artifact. The prefill ratio of 3.9× tracks the GPU core count almost exactly (10 vs 40). Decode's 3.4× is better than the 4.55× memory bandwidth ratio (120 vs 546 GB/s) would predict, because TurboQuant decode carries real arithmetic — the online Hadamard rotation and codebook lookup — so it is not purely bandwidth bound, and the compute portion partly hides the slower memory.

What 3.7 tok/s means in practice: a 200-token answer takes about a minute, and a 5,000-token prompt takes four minutes to prefill before the first token appears. That is fine for batch summarization, document Q&A and vision calls you walk away from. It is slow for interactive chat, and it is not enough for an agent loop — that was budgeted at roughly half an hour on this decode ratio, then measured, and it does not complete at all on 16 GB. See Agentic.

Quality

WikiText-2 perplexity, teacher-forced, 32 × 512-token chunks:

build	size	PPL	vs 4-bit
tq4-g64	15.15 GiB	7.2496	—
this build (tq3, 4-bit extras)	11.55 GiB	8.0052	+10.4%
tq3 with 8-bit extras	12.88 GiB	8.0551	+11.1%
tq3a-tq2e (2-bit MLP)	9.16 GiB	10.6549	+47% ❌

Two things that table settles:

Going below 3-bit does not work on this model. A 2-bit-MLP hybrid gets to 9.16 GiB and costs 47% perplexity — the same failure seen on other dense models, where a 2-bit dense MLP has none of the redundancy that lets 2-bit and ternary work on high-expert-count MoEs. 1.58-bit is not reachable at all: TurboQuant's ternary tier applies only to MoE expert weights, and this model has none.

The extras tier is free. Storing the embeddings, lm_head and vision tower at 4-bit affine rather than 8-bit saves 1.33 GiB and costs nothing measurable (8.0551 → 8.0052 — not worse, and too small a difference to call better). Both builds quantize exactly the same 2,844 tensors. The polar tier decides quality; the extras tier decides size.

Agentic — works from 24 GB up, not on 16 GB

Opencode against turboquant-serve-vlm, on a repo with a planted off-by-one in average() and a failing pytest suite. Pass = ran the given pytest command, read stats.py, made the correct minimal edit, re-ran to green, explained with file:line, and left the tests byte-identical.

machine	prefill step	peak @7.2K-token agent prompt	client	result
64 GB M4 Max	256	—	opencode 1.17.13	pass, 7m23s
64 GB M4 Max	2048	19.13 GiB	opencode 1.18.18	pass, 5m38s
64 GB M4 Max	256	14.04 GiB	opencode 1.18.18	pass, 5m24s
16 GB M4 mini	256	—	opencode 1.18.18	0 for 4

This build does not drive an agent on 16 GB. Four attempts, four failures, in three distinct modes: a prose preamble then EOS with no tool call; a correct tool call followed by a token-queue timeout; and a 4,096-token perseveration loop that degenerated into multilingual nonsense and ended in a Metal out-of-memory.

The compression is not the cause — these exact weights pass on a larger machine at both prefill-step settings, including the 256 that a 16 GB machine is forced to use. What has not been separated is 16 GB of RAM versus the different mlx-vlm build on that mini. Either way the recommendation is the same, because even a passing run would be unusable: an agent's ~7,200-token system prompt is re-sent every turn at ~20 tok/s on the mini — six minutes before the first token — and prefix caching cannot amortize it (see below).

24 GB is the entry tier for agentic use, and at that size prefer tq4-g64: 17.63 GiB at step 256, also tested-passing, at a better perplexity (7.2496 vs 8.0052). Use this build agentically when you want the extra headroom.

Prefix caching is a NO-GO on this architecture

Do not enable mlx-vlm's APC (APC_ENABLED=1) here. Measured: cached_tokens=0 on every request, APC harvest failed ... Insufficient Memory, then a fatal OOM inside _clone_prompt_cache_for_apc. The reason is structural — APC's own self-check reports 48 ArraysCache layers (cap=checkpoint) against 16 KVCache layers (cap=pageable). Gated-DeltaNet recurrent state cannot be sliced into reusable blocks, so the cache has to snapshot all 48 layers whole. The architecture that makes long prompts cheap here is the same one that stops you caching them.

turboquant-serve-vlm --model <this-repo> --port 8080 --prefill-step-size 256

Serve with the model's own sampling defaults. Setting temperature explicitly made things strictly worse in testing (0.55 produced the perseveration loop above); generation_config.json supplies temperature 1.0 / top_p 0.95 / top_k 20 coherently, and mlx-vlm applies them for any field the client omits.

Vision

Four synthetic images with ground truth drawn in, so scoring is objective: 4/4 — OCR, counting, chart reading, spatial relations.

Vision works on the 16 GB mini, within an image-size budget. Peak scales with pixel area, and area is the knob you have:

image	pixels	peak	machine	OCR
350×100	35 K	12.44 GiB	16 GB M4 mini	✅ correct
700×200	140 K	12.68 GiB	16 GB M4 mini	✅ correct
640×420	269 K	14.92 GiB	64 GB M4 Max	over a 14 GiB cap ❌

Both mini sizes read the target text correctly with well over a gigabyte to spare. The 640×420 battery image was never going to fit — and since mini peaks track ~0.15 GiB under the M4 Max, it would still be about 14.8 GiB there, over the cap. Keep images under roughly 150 K pixels on 16 GB, or run vision on a larger machine.

Method

randomized Hadamard rotation   (Gaussianize the weight distribution)
  → group along the input dim  (group size 64)
  → per-group RMS scale → clip → nearest Lloyd-Max centroid
  → 3-bit pack

Embeddings, the vision tower and lm_head are 4-bit affine; everything else is 3-bit TurboQuant. lm_head is deliberately kept off the polar path — at a 248K vocabulary it would add +13 GiB of transient peak above the fused kernel's 256-token bound, which alone would put this build far over a 16 GB machine. No calibration data is used at any point.

Reproduce:

python -m turboquant_mlx.convert_vlm \
  --hf-path Qwen/Qwen3.8-27B \
  --mlx-path ./Qwen3.8-27B-tq3-mini-g64 \
  --bits 3 --group-size 64 --quantize-extras --extras-bits 4

Measured on

• Apple M4 Mac mini, 16 GB, macOS 26.5.2 — the peak, prefill, decode and vision numbers on this page, at iogpu.wired_limit_mb=14336 and --prefill-step-size 256.
• Apple M4 Max, 64 GB, macOS 26.5.2 — perplexity, the Opencode run, and the comparison columns.

turboquant-mlx 0.24.0 · mlx-lm 0.31.3 · mlx-vlm 0.6.3

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B. The upstream LICENSE ships in this repo unchanged.

Citation

@software{turboquant_mlx,
  title  = {TurboQuant-MLX: Data-free weight quantization for MLX on Apple Silicon},
  author = {Manjunath Janardhan},
  year   = {2026},
  url    = {https://github.com/manjunathshiva/turboquant-mlx}
}