crucible-labs/Qwen3.6-35B-A3B-REAP-48-v2-GGUF

Qwen 3.6 35B-A3B · REAP-48 · v2

Second public release from Crucible. Same model, same 8.78 GB, measurably closer to the unquantized weights than v1.

TL;DR

48% of the routed experts pruned with REAP, then quantized with a measured per-tensor bit allocation instead of a hand-written one. Against v1, at the identical file size:

metric vs. f16	v1	v2	improvement
Mean KL-divergence	0.034093	0.019609	−42.5%
Median KL-divergence	0.002489	0.001186	−52.4%
99th-pct KL-divergence	0.536331	0.327945	−38.9%
Excess perplexity over f16	0.037205	0.022679	−39.0%
RMS Δp	6.294%	4.919%	−21.8%
Top-1 agreement with f16	94.678%	96.199%	+1.52pp

Both files are 8.78 GiB. On everything measured — every divergence metric, prefill and decode throughput, and all three benchmarks — v2 is better than v1 or equal to it.

One caveat worth stating plainly: v2 uses i-quants (IQ3_S, IQ4_XS) where v1 used only k-quants. Measured on Metal and ROCm, those are faster here, for a structural reason explained below. They have not been timed on CUDA, Vulkan, or CPU-only builds, and i-quant kernel performance does vary by backend. If you are on an untested backend and throughput matters more than fidelity, benchmark before switching.

What changed

Three things, in order of how much they contributed.

1. Calibration now carries the chat template. v1's importance matrix was computed on conversations flattened to plain text — every <|im_start|>, role marker and thinking tag discarded. An imatrix measures activation statistics, so v1 measured them on text the model never actually receives. v2 renders calibration through the model's own chat template.

2. Bits are reallocated by measurement. v1 held attention and embeddings at Q8_0 and left the shared expert at Q3_K. Measured per-role KL-divergence says that is backwards: the shared expert is ~53× more sensitive per parameter than the routed experts, while attn_q is among the least sensitive attention tensors. v2 moves that budget accordingly.

3. Per-layer allocation. v1 used one type per tensor role across all 40 layers. v2 varies by layer using imatrix energy within each role.

Two of those findings contradict the obvious approach. Raw imatrix energy is not comparable across tensor roles — it clusters by position relative to the nearest RMSNorm, not by importance, so ssm_out measures as the least important tensor in the model while measured KL-divergence ranks it first. And weight scale does not rescue the comparison: σ_W spans only 2.4× across these tensors where energy spans 3,317×.

Allocation

tensor group	v1	v2
routed experts (76% of params)	Q3_K uniform	IQ3_S / IQ4_XS / Q4_K per layer
shared expert (ffn_*_shexp)	Q3_K	Q5_K–Q8_0 per layer
attention (attn_q/k/v/qkv/output)	Q8_0	Q5_K–Q8_0 by measured sensitivity
attn_gate, ssm_out (GDN path)	Q3_K (uncovered by v1's recipe)	Q5_K+
token_embd	Q8_0	Q4_K
output	Q8_0	Q6_K

attn_v and attn_k sit at Q8_0 in v2: they are the most sensitive tensors per parameter in the whole model and, at 10.5M parameters each, nearly free to protect.

Performance — M4 Mac Mini, 16 GB, Metal

Both files measured on the same machine, same llama-bench invocation (-p 512 -n 128 -ngl 999 -fa 1 -r 3), same llama.cpp build:

	v1	v2
prefill (pp512)	411.29 ± 2.56 t/s	417.18 ± 2.43 t/s	+1.4%
decode (tg128)	26.71 ± 0.02 t/s	29.24 ± 0.02 t/s	+9.5%

v2 is faster despite using i-quants, and the reason is structural. Decode is bandwidth-bound and reads the dense path on every token while touching only 8 of 133 experts. v1 held attention, embeddings and the output projection at Q8_0 — 1.774 GiB re-read every token. v2 moves that budget down to Q6_K/Q4_K and spends it on the expert stacks, which are read sparsely. Fewer bytes per token on the hot path, more precision where the measurement says it matters.

Accuracy

Own-harness, temperature=0, one attempt per problem, generated code executed in a rootless-podman sandbox. Measured on gfx1151 / ROCm.

task	v2
HumanEval+	149/164 = 90.9%
MBPP+	286/378 = 75.7%
BFCL-simple	373/400 = 93.2%

These are absolute figures for v2. They are deliberately not presented against v1: those were measured on different hardware and with a harness that has since had two scoring bugs fixed, so any delta would be an artefact as much as a result. The v1-vs-v2 quality claim rests on the KL-divergence table above — same machine, same corpus, same f16 reference, same day.

Speculative decoding (DFlash) — big win on discrete/CUDA-class GPUs, not on Apple silicon

The z-lab DFlash drafter works against this pruned model. It is a dense 8-layer draft model (no experts), 0.328 GB at Q6_K, and its dimensional contract matches exactly (num_target_layers: 40, target_layer_ids: [1,10,19,28,37], hidden_size: 2048).

A 48% expert prune does not break it. Code-token draft acceptance measured 80.5% against this model — effectively the same as the unpruned base model's MTP acceptance (79%). The shared expert carries ~69% of layer output norm, so the residual stream stays close to base even with half the routed experts removed.

Radeon 8060S / gfx1151, ROCm 7.14

--spec-draft-n-max	code	code accept	prose	prose accept
off (baseline)	55.01 t/s	—	54.96 t/s	—
2	79.47 (+44.5%)	83.5%	59.89 (+9.0%)	44.3%
3 — optimum	89.63 (+62.9%)	80.5%	59.26 (+7.8%)	35.7%
4	86.77 (+57.7%)	67.7%	50.78 (−7.6%)	24.4%
8	87.86 (+59.7%)	48.4%	42.88 (−22.0%)	14.3%

llama-server -m Qwen3.6-35B-A3B-REAP-48-v2.gguf   -md qwen36-35b-a3b-dflash-Q6_K.gguf   --spec-type draft-dflash --spec-draft-n-max 3   -ngl 999 -c 16384 -b 2048 -ub 2048 -fa on -ctk bf16 -ctv bf16 -np 1 --jinja

Do not raise n-max past 3. Prose regresses at 4 and loses 22% at 8 — acceptance falls faster than the extra drafted tokens pay for it.

M4 Mac Mini / Metal — DFlash is a regression, do not enable it

--spec-draft-n-max	code	prose
off (baseline)	30.12 t/s	29.91 t/s
2	28.64 (−4.9%)	21.47 (−28%)
3	27.02 (−10.3%)	16.84 (−44%)
4	23.90 (−20.6%)	13.92 (−53%)

Acceptance is identical to the ROCm run (80.7% on code), so this is not a quality problem — the drafter predicts this model just as well on both machines. Closing the books on one speculation round at n=2 shows where it goes:

per round, n=2	M4	8060S
tokens delivered	2.61	2.67
measured round time	91.2 ms	33.6 ms
target forward (3 rows)	~59 ms	~27 ms
drafter + overhead	~32 ms	~6.6 ms
one baseline forward, for scale	33.9 ms	19.7 ms

On the 8060S the drafter costs a third of a forward pass; on the M4 it costs almost a whole one, which cancels the gain. Verification cost contributes too — a 4-row forward is 2.20x a 1-row forward on M4 versus 1.53x on the 8060S — but the drafter overhead is the larger term. Forcing the drafter fully onto Metal (-ngld 999) changes nothing, so this is not a placement issue.

Treat DFlash as a discrete-GPU feature. On Apple silicon, run the model without it.

Inference

llama-server -m Qwen3.6-35B-A3B-REAP-48-v2.gguf \
  -ngl 999 -c 16384 \
  -b 2048 -ub 2048 \
  -fa on -ctk bf16 -ctv bf16 -np 1 --jinja

-b and -ub must match. On quantized Qwen3.5/3.6 hybrids, splitting a batch across ubatches corrupts the GatedDeltaNet state — llama.cpp's default -b 2048 -ub 512 measures PPL 1402 where matched flags give 2.72. The model still generates coherent text when this happens, so it is easy to miss.

-ctk bf16 -ctv bf16 is not optional. The f16 KV default accumulates error on this architecture and corrupts tool-call output over long agent contexts.

Sampling

Non-thinking / agent: --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5. Thinking: --temp 1.0 --top-p 0.95. Code: --temp 0.6 --top-p 0.95.

Known issues and caveats

-b and -ub must match. On quantized Qwen3.5/3.6 hybrids, splitting a batch across ubatches corrupts the GatedDeltaNet state. llama.cpp's default -b 2048 -ub 512 measured PPL 1402.95 where matched flags gave 2.72 on identical input — while the model still generates coherent text, so it is easy to miss. f16 is unaffected. Reported upstream.

On a 16 GB Mac, watch the logits buffer, not just the weights. At this model's 248,320 vocab, -ub 2048 allocates 2048 x 248320 x 4B = 2.03 GiB of compute buffer. Weights (8.78 GiB) plus that buffer plus a draft model overflows the M4's 12 GiB Metal working set and OOMs. Use -b 512 -ub 512 there — it costs a little prefill and reclaims 1.5 GiB.

REAP-pruned checkpoints declare an MTP head they do not have. Convert with --no-mtp, or convert_hf_to_gguf.py writes 41 blocks of metadata over 40 blocks of tensors and the result loads nowhere.

Prose is more sensitive to speculation than code. Draft acceptance on prose is roughly half that of code at every setting, which is why n-max above 3 turns into a regression.

Reproducibility

crucible observe  Qwen/Qwen3.6-35B-A3B --samples 512 -o results/observation.json
crucible compress Qwen/Qwen3.6-35B-A3B --method reap --ratio 0.48 \
    --scores-file results/observation.json -o outputs/reap-48

python llama.cpp/convert_hf_to_gguf.py outputs/reap-48/... \
    --outfile f16.gguf --outtype f16 --no-mtp        # --no-mtp is required

python scripts/gen_imatrix_data.py -o calib.txt --samples 256 \
    --chat-template outputs/reap-48/...
llama-imatrix -m f16.gguf -f calib.txt -o model.imatrix.gguf \
    --n-gpu-layers 999 -c 4096 -ctk bf16 -ctv bf16

crucible plan f16.gguf --max-bytes 8.769GiB --backend metal \
    --imatrix model.imatrix.gguf \
    --sensitivity results/sensitivity-qwen36-reap48_20260825.json \
    --per-layer --floor IQ3_S -o plan.txt
llama-quantize --imatrix model.imatrix.gguf --tensor-type-file plan.txt \
    f16.gguf Qwen3.6-35B-A3B-REAP-48-v2.gguf Q4_K_M

--no-mtp is required: REAP drops the multi-token-prediction head but the config still declares it, and current convert_hf_to_gguf.py would otherwise write 41 blocks of metadata over 40 blocks of tensors, producing a GGUF nothing can load.

License

Apache-2.0, inherited from Qwen3.6-35B-A3B.