Uniform GGUF quants silently break Qwen3.8-27B's deep thinking — reproduced on llama.cpp AND vLLM (short tasks unaffected)
Uniform GGUF quants silently break Qwen3.8-27B's deep thinking — reproduced on llama.cpp AND vLLM (short tasks unaffected)
TL;DR — Qwen3.8-27B is a hybrid model (Gated DeltaNet linear attention + full attention). Unsloth Dynamic GGUFs of it (Q4_K_XL, Q6_K_XL) work fine for everyday chat, but whenever I let it think deeply on a long task, it never converges: no closing token, endless tail-looping, or the engine just dies mid-generation. I reproduced this on two completely different runtimes (llama.cpp and vLLM + official vllm-gguf-plugin), and the same model family in a mixed-precision quant (NVFP4: FP8 on attention/recurrent layers, FP4 on MLP) converges in ~17K thinking tokens on the same prompt. This smells like recurrent state error accumulation under uniform integer quantization, not a runtime bug.
The setup
- 2× RTX 5060 Ti 16GB, tensor-parallel/2-GPU splits, Linux
- Model:
unsloth/Qwen3.8-27B-GGUF(UD-Q4_K_XL 16.4 GB, UD-Q6_K_XL 23.6 GB) - Canary task (deliberately absurd, deliberately long-horizon):
"Create an HTML page containing an SVG 2D animation of a pelican riding a bicycle."
- Thinking mode ON,
reasoning_effort: xhigh, Qwen's recommended thinking-mode sampling (temp 1.0). The pelican needs to actually pedal — the model has to write a complete self-consistent HTML/SVG/SMIL file in one shot.
Why this prompt: it's a convergence test, not a quality test. A healthy Qwen3.8 spends ~15–20K thinking tokens, then emits the file. A broken one just… keeps going.
Act 1 — llama.cpp (Oct 3)
| GGUF | Budget | Result |
|---|---|---|
| UD-Q4_K_XL | 32K | finish=length, 0 characters of actual output — burned the whole budget inside thinking |
| UD-Q6_K_XL | 76K | never stops; final |
UD-Q6_K_XL + --jinja (force GGUF-embedded chat template, verified byte-identical to HF's) |
24.6K | still no termination |
Throughput was healthy (23–41 t/s depending on split mode), perplexity-ish sanity checks on short prompts were fine, and non-thinking tasks (counting, Q&A, tool-ish formats) all worked. So I filed it as: llama.cpp loses the plot on this model's long chain-of-thought — plausibly a runtime bug with the Gated DeltaNet state handling. I even declared the llama.cpp route dead for this model.
Then I saw other people's llama.cpp runs of this model going weird on HN too (the "high fever ramblings … ended with amen" anecdote on Q8_K_XL — not my test, but same disease).
Act 2 — vLLM + official GGUF plugin (Oct 5)
To separate "runtime bug" from "quantization damage", I reran the exact same file on a completely different stack: vLLM (V1 engine, CUDA kernels, f16 KV, no speculative decoding — shares essentially zero code path with llama.cpp). GGUF support was recently moved out of vLLM core into vllm-project/vllm-gguf-plugin; Qwen3.5/3.6-VL are in its tested list, so the weight mapping for the Qwen3.x hybrid family exists.
vllm ~/models/Qwen3.8-27B-UD-Q6_K_XL.gguf \
--tokenizer unsloth/Qwen3.8-27B --dtype float16 \
--tensor-parallel-size 2 --max-model-len 52000 \
--gpu-memory-utilization 0.96 --language-model-only \
--reasoning-parser qwen3
(chat kwargs: {"enable_thinking": true, "reasoning_effort": "xhigh"}, temp 1.0, max_tokens 45K, KV pool 55.6K tokens, steady ~22–23 t/s)
| Run | Outcome |
|---|---|
| A | request dies with HTTP 500 after 1325 s ≈ 30K tokens generated (engine crash mid-generation; the server-side stack was overwritten by my chain's log rotation before I grabbed it — honest gap in the evidence) |
| B | still generating at 28 min ≈ 38K tokens, approaching the 45K budget with no completed response; harness wall-timeout killed it |
Same file, same prompt, same sampling — and vLLM also can't make this model finish thinking about a pelican. Note what vLLM did NOT show: I never got a finished response body back, so I can't plot its tail-repeat score directly. The vLLM evidence is non-convergence deep in the token budget (where NVFP4 converged at 17K) + one mid-generation crash. The direct loop evidence lives in the llama.cpp runs.
Act 3 — the control that convicts the quant format
Same 27B model, same prompt, same thinking mode, same machine, different quantization structure: the official NVFP4 checkpoints (unsloth's and RadixArk's) quantize MLP/lm_head to FP4 but keep attention and the Gated DeltaNet layers at FP8 W8A8. On SGLang this converges every time:
- ~17K thinking tokens → full, working HTML pelican (legs pedaling, wheels rotating) — measured repeatedly, Oct 3 and again today (prod config: nvfp4 KV + MTP, creative decode 47–50 t/s)
- same checkpoint family also passes 100K–133K needle-at-depth, tools, greedy determinism — so it's not "the GGUF works and I'm cherry-picking"
Also worth noting from my own runs: after the GGUFs fail this way, the same GGUF files remain perfectly healthy on short tasks (chat, counting, Q&A, vision if you attach mmproj). The damage only shows up after thousands of autoregressive steps inside the thinking trace.
My hypothesis (clearly labeled: inference, not proven)
Hybrid linear-attention models (Qwen3.5/3.6/3.8, and friends like Kimi Linear / Ring-linear) carry a fixed-size recurrent state updated every token. Uniform block quantization (even a very careful 6-bit dynamic one) adds a small rounding error to every state-transition product. Over a few hundred tokens: invisible. Over 30K–80K tokens of chain-of-thought: compounding drift into a degenerate attractor — repetition loops, refusal to close, in one case engine instability.
The strongest circumstantial support is the model publishers' own behavior: unsloth's NVFP4 recipe deliberately keeps the attention/recurrent path at FP8 and only crushes the MLP to FP4. The one quant structure they kept careful about is exactly where I see the disease. My GGUF runs had "careful" precision too — but uniform-ish across 6-bit blocks — and still broke.
What would falsify or sharpen this: per-tensor overrides keeping GDN projections (in_proj_*, conv, gates) at f16/q8 in llama.cpp (--override-tensor) while MLP stays 6-bit. If the pelican suddenly pedals, case closed. I haven't run this yet — if someone with spare time on the same model tries it, please post.
Caveats (you will ask, so here they are)
- n=1 per configuration, temp=1.0 — high variance by design; runs A and B of the identical config died differently, which is itself the picture: unstable deep-think convergence.
- No bf16 full-precision baseline — 27B bf16 doesn't fit 2×16GB. My "control" is another quant (NVFP4/F8 mixed). The argument is about where precision is kept, not quant-vs-none.
- Could be prompt-specific. All I can say: the exact same prompt converges in 17K tokens on the mixed-precision checkpoint on the same box, and this failure mode appeared across ≥3 prompts in the llama.cpp session (counting-to-huge also looped; SVG-alternate prompt looped).
- The vllm-gguf-plugin is 50 stars old — but the llama.cpp half of the evidence doesn't involve it at all.
Practical takeaways
- For Qwen3.8-27B-class hybrids on consumer GPUs: prefer the official FP8/NVFP4 mixed quants over GGUF if you care about thinking mode at all. (RadixArk/unsloth NVFP4; on 5060Ti-class cards with TP2 it also runs 2–4× faster than the GGUF once you add speculative decoding.)
- If you're GGUF-bound (VRAM or ecosystem): run with thinking off or budget-capped small. Everyday chat is genuinely fine.
- Perplexity and short-context evals will not catch this. Use a long-horizon canary (the pelican test, or "count from 1 to 500 while…", or a whole-file coding task) with thinking ON when you're screening quants of recurrent-hybrid models. It caught this at the "does it terminate" level with zero scoring infrastructure.
- Unsloth's UD "Dynamic" quants are heavily boosted on important tensors and still hit this — so "we raised precision on ranked-important tensors" may not be the right importance criterion for recurrent-hybrid long-horizon behavior. Worth flagging upstream; I may open an issue on the GGUF repo if I can trim this to a one-prompt repro.
Repro recipe
- Prompt (verbatim, above),
enable_thinking: true,reasoning_effort: xhigh, temp 1.0 / top_p 0.95 / top_k 20,max_tokens ≥ 40K - llama.cpp:
llama-server/llama-cli, UD-Q4_K_XL or UD-Q6_K_XL, any ctx ≥ 48K, default template and--jinjaboth fail the same way - vLLM:
vllm-gguf-plugin(pip, or source install with--no-build-isolationagainst the CUDA torch), local.gguf+config.jsonfrom the same repo +--tokenizer unsloth/Qwen3.8-27B, f16 KV - Pass condition: a complete HTML file is emitted within ≤ ~25K thinking tokens. NVFP4/SGLang passes every time. Q6_K_XL: 0/2 on vLLM, 0/3 on llama.cpp.
Model card says this model does deep agentic work locally on ~17GB — true for chat. For thinking, the quant structure matters more than the average bits. Would love other reports (4090/3090 pairs, MI300, whatever) before I draw the final line — especially anyone who can run Q8_0 and Qwen3.6-VL (non-GDN) as contrasts, to show this is the recurrence, not just "big model + low bits".
Hardware/versions: 2× RTX 5060 Ti 16GB; llama.cpp current master (Oct 3); vLLM 0.30.x-V1 + vllm-gguf-plugin 0.0.5 (Oct 5); SGLang 0.5.21 + NVFP4 checkpoint for the control (RadixArk Qwen3.8-27B-NVFP4: FP8 W8A8 attention/GDN, NVFP4 MLP/lm_head).
Interesting, can you try Q4_K_M / Q4_K_XL by other known quantizers and did you try previous Dynamic v2 of UD-Q4_K_XL (you can still download it: click History, choose first commit, then click Browse files)?
At least according to reddit.com/r/LocalLLaMA/comments/1vrchn9/optimizing_qwen36_qwen3827b_on_16gb_vram_complete "Master Quantization Reference Table", NVFP4 has quite a bit worse KLD than UD-Q4_K_XL and even UD-Q3_K_XL has a slightly better KLD.
Good catch on that table, and it lands on the thesis rather than against it. KLD is a short-context metric — teacher forcing over a few hundred tokens. What I'm claiming is invisible to exactly that class of metric: DeltaNet state error needs ~10k+ autoregressive steps before it changes behavior. So the ranking you quote (NVFP4 KLD worse than UD-Q4_K_XL, even UD-Q3_K_XL nearby) sitting inverted against deep-think convergence is the cleanest free data point this argument has gotten since posting. The quant with the worse KLD thinks clean for 36K tokens. The better-KLD quants never emit a closing token inside 50K, on two runtimes. Aggregate divergence says nothing about where precision sits in a recurrent state transition.
Since the HF post, the r/LocalLLM thread filled in the part I had to leave as inference — not just "does it converge" but measurable quality of what comes out. All runs below use one frozen prompt (sha256-locked file, byte-identical across every cell), reasoning_effort: xhigh:
Convergence, same box, same llama.cpp master (TP2/CUDA), same default sampling (t=1.0/0.95/20):
- NVFP4 (SGLang control): natural EOS @ 35,851 thinking tokens, clean output. MTP on/off: pixel-identical SVG, 5 min vs 17 min — spec decode is neutral on healthy weights.
- plain UD-Q6_K + MTP: natural EOS @ 57,375 (journal: 901 s, 63.7 t/s, acceptance 0.62, truncated=0)
- plain UD-Q6_K, MTP off: natural EOS @ 46,239 (24.5 min, no spec decode) — converges earlier; MTP neither rescues nor dooms convergence, that's just sampler variance
- UD-Q6_K_XL: 0/5 across llama.cpp and vLLM. Never converges inside 45-90K; one engine crash mid-generation.
So plain Q6_K pays a 1.3-1.6x thinking-budget tax vs the mixed-precision control, and XL pays infinity.
The quality gap, proven in pixels. I diffed the generated SVG palettes programmatically:
- Q6_K + MTP run: the fish inside the beak is
#E08A33. The beak's own orange is#F7B84Eand the pouch is#F2A03D— RGB distances of 58 and 30. The fish is painted with the beak's color family, i.e. effectively invisible inside the beak. Geometry correct, semantics correct, attribute binding broken. - An independent user on the Reddit thread (AMD GPU, Vulkan, a different llama.cpp build entirely) got the same class of damage on the same file: beige wing and tail on a white body, charcoal legs on a white bird, and — verified frame-by-frame after zooming their gif — the fish sitting pasted on the outside of a closed beak, never occluded by the pouch in any frame. The containment relation ("fish inside pouch") was lost.
- Non-MTP plain Q6_K run and the NVFP4 control under the same palette audit: zero inherited colors of that kind. NVFP4 output is the only one with correct bindings end to end.
Same signature on NVIDIA/CUDA and AMD/Vulkan, on the same quant file: that's the cross-runtime confirmation the quant-structure theory wanted. And it explains what "silently" means — perplexity and KLD stay green while object↔color and container↔content bindings rot during the thinking phase.
Bits gradient, not a cliff: Q4_K_XL ~never converges → Q6_K_XL never → plain Q6_K converges at 1.3-1.6x budget with binding damage → FP8-recurrent mixed quants clean at 1.0x. Average bits set how hard you land on the gradient; where the precision sits decides whether you slide down it at all.
On your two asks:
Other quantizers is the highest-value test on the list. If community Q4_K_M / Q6_K files built with different imatrix corpora think clean (the quantizers in Qwen/Qwen3.8-27B discussion #65 published per-tensor layouts and top-1 sweeps, so their files are especially interesting), the indictment narrows from "uniform integer GGUF quantization" to "unsloth's importance ranking mis-scores the GDN path in this architecture" — a fixable, shippable finding, and honestly the outcome I most want.
Dynamic v2 from history: different importance data, same uniform block structure. My prior is it behaves like v3 at this size. But it's one download away and worth a run for exactly that reason.
The falsification test stays open, too: --override-tensor keeping in_proj_*/conv/gates at f16 or q8 while MLP stays 6-bit. If the pelican suddenly pedals clean, case closed on mechanism.
If anyone here runs another quantizer's file through the pelican with thinking on and max_tokens >= 90K, please post the finish reason, the thinking-token count, and — now that we know what to look for — a screenshot. A clean pass is more valuable to this investigation than another failure. Full Reddit thread with the palette diffs and the cross-vendor reproduction: reddit.com/r/LocalLLM/comments/1wyui4o
Read your post in LocalLLM and some seem to confirm it (those who said negative things didn't even post a counter example). Test the mentioned cases of other quantizers and v2 dynamic and if the issue still persists, maybe also post in r/LocalLLaMA , it has more people and also post the full llama.cpp command.
A data point on reach, plus a candidate for the test @rtzurtz suggested. The 1,157 GGUF repos of Qwen3.8-27B had 41.6M downloads in the last 30 days, so if long thinking breaks under uniform integer quants, a lot of people are hitting it without noticing.
Among plain requants, the most downloaded after unsloth (6.5M) are ggml-org (2.2M), ISTA-DASLab GSQ-RCO (1.5M), byteshape (1.3M) and lmstudio-community (1.2M). And the third most downloaded GGUF overall, cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF (3.6M), is a mixed-precision GGUF that spends more bits on the attention and Gated DeltaNet tensors, so it could test your thesis on llama.cpp itself (it needs the patches in its recipe/ folder).
30-day downloads as of Oct 11, from Model Pulse.
I can't reproduce. Using 2x5060Ti 16GB, and Qwen3.8-27B-UD-Q6_K_L.gguf.
It works fine.
https://gofile.io/d/9wyemQVl
@jjanovsky thanks, the site says: "This file could not be loaded.". can you upload a picture to some site or probably better embed it here, maybe also zoomed into the mentioned parts.
@jjanovsky appreciate the counterexample — and I did what the thread asked: pulled your full source, audited it programmatically, and did the same to our NVFP4 control. Your run converges with genuinely excellent engineering (clamped IK, phase-correct cranks, SMIL fallback). But the source also shows this isn't a clean pass, and one finding reframes the whole thesis.
- The pelican itself is drawn wrong — at source level, not eyeball level:
Leg color: Both runs two-tone the legs (far ≈ near × 0.79 / × 0.83 — same depth convention). The difference is palette coherence: NVFP4's legs sit within 3° hue of its beak at saturation 0.80 — intentional, the shading reads as the same leg in shade. The Q6_K_L legs are 17° off-beak hue at saturation 0.40 — a salmon that belongs to no part of this bird, so the two-tone shading reads as two different legs. Same trick, one deliberate, one accidental.
The pouch is geometrically a line: its two Bézier arcs (outer M389,100 C398,122 430,130 463,97, inner return) close to 0.4px separation at t=0.7 on a 520-unit viewBox. The throat pouch — the pelican's most defining feature — collapsed into a stroke.
Beak = flat triangle: root 20px → tip 5px, straight edges, no culmen curve, no nostril, buried only 3px into the head circle.
Related: head crests touch the skull by 0.5–0.8px (they do look like they're about to fall off — because geometrically they are); the tail triangle hides inside the body ellipse with ΔRGB 69 and no outline; the wingtip hovers 1.8px off the handlebar gripping nothing.
- The important one: every uniform-quant output we've now audited gives the pelican a HUMAN knee. The comment in your html source file says it out loud — // bent forward, like a cyclist — and the geometry follows: your IK (L1=86, L2=89) drives knee flexion 43°–102° across the crank cycle. 102° at bottom stroke is a textbook human cyclist's knee. A bird's visible "knee" is the tarsal joint — it physically cannot flex past ~60°; it's a locking hinge, not a bicycle hinge. The model drew a commuting accountant with a beak bolted on, and never noticed the rider is a pelican.
And this is not your file alone — it's every uniform-GGUF output we can see: ours (plain Q6_K, NVIDIA/CUDA) solved the same way, and the AMD/Vulkan community render in the Reddit thread has the same human-knee pedaling pose on charcoal legs. Three different quantizers, runtimes and vendors, same anatomy. That's a signature, not a coincidence.
Our NVFP4 control, full source attached for anyone to measure: https://gofile.io/d/cmXnmjVk — hand-keyframed legs, knee flexion 6°–32° across the cycle: nearly straight trailing legs that only fold on recovery. That is how a bird actually pedals. Same prompt, same 2×5060 Ti, same xhigh thinking — one output contains a pelican riding a bicycle, the others contain a human riding a bicycle wearing a pelican. Measurement takes one minute: read the knee polyline from the d attributes across keyframes, interior angle at the knee point, hip-knee-foot. The NVFP4 pouch is a real volume (17–36px thick, #f9c453 under a #f5a623 mandible with a nostril scribe), warm-consistent leg palette (#e89a2e), three separate tail-feather ellipses, wingtip ellipse physically resting on the bar.
To be scrupulous: the NVFP4 run also has no fish, and no crest — the claim is not "flawless", it's that the damage class differs. Uniform integer quants leave every hard constraint intact (the file runs) while quietly degrading everything that requires knowing what the subject is: species anatomy, joint limits, attribute identities, object containment. Perplexity and KLD can't see it; knee-angle telemetry can.
If your Q6_K_L run spent its tokens thinking about a pelican, somewhere in those 90K tokens it decided birds ride bikes like humans. Happy to be proven wrong by the next source file — measurement recipe above.
So it's confirmed bug or not? How to requantize without this bug? Is Q8-XL also affected as much?
What about:
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/discussions/120
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/discussions/113
@akierum Maybe: @jjanovsky 's beak example looks not how it should. That's the question, maybe there are just not enough bits in e.g. Q4_K_M's specific layers and the rounding errors simply pile up ((even a very careful 6-bit dynamic one) adds a small rounding error to every state-transition product).
According to @Jerryccccc , a custom .gguf quant could answer the question?:
- "What would falsify or sharpen this:"
- @Jerryccccc , by "if someone with spare time on the same model tries it, please post.", do you mean a custom quant with certain suspected layers at a higher bit precision? If yes, can you make such quant?
But how to make this quant?
@akierum Regarding Q8_K_XL, @Jerryccccc writes:
Then I saw other people's llama.cpp runs of this model going weird on HN too (the "high fever ramblings … ended with amen" anecdote on Q8_K_XL — not my test, but same disease).
(unsloth's and RadixArk's)
I wonder what the result of https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4 would be, it's a bit smaller than unsloth's and maybe its quantization will affect the result (and, if so, which layers do they have in common that make them work).
PS: Does Qwen3.8-Flash-Next Q4_K_M have the same issue?
has anyone checked if the mlx 4-bit quants do the same thing? i'm running mlx-community/Qwen3.8-27B-4bit on a 64GB M5 Pro this weekend
if you share the long-think prompt you used i can run it and report back