Qwen3.8-Flash-Next-Q4_K_XL-DN4

This build is what works best on my 128 GB LPDDR5X AMD Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S) under Windows 11, with llama.cpp on Vulkan.

Decode runs 28-32 tok/s on fresh text with the MTP head, sustains about 20 tok/s through a real working session, and falls to 11-13 tok/s at the tail end of a 262k context. It has proven itself a very good daily driver for my agentic use.

This model is unsloth's UD-Q4_K_XL with one change: the 48 expert down projections are IQ4_NL instead of Q5_1. That takes the file from 111.3 GB to 102.5 GB.

Why that matters here. Windows gives you two ways to serve a model this size: the fixed 96 GB carve-out, or dynamic memory allocation, which at its maximum setting lets one process put about 80 GiB on the GPU. On both mainline b10737 and unsloth's b10715-mix, llama.cpp treats the n-gram table as an input-layer lookup and keeps it in host RAM, never offloading it, so 28.8 GB of this file can never sit on the GPU whichever mode you pick. Take the carve-out and you buy GPU headroom at the price of leaving 32 GB for Windows, your apps and that table; the paging that follows makes the whole desktop worse. Dynamic memory allocation is the mode that works.

But UD-Q4_K_XL does not fit in 80 GiB unless you page or move expert layers to the CPU, and the next size down, UD-IQ4_XS, is a worse model. Nobody ships the size in between, so I made it. If your machine has room for something between 94 GB and 111 GB (table, weights, KV cache), this is for you.

Every number below was measured on that machine. The logs are in the method notes.

What changed

tensor group	size	UD-Q4_K_XL	this file
expert gate and up (ffn_gate_exps, ffn_up_exps)	44.4 GB	Q4_K, layer 2 at Q5_K	same
expert down (ffn_down_exps, 48 layers)	27.1 GB → 22.6 GB	Q5_1, 5 layers at Q8_0	IQ4_NL
n-gram table (per-layer embeddings)	28.8 GB	IQ4_NL	same
attention, linear attention, hyper-connections, shared expert, embeddings, output	5.5 GB	Q8_0	same
total		111.3 GB	102.5 GB

1,224 tensors in four shards, same split layout as the source. 48 tensors differ. Every other byte is unsloth's.

Why the down projections, and why IQ4_NL: ffn_down_exps is 640 values wide. That is not a multiple of 256, so none of the K-quants or I-quants with 256-value blocks can touch it. unsloth had to leave it at Q5_1, a 32-block type at 6 bits, which makes it the one big tensor group in the file with more bits than it needs. The 32-block family also has IQ4_NL at 4.5 bits.

Moving the downs there saves 8.9 GB which is enough headroom. The same rule is why the 28.8 GB table (rows 160 wide) cannot get any smaller.

Quality

Same engine, same text (wikitext-2 test, 24 chunks of 2048 tokens, -b 2048 -ub 1024 -fa on), same machine.

file	perplexity	KL divergence vs UD-Q4_K_XL	same top-1 token
unsloth UD-Q4_K_XL, 111.3 GB	2.9117 ± 0.0363	reference	reference
this file, 102.5 GB	2.9165 ± 0.0363	0.028 ± 0.001	94.5 %
unsloth UD-IQ4_XS, 93.7 GB	2.998 ± 0.037	0.089	90.75 %

Perplexity moves 0.17 %, which is inside the error bar. The KL divergence is the number that matters: median 0.005, 99th percentile 0.34, one token in eighteen changes its top pick. IQ4_XS changes one in eleven and its tail is three times worse.

Two checks on that number. The reference had to run with eight expert layers on the CPU, because 111 GB does not fit my GPU. Scoring this file the same way gives 0.0277 and 94.8 %, so the CPU kernels add nothing and the 0.028 is the quantization itself. And for scale: my own KV cache puts 0.012 between this file and itself at q8_0, where I run it today, and put 0.026 there through the weeks I ran it at q4_0. I never noticed either.

The imatrix made no difference

The downs here were requantized from unsloth's Q5_1 without an importance matrix. I expected that to be the weak point, so I tried it: the same 48 tensors requantized with unsloth's published imatrix_unsloth.gguf_file, everything else identical.

downs requantized	KL divergence vs UD-Q4_K_XL	same top-1
without the imatrix (this file)	0.0282 ± 0.0007	94.50 %
with the imatrix	0.0277 ± 0.0007	94.60 %

Inside one error bar on every statistic except the single worst token. From a 6-bit source the IQ4_NL grid sets the error and the importance weighting has nothing left to steer. I deleted the imatrix build and kept this one. Quantizing down from the BF16 checkpoint with the imatrix is probably still the better way to make a file like this I guess. Requantizing from Q5_1 is the cheap way, and on this evidence the cheap way costs nothing really.

Running it on a 128 GB Strix Halo, Windows, Vulkan

This is what serves on my machine every day, at 262144 context, the whole transformer on the GPU, no expert layers on the CPU:

llama-server -m Qwen3.8-Flash-Next-Q4_K_XL-DN4-00001-of-00004.gguf \
  -c 262144 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -ub 512 -np 1 \
  -lm mmap --lazy-mode on --cache-ram 0 --ctx-checkpoints 8 --jinja \
  -md mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
  --spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 128 --spec-ngram-mod-n-match 48

Memory mode: dynamic settings with the shared memory slider at its 124 GB maximum. The driver then lets one process put about 80 GiB on the GPU. With the command line above, q8_0 cache, Q4_K_M draft head, 262k context. This file lands at 79.9 GiB and serves. If it ever dies at warm-up, -ub 256 frees 1.7 GiB. I tested that fallback and it works.

The table streams from the file. -lm mmap --lazy-mode on keeps the 28.8 GB n-gram table out of committed memory and reads rows as needed. It measured the same as a resident table. mlock cannot pin a region this big on Windows, so do not bother.

The draft head is unsloth's mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf. Those are Qwen's own MTP weights, exported by unsloth. The head borrows embeddings from the main model by shape, so it pairs with this file unchanged. It is worth about 50 % more speed on fresh text. The n-gram drafter adds about 13 % on edits.

The KV cache. I serve it at q8_0 on both sides. Against an f16 cache that costs 0.012 of KL divergence at 2048 tokens; q4_0 on both sides costs 0.026, and keys at q4_0 with values at q8_0 costs 0.019. I measured all three at depth under real work, attended and with a power log beside them, and none of them changes decode speed. The depth curve below is the same for q8_0 and q4_0. What separates them is 1.5 GiB at 262k, the difference between landing at 79.9 GiB and at 78.5. If your machine will not hold the larger one, q4_0 gives that gigabyte and a half back for 0.014 of KL divergence and nothing in speed.

Speed on my machine. Prefill about 180 tokens per second at shallow depth, 120 with the MTP head on a 22k prompt. Decode 28 to 32 on fresh text with the head, 45 to 50 on edits with both drafters, 11 to 13 past 200k tokens of context. Across a real working session, every request logged, decode holds 19.6 / 18.5 / 17.1 / 14.8 tokens per second in the under-25k, 25 to 50k, 50 to 100k and 100k+ bands. On fresh text this file runs about 20 % slower than the source did in a fixed 96 GB carve-out, because IQ4_NL kernels cost more per step on this driver than Q5_1. That is the one price.

One warning about measuring any of it: this tablet cuts its power limit to about 15 W whenever nobody is using it, which halves everything. If your Strix Halo runs this as a server over Remote Desktop, keep the session attended or disable Modern Standby, and never trust a benchmark taken while you were away.

Vision. unsloth's mmproj for this model works with it. On this driver run the projector on the CPU with --no-mmproj-offload.

Engine used throughout: unsloth's llama.cpp fork build b10715 with PR 27836 and unslothai#142 for MTP. Mainline llama.cpp loads the file the same way, as long as your build knows the qwen4exp architecture.

Files

file	bytes	sha256
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00001-of-00004.gguf	10,946,624	4448186216b3af4cc558bbce2c3213f01608f8f8b2e5267a9767971dd3ec8082
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00002-of-00004.gguf	47,447,858,336	9904da25dd2dc97c1756ac88e0d56ff30012b65e890161b90b14b378a7d3d883
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00003-of-00004.gguf	44,552,691,904	2685325f5dfab3e4abad9f9519d1236b31b112b2bd7e5a3ff0afefd34e91b4cd
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00004-of-00004.gguf	10,462,690,720	e656c8d08713915c9005ee2a631e8c02efa6c5914cc5c492c97e2a1c5ff82f92

Total 102,474,187,584 bytes, 102.47 GB, 95.43 GiB. Shard 1 holds the metadata and the split header only, as in the source.

How I built it, and how to build the next one

method/ next to this card has the three scripts (plain Python, no numpy) and the exact commands. The short version:

1. gguf_tensor_map.py reads a GGUF's real composition from its header alone, from a local file or straight from a Hugging Face URL. Judge a quant by this map, never by its name.
2. gguf_extract.py pulls the 48 down tensors out of the four source shards into one loadable GGUF. 29 GB, three minutes.
3. llama-quantize --allow-requantize --tensor-type ffn_down_exps=iq4_nl downs.gguf downs-iq4nl.gguf IQ4_NL, about 25 minutes. The explicit --tensor-type is not optional. With the IQ4_NL file type alone, llama-quantize picks q5_K for the downs, hits the 256-block rule, and silently falls back to q5_1. You get the same file back and a success message. Read the log.
4. gguf_splice.py writes a new shard set. Every tensor is copied byte for byte except the 48 replacements, with offsets recomputed. Eleven minutes. The originals are never touched.

The same recipe works on any tensor whose width is not a multiple of 256, on any model.

Credits and license

Qwen made the model. It is under the Qwen Community License 1.0, included here as LICENSE, as the license requires for derivatives.

unsloth made UD-Q4_K_XL, which is every tensor in this file but 48, and exported the MTP head. Their choice to keep attention, the linear attention blocks and the hyper-connections at Q8_0 is what makes this model's 4-bit quants good. My measurements say those small tensors decide quality more than the expert bits do.

I built and measured this on my own machine. I am not affiliated with Qwen or unsloth.