method/METHOD.md

How Qwen3.8-Flash-Next-Q4_K_XL-DN4 was built

Three pure-Python scripts (Python 3.10+, no numpy, no gguf package) and one llama.cpp tool. Everything here is byte-exact and reversible: the source shards are never modified.

Requirements: llama-quantize from any llama.cpp build that knows the qwen4exp architecture (unsloth's fork b10715 was used), about 35 GB of scratch space, and the four unsloth/Qwen3.8-Flash-Next-GGUF UD-Q4_K_XL shards.

0. Look before you touch: the tensor map

python gguf_tensor_map.py Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
python gguf_tensor_map.py hf:unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

Reads the header only (about 32 MB), locally or over HTTP range requests, and prints every tensor group with its type and size, the expert gate/up/down types per layer band, and the KV-cache cost per token for this architecture. This is how the down projections were found to be Q5_1 while everything around them was Q4_K, and how the 640-wide shape (not a multiple of 256) explained why.

1. Extract the tensors to requantize

python gguf_extract.py Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf downs.gguf "ffn_down_exps"

Writes one loadable GGUF holding only the tensors whose name matches the regex, carrying the first shard's full metadata with the split.* keys rewritten for a single file, so llama-quantize accepts it. 48 tensors, 29 GB, three minutes on NVMe.

2. Requantize, with the explicit tensor type

llama-quantize --allow-requantize --tensor-type ffn_down_exps=iq4_nl downs.gguf downs-iq4nl.gguf IQ4_NL

20 to 26 minutes on 16 Zen 5 cores. --tensor-type ffn_down_exps=iq4_nl is not optional. With only the IQ4_NL file type, llama-quantize's per-tensor heuristic chooses q5_K for down projections, finds that 640 is not divisible by 256, and falls back to q5_1, the type the tensors already had. The tool reports success and the output is byte-for-byte the input. Read the per-tensor lines in the log: every ffn_down_exps line must end in iq4_nl, and the output must be about 22.6 GB.

--allow-requantize is needed because the source is already quantized. A 6-bit Q5_1 source is close enough to lossless that the extra step costs little; do not use this recipe to requantize a 4-bit tensor into another 4-bit type, where the errors compound.

3. Splice the replacements into a new shard set

python gguf_splice.py Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf downs-iq4nl.gguf Qwen3.8-Flash-Next-Q4_K_XL-DN4

Writes Qwen3.8-Flash-Next-Q4_K_XL-DN4-0000N-of-00004.gguf next to the source shards. Every tensor is copied byte for byte unless a tensor of the same name exists in the replacement file, in which case the replacement's shape, type and data are used and the offsets are recomputed; shard 1 (metadata only) is copied under the new name; split.* keys are regenerated. 11 minutes.

Verify with the tensor map on the new first shard: the tensor count must match split.tensors.count (1,224), the down projections must read IQ4_NL on every layer, and nothing else may have changed.

4. Measure before you believe it

llama-perplexity -m <REF>.gguf  -f wiki.test.raw -c 2048 -b 2048 -ub 1024 --chunks 24 -ngl 99 -fa on --kl-divergence-base base.kld
llama-perplexity -m <CAND>.gguf -f wiki.test.raw -c 2048 -b 2048 -ub 1024 --chunks 24 -ngl 99 -fa on --kl-divergence-base base.kld --kl-divergence

The reference run writes every scored token's logits (12.2 GB for 24 × 1024 scored tokens); the candidate run reads them back and reports perplexity, mean and percentile KL divergence, and the share of tokens whose top-1 prediction is unchanged. Perplexity alone is too blunt for a change this small; the KLD is the number that says whether the file is the same model. Corpus: wikitext-2 raw test set (https://huggingface.co/datasets/ggml-org/ci/resolve/main/wikitext-2-raw-v1.zip).

If the reference does not fit on your GPU, --n-cpu-moe N --no-host -lm mmap -lzm on runs it with the first N expert layers on the CPU at roughly 30 % lower prefill speed; that is how the 111 GB reference was scored on the 128 GB build machine.

Why this generalises

Any tensor whose row width is not a multiple of 256 is limited to the 32-block types (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, IQ4_NL), and quantizers tend to leave such tensors at a fat 32-block type rather than reason about them. Wherever a large tensor group sits at Q5_1 or Q5_0 for that reason alone, IQ4_NL is a 25 % cut that costs about what a Q5 to Q4 step costs anywhere else. That is usually nothing you can measure when the tensor feeds an addition rather than a gate.