vcruz305/Qwen3.8-Flash-Next-GGUF

Qwen3.8-Flash-Next GGUF (vcruz305)

Real K-quant ladder from full HF BF16 weights with MTP heads in-main (blk.48 / nextn.*) and a shared BF16 PLE.

Layout

path	content
Q2_K/ … Q6_K/	Backbone shards 00001–00006 (MTP included in tensor set)
PLE-BF16/Qwen3.8-Flash-Next-PLE-BF16.gguf	Shared BF16 PLE (~95.4 GiB) — use as each quant's 00007
imatrix/	Importance matrix used for Q2–Q6
scripts/link-ple.sh	ln/cp PLE → …-00007-of-00007.gguf per quant
MTP/	Optional standalone MTP draft heads (if present)

Setup

bash scripts/link-ple.sh

Approximate backbone sizes (GiB) + shared PLE

quant	backbone	MTP tensors
Q2_K	~49.5	≥20
Q3_K_M	~62.5	≥20
Q4_K_M	~80.2	≥20
Q5_K_M	~91.3	≥20
Q6_K	~107.8	≥20
PLE BF16 (shared)	~95.4	—

Run (DGX Spark / unified memory)

Requires qwen4exp-capable llama.cpp (PR #27742 class).

llama-cli \
  -m Q4_K_M/Qwen3.8-Flash-Next-Q4_K_M-00001-of-00007.gguf \
  --load-mode mmap \
  -ngl 99 \
  -ot "per_layer_token_embd.weight=CPU" \
  -c 1024 -n 64 -st --temp 0 \
  -p "The capital of France is"

Do not mlock the PLE on 128G unified-memory boxes. PLE stays NVMe-backed via mmap.

MTP speculative decode

Weights include MTP (nextn_predict_layers / blk.48). Runtime --spec-type draft-mtp needs a llama.cpp build with qwen4exp graph_mtp (not all mainline builds yet).

Build notes

• Converter: llama.cpp qwen4exp with MTP export enabled
• Imatrix: AtomicChat-compatible matrix
• PLE left BF16 (not re-quantized); tok emb Q8_0; output Q6_K