unsloth

/

Qwen3.8-Flash-Next-GGUF

like 867

Follow

Unsloth AI 34.8k

My experience of the various quants on a 3060 laptop (VRAM modded to 12GB) with 48GB of DDR4-3200 system RAM.

Config (latest build of llama.cpp master):

./llama-server \
-a Qwen3.8-Flash-Next-UD-Q2_K_XL \
-m Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf \
-mm mmproj-BF16.gguf \
-cmoe \
-cram 2048 \
-ctk q8_0 \
-ctv q8_0 \
-fit off \
-kvu \
-np 2 \
--min-p 0 \
--override-kv "qwen4exp.attention.indexer.top_k=int:4096" \
--temp 1 \
--top-k 20

Which uses 11.6GB of VRAM at full 256K context, achieving 10+ tok/s on decode and 40+ tok/s on prompt processing. Any larger quants would suffer significant speed degradation due to constant page faults. UD-Q4_K_XL for instance only achieves 5 tok/s decode and prompt processing takes FOREVER!

Tips to further reduce VRAM consumption:
--no-mmproj-offload (pins mmproj to the CPU)
-c 131072 (limits the context window)
-np 1 (limits to one concurrent task)

EDIT 1: -ub 1024 pushes pp to 60+ tok/s, but at the cost of more VRAM. I was able to stuff 256K context into 11.9GB with --no-mmproj-offload and -np 1, but it's too close to the VRAM ceiling for my liking. If you have 16GB VRAM, you can set -ub to 1024 or 2048 to get better pp speeds. MTP was also tested, but it didn't offer any meaningful improvements for my heavily CPU-offloaded config.

EDIT 2: Limited -cram to 2048MiB (default 8192) to prevent prompt cache from eating too much available RAM on long-horizon tasks.

EDIT 3: I was able to squeeze ~25% more decode speed using codacus' fork (https://github.com/thecodacus/llama.cpp), which takes advantage of unused VRAM by caching the most frequently used experts based on an MoE cache profile (which is much more intelligent than manually setting -ncmoe). I was still able to get 256K context, but at q4_0 KV quant with --no-mmproj-offload and -np 1. It now uses 11.8GB of VRAM with 42 cache slots. More slots would yield a more dramatic speedup, so this would work better on 16GB GPUs.

To get the profile:

MOE_TRACE_OUT=code.csv ./llama-moe-trace \
-m Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf \
-c 4096 \
-cmoe \
-fit off \
-n 512 \
-p "Implement the game Snake in Python." \
--min-p 0 \
--override-kv "qwen4exp.attention.indexer.top_k=int:4096" \
--temp 1 \
--top-k 20; MOE_TRACE_OUT=chat.csv ./llama-moe-trace \
-m Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf \
-c 4096 \
-cmoe \
-fit off \
-n 512 \
-p "What is model quantization?" \
--min-p 0 \
--override-kv "qwen4exp.attention.indexer.top_k=int:4096" \
--temp 1 \
--top-k 20; cat code.csv chat.csv > moe-cache-profile.csv; rm code.csv chat.csv

And the updated config:

./llama-server \
-a Qwen3.8-Flash-Next-UD-Q2_K_XL \
-m Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf \
-mm mmproj-BF16.gguf \
-cmoe \
-cram 2048 \
-ctk q4_0 \
-ctv q4_0 \
-fit off \
-kvu \
-np 1 \
--min-p 0 \
--moe-cache-profile moe-cache-profile.csv \
--moe-cache-slots 42 \
--no-mmproj-offload \
--override-kv "qwen4exp.attention.indexer.top_k=int:4096" \
--temp 1 \
--top-k 20

Other findings:

1. If available system RAM > ~half the model size, it would run almost as if it's fully loaded into RAM. It would be slower for the first minute or so after the model is freshly loaded, or when it's switching between experts (for example when it switches from reasoning to coding), but otherwise it would run at almost the same speed as fully loaded into RAM. There are enough cached weights that it's not constantly hitting page faults and slowing down the generation. Check you disk reads and GPU power consumption while running various quants to see which would work.
2. There's a big jump in quality after UD-Q4_K_XL. The reasoning is shorter, and there are noticeably less hallucinations, even comparing with UD-IQ4_XS. Definitely go with this if you have >64GB RAM.
3. Do not over-optimize, keep the configurations simple. Use -cmoe instead of manually setting -ncmoe if it's only partially loaded into RAM. Don't use ngram speculative decoding (hitches during generation due to slow prompt processing). Most optimizations are for configurations with large VRAM and fully RAM-loaded, which are not suitable for this config.