unsloth

/

Qwen3.8-Flash-Next-GGUF

Like 1.58k

Follow

Unsloth AI 39.7k

ok but i dont have a god gpu

but i could run this on colab!

When MTP support will be merged to llama.cpp upstream?

ok but i dont have a god gpu

You dont need to

When MTP support will be merged to llama.cpp upstream?

We're working on it

Hi, thank you for all your work. Can you advise if the mtp feature offloads to metal when it can't fit in vram? (256gb metal, 56gb gpu)

llama.cpp/llama-server
-hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
-md unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf
--spec-type draft-mtp --spec-draft-n-max 2

I cmn init: llama threadpool init, n_threads = 64
I common_speculative_init_result: loading draft model 'unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2647.04 MiB on device 1: cudaMalloc failed: out of memory
E alloc_tensor_range: failed to allocate CUDA1 buffer of size 2775623424
E llama_model_load: error loading model: unable to allocate CUDA1 buffer
E llama_model_load_from_file_impl: failed to load model
E common_speculative_init_result: failed to load draft model, 'unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
E srv load_model: failed to load draft model, 'unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
srv operator(): operator(): cleaning up before exit...
E srv llama_server: exiting due to model loading error

Seeing the same with smaller -shared-Q4 as well.

Hi, thank you for all your work. Can you advise if the mtp feature offloads to metal when it can't fit in vram? (256gb metal, 56gb gpu)

@modelmode . with your lauch setting llama.ccp is reserving space in vram for context multipied by parralele slots (default 4), so you eather optimize setting a limited context or/and limit parallel slots. you must find yout balance.
or run llama-serve with flag -ngld 0 to force load MTP to system RAM

weirdly I don't have any settings icons for models loaded by Unsloth Desktop itself, but it shows the settings for models from Lm studio:

This comment has been hidden

This reply has been hidden

hi, I build the llama-server as said in https://unsloth.ai/docs/models/qwen3.8-next#mtp-llama.cpp-guide, but report the following error:

0.00.597.788 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.01.096.341 I srv    load_model: loading model 'llama/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.01.297.515 E llama_model_load: error loading model: unknown model architecture: 'qwen4exp'
0.01.297.528 E llama_model_load_from_file_impl: failed to load model
0.01.297.563 W srv    load_model: [spec] failed to measure draft model memory: failed to load model
0.01.497.361 E llama_model_load: error loading model: unknown model architecture: 'qwen4exp'
0.01.497.366 E llama_model_load_from_file_impl: failed to load model
0.01.497.379 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.01.694.125 E llama_model_load: error loading model: unknown model architecture: 'qwen4exp'
0.01.694.129 E llama_model_load_from_file_impl: failed to load model
0.01.694.133 E cmn  common_init_: failed to load model '/llama/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.01.694.134 E srv    load_model: failed to load model, '/llama/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.01.694.138 I srv    operator(): operator(): cleaning up before exit...
0.01.701.068 E srv  llama_server: exiting due to model loading error

sorry, I enter the wrong llama.cpp path

x3 3090 192GB RAM 6250R x2 XEON
Why it does not load?

Gemini and ChatGPT cant fix it: (

K:\llamacpp4>cd /d K:\llamacpp4

K:\llamacpp4>llama-server.exe -m "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" -md "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" --alias "Qwen3.8-Flash-Next-Q4-XL-MTP" --mmproj "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mmproj-F16.gguf" -sm layer -fit on -fitt 1536,1536,1536 -c 131072 -np 1 -fa on -b 2048 -ub 1024 -t 8 -ctk q8_0 -ctv q8_0 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --top-k 20 --top-p 0.95 --presence-penalty 0 --repeat-penalty 1.0 --port 8080 --chat-template-kwargs {"preserve_thinking":true} --reasoning-format deepseek --image-min-tokens 1024 --seed 9000
0.00.247.808 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.248.355 W srv llama_server: -----------------
0.00.248.356 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.248.356 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.248.357 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.248.357 W srv llama_server: -----------------
0.00.258.652 I srv load_model: loading model 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.02.124.307 E llama_model_load: error loading model: borrow_shared_tensor: this model is a draft head without its own 'token_embd.weight'; load it as a draft of its target model, not on its own
0.02.124.319 E llama_model_load_from_file_impl: failed to load model
0.02.124.397 W operator (): failed to measure the memory of the extra model, fitting without it: failed to load model
0.18.081.288 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance
1.09.330.692 I cmn init: llama threadpool init, n_threads = 8
1.10.107.657 I common_speculative_init_result: loading draft model 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
1.11.293.653 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2647.04 MiB on device 2: cudaMalloc failed: out of memory
1.11.293.663 E alloc_tensor_range: failed to allocate CUDA2 buffer of size 2775623424
1.11.374.500 E llama_model_load: error loading model: unable to allocate CUDA2 buffer
1.11.374.740 E llama_model_load_from_file_impl: failed to load model
1.11.374.742 E common_speculative_init_result: failed to load draft model, 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
1.11.374.762 E srv load_model: failed to load draft model, 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
1.11.374.773 I srv operator (): operator (): cleaning up before exit...
1.11.391.101 E srv llama_server: exiting due to model loading error

K:\llamacpp4>pause
Press any key to continue . . .

cd /d K:\llamacpp4

llama-server.exe ^
-m "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" ^
-md "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" ^
--alias "Qwen3.8-Flash-Next-Q4-XL-MTP" ^
--mmproj "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mmproj-F16.gguf" ^
-sm layer ^
-fit on ^
-fitt 1536,1536,1536 ^
-c 131072 ^
-np 1 ^
-fa on ^
-b 2048 ^
-ub 1024 ^
-t 8 ^
-ctk q8_0 ^
-ctv q8_0 ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
--spec-draft-n-min 0 ^
--top-k 20 ^
--top-p 0.95 ^
--presence-penalty 0 ^
--repeat-penalty 1.0 ^
--port 8080 ^
--chat-template-kwargs {"preserve_thinking":true} ^
--reasoning-format deepseek ^
--image-min-tokens 1024 ^
--seed 9000

pause

x3 3090 192GB RAM 6250R x2 XEON
Why it does not load?

Gemini and ChatGPT cant fix it: (

...

allocating 2647.04 MiB on device 2: cudaMalloc failed: out of memory

For whatever reason it didn't fit on your GPU(s).

multi gpu and buffer size got fixed please check this pr

export LLAMA_PLE_RESIDENT=1
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf
--chat-template-file /mnt/nvme/or.jinja
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
--model-draft /mnt/nvme/mtp-Qwen3.8-Flash-Next-BF16.gguf
-devd CUDA3
--spec-type draft-mtp
--spec-draft-n-max 3
--tensor-split 3.2,4,1.7,0
-cmoe
-cmoed
--numa distribute
-c 260000
--batch-size 7100
--ubatch-size 7100
--load-mode none
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja

https://github.com/ggml-org/llama.cpp/pull/28623

the above is example so use it one your test the ple is put the ngram in ram and also load non also do the same so test it properly with your flags and share your commend

i spend a lots of time for that i hope its helps some people

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

@techhermit same here, it was working before b11007-mix-3e83366 llama.cpp update

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

@techhermit same here, it was working before b11007-mix-3e83366 llama.cpp update

I am still on b10837

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

@techhermit same here, it was working before b11007-mix-3e83366 llama.cpp update

I am still on b10837

check if it is forcing the use of mtp-Qwen3.8-Flash-Next-Q8_0.gguf or mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf
was working for me with the shared one untill last updates (bundled llama + unsloth)

Any estimated time to be included in lemonade once llama.cpp is updated? Thanks!!

Hey guys, Qwen3.8-Flash-Next can now run 1.3-1.7× faster locally with no accuracy change via MTP! ⚡️ We released a new Unsloth Desktop release with MTP should be enabled by default and you can change the draft tokens. We also made the llama.cpp PR for it which you can read instructions for in our guide.

Qwen3.8-Flash GGUFs can reach 170 tokens/s on a RTX PRO 6000 which is insane!!

GGUFs: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
Guide: https://unsloth.ai/docs/models/qwen3.8-next#mtp-guide

Thanks so much guys!

I'm using a Z8G4 with an RTX 3090 and/or RTX 8000. 208GB of system RAM.

I'm new to unsloth studio, seems very promising but I don't know what parameters/flags you recommend for running Qwen3.8-Flash-Next on unsloth, how to offload N-grams to RAM and keep what is necessary on GPU. Can you advise please?

Thank you.

I'm getting about 30-40 TPS using an RTX 3090 and an RTX 8000 with 208 CPU RAM, loading Qwen 3.8 Flash Next GGUF UD IQ4_XS.

With no flags it loads everything into the 3090 and 8000. RAM does not increase at all, somehow (stays at 30GB used)

offloading Ngrams to CPU sends RAM up to 90GB used, little to no increase in tokens per second.

Note: This is incredibly faster than using LM Studio--I was getting 3 TPS using LM Studio, but certainly not 160TPS as advertised above.

I use Qwen3.8-Flash-Next-UD-Q4_K_XL quant and I get 40-50 tps generation and 700-900 tps prefill (1300 tps on large prefills) with 2x 5060 Ti (pcie4 8x for each), AMD Ryzen 9 5950X and 96Gb of DDR4. This are results for Strata engine. I tried Unsloth Desktop but I got only 5 tps generation

multi gpu and buffer size got fixed please check this pr

export LLAMA_PLE_RESIDENT=1
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf
--chat-template-file /mnt/nvme/or.jinja
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
--model-draft /mnt/nvme/mtp-Qwen3.8-Flash-Next-BF16.gguf
-devd CUDA3
--spec-type draft-mtp
--spec-draft-n-max 3
--tensor-split 3.2,4,1.7,0
-cmoe
-cmoed
--numa distribute
-c 260000
--batch-size 7100
--ubatch-size 7100
--load-mode none
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja

https://github.com/ggml-org/llama.cpp/pull/28623

the above is example so use it one your test the ple is put the ngram in ram and also load non also do the same so test it properly with your flags and share your commend

i ported them to Strata now its working like charm for everyone.

It seems, under new version (llama.cpp prebuilt b11541-mix-2500203) of unslothai/llama.cpp, the generation behaved a little slower.
The mainline of llama.cpp+ mtp-Qwen3.8-Flash-Next-Q8_0.gguf (New version added in 06-Oct ) make it much worse, under my PC tps < 40tps.
So I have to keep using the unslothai/llama.cpp + old mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf .

Any suggestion about correct way to use either of :
[1] the mainline of llama.cpp+ mtp-Qwen3.8-Flash-Next-Q8_0.gguf
or
[2] new version (new than llama.cpp prebuilt b11541-mix-2500203) of unslothai/llama.cpp + ?? mtp file ?

Used the same model of GGUF/MTP and parameters.
Only difference is the version of Unslothai/llama.cpp.
I am using windows11+ Intel ultra 275K plus+ pro6000 Max-Q + 128GB RAM.
Not tested directly but worked on real project via Pi-agent for coding.

• New version (llama.cpp prebuilt b11541-mix-2500203) :
  -- GPU usage(Windows resource management) < 50%
  -- 50-60 tps under 80k context used of 220k when thinking; 40-50 tps under 200k context used
• Old version (llama.cpp prebuilt b11160-mix-a6922cc), performance is much better:
  -- GPU usage(Windows resource management) > 70%
  -- 70-90tps under 80k context used of 220k when thinking(When code generation it can be > 100) ; 60-70 tps under 200k context used of 220k

[ llamacpp arameters ]
set CUDA_VISIBLE_DEVICES=1
.\app-b11160-mix-a6922cc-windows-x64-cuda13-newer\llama-server.exe ^
--model Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--mmproj mmproj-F16.gguf ^
--port 9002 ^
-ngl 999 ^
--n-cpu-moe 0 ^
--cache-ram 24576 ^
--threads 8 ^
--threads-batch 8 ^
-fa on ^
-fit off ^
-n 65536 ^
-c 458752 ^
--cache-type-k q8_0 ^
--cache-type-v q8_0 ^
-md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-type draft-mtp ^
--spec-draft-n-max 3 ^
--spec-draft-type-k q8_0 ^
--spec-draft-type-v q8_0 ^
--temp 0.7 ^
--top-p 0.9 ^
--top-k 20 ^
--min-p 0.00 ^
--parallel 2 ^
-b 2048 ^
-ub 512 ^
--load-mode none ^
--repeat-penalty 1.03 ^
--jinja ^
--chat-template-kwargs "{"preserve-thinking":true,"reasoning_effort":"medium"}"