Tested new MTP module with UD-Q4_K_XL on Pro 6000 Max-Q
Conclusion: Wow! Thank you so much!
I tested the MTP model (mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf) using the patched version of llama.cpp mentioned in the README.
Here are the results for generating 30k tokens:
MTP=3: ~120 tps
MTP=5: ~140 tps
Comparing to no-MTP: ~70 tps (with the exact same config and prompt)
The speedup is absolutely impressive!
Hardware / Model
- Intel Core Ultra 7 270K plus
- GPU: Rtx Pro 6000 Max-Q , fixed at 250W
- RAM: 96GB
- Qwen3.8-Flash-Next-UD-Q4_K_XL
Llama.cpp config
.\app-b10715-mix-86bd2d3-windows-x64-cuda13-newer\llama-server.exe ^
--model Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--host 127.0.0.1 ^
--port 9002 ^
-ngl 999 ^
--n-cpu-moe 0 ^
--threads 12 ^
--threads-batch 12 ^
-fa on ^
-fit off ^
-n 65536 ^
-c 262144 ^
--cache-type-k q8_0 ^
--cache-type-v q8_0 ^
-md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-type draft-mtp ^
--spec-draft-n-max 5 ^
--spec-draft-type-k q8_0 ^
--spec-draft-type-v q8_0 ^
--temp 0.7 ^
--top-p 0.9 ^
--top-k 20 ^
--min-p 0.00 ^
--parallel 2 ^
-b 2048 ^
-ub 512 ^
--load-mode none ^
--repeat-penalty 1.03 ^
--jinja ^
--chat-template-kwargs "{"preserve-thinking":true,"reasoning_effort":"medium"}"
Using unsloth llamacpp qwen4exp branch, with host buffer patch, compiled with ROCm (latest TheRock nightly build) using these parameters and 256k context:
-ngl 999 -fa on --spec-type draft-mtp --spec-draft-n-max 4 --cache-type-k q8_0 --cache-type-v q8_0 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --spec-draft-n-min 0 --spec-draft-p-min 0.0 --backend-sampling --load-mode none --chat-template-kwargs '{"preserve-thinking":true,"reasoning_effort":"medium"}' --repeat-penalty 1.03
Getting 37.5 TPS !
I have a server:
Ubuntu 24.04
HUANANZHI X99 DUAL-F8D PLUS 2 x LGA Xeon E5 2696V4 — two processors with 22 cores each, totaling 44 cores, 88 threads
128 GB DDR4 RAM
Toughpower GT 1200W power supply
2 x NVIDIA GeForce RTX 5060 Ti Infinity 3, 16 GB, totaling 32 GB of video memory.
SSD 1.5 TB.
Very strange:
MTP disabled.
ExecStart=/home/artem1/vllm_server/llama.cpp-flash-next-mtp/build/bin/llama-server
-m /home/artem1/ollama_models/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
// --model-draft /home/artem1/ollama_models/mtp/MTP/mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf
--threads 88
--parallel 1
--threads-batch 44
--temperature 1.0
// --no-mmproj-offload
// --spec-type draft-mtp
// --spec-draft-n-max 3
// --spec-draft-p-min 1
-b 4096
-ub 2048
--ssl-key-file /home/artem1/vllm_server/llama.cpp/server.key
--ssl-cert-file /home/artem1/vllm_server/llama.cpp/server.crt
--ctx-size 184320
-ngl -1
// -ngld 0
// --n-cpu-moe 43
// --split-mode tensor
// --tensor-split 1,1
--host 0.0.0.0
--port 8000
--cache-type-k q8_0
--cache-type-v q8_0
--load-mode mmap
--flash-attn on
--min-p 0.0
--top-k 20
--top-p 0.95
--no-context-shift
--timeout 100000
--repeat-penalty 1.0
--repeat-last-n 512
467.36.165.761 I slot release: id 2 | task 68177 | stop processing: n_tokens = 152801, truncated = 0
467.39.529.829 I slot get_availabl: id 2 | task -1 | selected slot by LCP similarity, f_sim_best = 0.992 (> 0.100 thold), f_keep = 0.999
467.39.530.170 I slot launch_slot_: id 2 | task 68367 | processing task, is_child = 0
467.53.947.958 I slot print_timing: id 2 | task 68367 | prompt processing, n_tokens = 1293, progress = 1.00, t = 14.41 s / 89.70 tokens per second
468.07.972.178 I slot print_timing: id 2 | task 68367 | n_gen = 100, tg = 7.22 t/s, tg_3s = 7.29 t/s
468.10.986.309 I slot print_timing: id 2 | task 68367 | n_gen = 122, tg = 7.24 t/s, tg_3s = 7.30 t/s
468.14.050.779 I slot print_timing: id 2 | task 68367 | n_gen = 144, tg = 7.23 t/s, tg_3s = 7.18 t/s
468.17.185.109 I slot print_timing: id 2 | task 68367 | n_gen = 167, tg = 7.24 t/s, tg_3s = 7.34 t/s
468.20.209.259 I slot print_timing: id 2 | task 68367 | n_gen = 189, tg = 7.25 t/s, tg_3s = 7.27 t/s
468.23.325.516 I slot print_timing: id 2 | task 68367 | n_gen = 211, tg = 7.23 t/s, tg_3s = 7.06 t/s
468.26.347.377 I slot print_timing: id 2 | task 68367 | n_gen = 233, tg = 7.23 t/s, tg_3s = 7.28 t/s
MTP enabled.
ExecStart=/home/artem1/vllm_server/llama.cpp-flash-next-mtp/build/bin/llama-server
-m /home/artem1/ollama_models/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--model-draft /home/artem1/ollama_models/mtp/MTP/mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf
--threads 88
--parallel 1
--threads-batch 44
--temperature 1.0
// --no-mmproj-offload
--spec-type draft-mtp
--spec-draft-n-max 3
--spec-draft-p-min 1
-b 4096
-ub 2048
--ssl-key-file /home/artem1/vllm_server/llama.cpp/server.key
--ssl-cert-file /home/artem1/vllm_server/llama.cpp/server.crt
--ctx-size 184320
-ngl -1
-ngld 0
// --n-cpu-moe 43
// --split-mode tensor
// --tensor-split 1,1
--host 0.0.0.0
--port 8000
--cache-type-k q8_0
--cache-type-v q8_0
--load-mode mmap
--flash-attn on
--min-p 0.0
--top-k 20
--top-p 0.95
--no-context-shift
--timeout 100000
--repeat-penalty 1.0
--repeat-last-n 512
19.36.840.842 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 152638, progress = 0.98, t = 1141.47 s / 133.72 tokens per second
19.49.851.627 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 153558, progress = 0.99, t = 1155.61 s / 132.88 tokens per second
20.08.541.171 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 155606, progress = 1.00, t = 1173.85 s / 132.56 tokens per second
20.45.250.946 I slot print_timing: id 0 | task 0 | n_gen = 100, tg = 2.75 t/s, tg_3s = 2.78 t/s
20.48.515.202 I slot print_timing: id 0 | task 0 | n_gen = 109, tg = 2.75 t/s, tg_3s = 2.76 t/s
20.51.782.359 I slot print_timing: id 0 | task 0 | n_gen = 118, tg = 2.75 t/s, tg_3s = 2.75 t/s
20.55.035.866 I slot print_timing: id 0 | task 0 | n_gen = 127, tg = 2.75 t/s, tg_3s = 2.77 t/s
20.58.270.868 I slot print_timing: id 0 | task 0 | n_gen = 136, tg = 2.75 t/s, tg_3s = 2.78 t/s
21.01.528.518 I slot print_timing: id 0 | task 0 | n_gen = 145, tg = 2.76 t/s, tg_3s = 2.76 t/s
The speed drops by more than half.