llama.cpp Strix Halo 128GB config for coding tasks
Hi, sharing my config here. Improvements are welcome.
I use it for coding with llama-vscode. The quality of the output is very good. Speed is acceptable.
Backend
Model is served by llama-cpp-for-strix-halo.
Configuration
model = models/qwen3.8/Qwen3.8-27B-UD-Q8_K_XL.gguf
chat-template-file = models/chat_template.jinja
ctx-size = 262144
flash-attn = true
no-mmap = true
n-gpu-layers = 999
batch-size = 4096
ubatch-size = 4096
models-max = 1
metrics = false
spec-type = draft-mtp
spec-draft-n-max = 4
spec-draft-ngl = all
parallel = 1
temp = 0.7
top-p = 0.80
top-k = 20
min-p = 0.0
presence-penalty = 1.5
repeat-penalty = 1.0
reasoning-preserve = true
reasoning-budget = 65536
n-predict = 131072
reasoning-budget-message="Reasoning budget exhausted — answering now."
Thanks for sharing, but what speeds are you getting in terms of PP and Decode with this setup?
I did not fully understand the question. I do not know the meaning of PP and Decode.
But I offer the following.
"timings": {
"cache_n": 0,
"prompt_n": 32,
"prompt_ms": 481.358,
"prompt_per_token_ms": 15.0424375,
"prompt_per_second": 66.47858766240512,
"predicted_n": 2567,
"predicted_ms": 182115.049,
"predicted_per_token_ms": 70.97234957131722,
"predicted_per_second": 14.089994287072892,
"draft_n": 3444,
"draft_n_accepted": 1709
}
It is the result of the below. It is not coding as such, but I do not know how to benchmark it properly.
curl http://localhost/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3.8-27B",
"messages": [{"role": "user", "content": "How to get ls results with full path? Describe alternatives with detail. Offer examples. Suggest other approaches."}],
"stream": false
}'
Ah sorry, PP = Prompt Processing. How many inbound tokens per second that is being processed.
Decode is how many tokens it can generate per second.
They are usually measured at different context sizes like 8k, 32k, 64k and so on.
Do you feel like it works well?
Thank you. So the results I posted previously at least try to provide some measurement.
As of the general feel: It is slow but it is worth waiting. I mean that it thinks a lot. Sometimes not communicating much on the progress. Strix may be humming for an hour or more... Ultimately it yields working solution. With 0 to 2 bugs that can mostly be traced back to vague requirements. All accompanied with documentation and unit tests. The resulting code is clean and understandable enough to do final manual tweaks.
I suggest to change your params ubatch-size and spec-draft-n-max to improve the speed:
# ubatch-size:batch-size = 1:4 ratio is optimal on strix halo with MTP enabled
ubatch-size = 1024
# better acceptance rate
spec-draft-n-max = 3
I noticed Q8 has a better acceptance rate so I set spec-draft-n-max to 4 and it's slightly faster.