Bug: Gibberish output on Strix Halo since Unsloth Desktop sets GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 on HIP build
The TLDR is in the title itself, but I give more background below:
First off, thanks for your hard work as always. I wanted to report a bug that I noticed on my machine that I didn't really see much anywhere. I'm running cachyos and using Unsloth Desktop setup via the official installer. I kept getting scrambled responses both in thinking traces for any generation along the lines of
The user simulate此人FPopsim kinds-disable PintMBp (continued...)
First I thought it was launch flags, but all of it worked fine when running the actual bundled llama.cpp myself. It wasn't jinja either.
To be more specific, I ruled out -fa on/off, --parallel 4 --kv-unified, --spec-default, --jinja, -c 262144, and prefix caching
I think I tracked it down to Unsloth Desktop setting GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 in the server process env.
My hardware is:
Ryzen AI Max 395 (gfx1151) 128gb unified
CachyOS kernel 7.2.0 (problem still exists in other kernel vers)
Model: UD-IQ4_XS snapshot 178b998.
Binary: ~/.unsloth/llama.cpp/llama-server, build b10639-2a36554fc, which is the HIP build (libggml-hip.so, no vulkan).
You can reproduce it directly by:
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 LD_LIBRARY_PATH=~/.unsloth/llama.cpp/build/bin \
~/.unsloth/llama.cpp/llama-server -m <path>/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
-ngl 99 -c 8192 --parallel 1 -fa off --port 8099
and then curl http://localhost:8099/completion -d '{"prompt":"The capital of France is","temperature":0,"top_k":0,"top_p":1,"n_predict":30,"cache_prompt":false}'
What you directly get is: Paris.. punct punctFIFI Postado Postado Postado additiveFISBXR temprFI tempr tempr tempr...
You get same output every single time with greedy sampling. no prompt cache.
but the moment you drop the env var and keep everything else the same, Paris. The capital of Germany is Berlin. The capital of Italy is Rome...
My guess is the HIP backend honored GGML_CUDA_* vars and this swaps in hipMallocManaged, which doesn't do anything useful on an APU where the allocation is GTT already. Not sure if this is set unconditionally or by something else.
Thanks again, take care you guys!
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is broken, and you should allocate maybe 512gb vram and have the rest allocated as GTT via amdgpu.gttsize=122880.
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is broken, and you should allocate maybe 512gb vram and have the rest allocated as GTT via amdgpu.gttsize=122880.
Hey, I think there is a misunderstanding. I don't have issues on fitting the model. GTT is already set and model fits fine. even with 8192 context (which gives plenty of headroom) the behavior still exists, so I have ruled that out.
I'm trying to point out that, at least from what I see on my end, this env var is set on a HIP build, and seems to be the cause of gibberish output on my device. I'm mainly asking if unsloth desktop sets it unconditionally or due to something else, or if it's a bug on my specific installation since I don't think it should be set on an APU target from what I understand.
ubuntu 26, halo strix 128 too, using distrobox rocm-7.14-qwen-3.8-flash-next form https://github.com/kyuz0/amd-strix-halo-toolboxes
0 tweeking just wanted to check does it works, so copied parameters from other model:
distrobox enter "$CONTAINER_NAME" -- bash -c "
cd '$FILES_DIR' &&
llama-server
-m '$MODEL_FILE'
--host 0.0.0.0 --port 8080 --alias local-model-ai
--no-mmap
-c 262144 -ctk q8_0 -ctv q8_0 -b 2048 -ub 512
-ngl 99 -np 1 -t 12 -tb 16 -lm none -fa on -lv 4
--n-cpu-moe 16
"
46.39.139.549 I slot print_timing: id 0 | task 2939 | prompt eval time = 15862.63 ms / 2053 tokens ( 7.73 ms per token, 129.42 tokens per second)
46.39.139.554 I slot print_timing: id 0 | task 2939 | eval time = 25852.21 ms / 262 tokens ( 99.05 ms per token, 10.10 tokens per second)
46.39.139.556 I slot print_timing: id 0 | task 2939 | total time = 41714.84 ms / 2315 tokens
46.39.139.558 I slot print_timing: id 0 | task 2939 | graphs reused = 0
46.39.141.184 I slot release: id 0 | task 2939 | stop processing: n_tokens = 30295, truncated = 0
Thanks to the original post of adi-0, I found a workaround for folks running llama.cpp through unsloth studio:
passing UNSLOTH_DISABLE_UNIFIED_MEMORY=1 to the unsloth studio command help to avoid gibberish of:
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XSunsloth/DeepSeek-V4-Flash-0731:UD-IQ2_M
I am on almost the same setup as the original post, except from that I am using Ubuntu 24.04 with kernel 7.0.0-28-generic and rocm 7.14.
My unsloth studio command (inside e.g a tmux session):
UNSLOTH_DISABLE_UNIFIED_MEMORY=1 LLAMA_ARG_NO_MMAP=1 LD_LIBRARY_PATH=/home/<my-user>/.unsloth/llama.cpp/build/bin unsloth studio -H 0.0.0.0 --silent
glad it helped @huaj1ng !
let's hope the devs see and patch this on their side in the next release ;-;
I wonder why it's only a subset and not more people using unsloth desktop on strix halo, I really didn't come across anyone at all with the same problem until I made this post
Thanks that works. On Halo Strix AI Max 395+
❌ launching with unsloth studio results in garbage tokens being generated
✅ launching with UNSLOTH_DISABLE_UNIFIED_MEMORY=1 unsloth studio worked immediately.
Tested it with
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XSunsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL
Getting 25 tok/s normally (and up to 60 tok/s when prediction is good)
Despite the new update on unsloth desktop the env issue is still present, so I'll mention you @danielhanchen just in case this helps visibility! (not sure if it will, but it's worth a shot)
This is a AMD + llama.cpp specific problem most likely, we're working on a fix
Compile llamacpp with this patch applied: https://github.com/kyuz0/amd-strix-halo-toolboxes/raw/refs/heads/main/toolboxes/llama-cpp-25992-rocm-host-buffer.patch