Question about loading the model with RTX 4090 and RTX 6000 pro?
Sorry I am new at this, I downloaded the Q4 quant which is 111GB, I have a rtx 4090 + rtx 6k pro that gives me 120GB vram. I thought I can load the model into the two cards so I have 9gb left over for kv cache, but I am having issues. This is my llama.cpp setting.
llama-server
--no-warmup
--model ~/models/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--host 0.0.0.0 --port 8000
--load-mode none
--fit on
--main-gpu 1
--spec-type draft-mtp
--spec-draft-n-max 3
--device CUDA0,CUDA1
--threads 16
--cont-batching
--ctx-size 200400
--batch-size 2048
--ubatch-size 2048
-fa on
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--jinja
--alias 'Qwen3.8-Flash-Next-Q4'
Between the two graphics cards 85GB/120GB was utilized, I am kind of confused on why this is happening? How do I make sure 111GB fully fits in my 120GB vram?
I'm not 100% sure but I think llama cpp uses mmap flag by default, so the ngram weights are sitting in ssd and are not being loaded, which should not affect performance, but if you still dont want it run --load-mode none
@LukeC110 , you're right I missed it in your config, I'm going to be honest, Idk why that is.
what kind of speeds are you seeing ? any elevated use in RAM when loading the model ?
@MohamedKhayat concurrency 1 gets me 1-1.4k prefil and 70 token/s decoding no mtp. I think there was a 10GB cache in ram or something, I don't remember. Just to clarify when I ran my benchmark test I offloaded 85GB to rtx 6000 entirely instead of spreading it between two cards. So basically I was running Q4 with a single RTX 6000 pro. Right now, I just tried to run the rtx 4090 + rtx 6000 pro split and during benchmark it crashed.
doesn't really sound like there is any spill over to ram to me ? I'm no expert but off the top of my head, maybe the model's footprint when loaded in vram is smaller than the download version ?
I don't think I can help with this I'm sorry, I hope you figure it out!
@LukeC110 yeah there are still a bunch of bugs to be ironed out you can follow progress here https://github.com/ggml-org/llama.cpp/pull/27742 but the model is still new and support is not completely there yet
same here with 2x CMP 170HX (-sm layer, 128GB vram total, so presumably more than enough to fit IQ4_XS), EPYC 7302P with 192GB DDR4-2666
=> only 82G vram used between both cards, 33GB used in host RAM by llama-server's process, and just one CPU core pinned at 100% during decode (~200 pp / 20 tg at empty context).
looks like some GPU offload code hasn't been written yet in llama.cpp's PR 27742
/opt/ai/llama.cpp/build-pr27742/bin/llama-server \
--chat-template-kwargs {"preserve_thinking": true} \
--host 127.0.0.1 \
--min-p 0.0 \
--no-mmap \
--presence-penalty 0.0 \
--reasoning-preserve \
--repeat-penalty 1.0 \
--temperature 1.0 \
--top-k 20 \
--top-p 0.95 \
--alias unsloth/Qwen3.8-Flash-Next:IQ4_XS \
--batch-size 2048 \
--ctx-size 262144 \
--cache-ram -1 \
--flash-attn on \
--fit on \
--kv-unified \
--model /ssd/ai/models/unsloth/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
--mmproj /ssd/ai/models/unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-BF16.gguf \
--n-gpu-layers 999 \
--parallel 1 \
--split-mode layer \
--threads 32 \
--tensor-split 50,50 \
--ubatch-size 2048