UD quants for 16GB VRAM?
Hello,
Are any UD quants ~15GBs planned? These would be really helpful for 16GB VRAM warriors. Even something like a UD-IQ4_XS would be excellent, as MTP can be run on CPU and model on GPU. I was able to get ~40tok/s on 5070Ti with IQ4_XS for Qwen3.6 with MTP on CPU*, but sadly it is not a UD quant. If not planned, curious what you think would be the best quant level for 16GB VRAM.
Interpolating from the benchmark graph, this seems like an ideal quant.
*: Had to split MTP layer into separate GGUF and feed that to llama.cpp to avoid MTP KV cache on GPU
Thanks!
Even 14.6 GB doesn't fit into my 16 GB VRAM anymore. What am I doing wrong?
- Try getting agent to split MTP layer into separate GGUF and feed that to llama.cpp and tell it to be CPU only to avoid MTP KV cache on GPU (will slow down inference but quality > speed for me)
- Using the standard GGUF with CPU MTP still allocates MTP's KV cache on GPU for me
- Use KV cache quantization for MTP itself
For now I made the best setup suitable for the 16GB card:
Model: Qwen 3.8 Q3_K_XL
Context: 58880 tokens (Maximum is 60160, but it is not stable enough)
Important options:
--flash-attn "on"
--cache-type-k "q8_0"
--cache-type-v "f16"
--spec-type "draft-mtp"
--spec-draft-n-max "3"
--fit "off"
--n-gpu-layers "all"
--kv-unified
--batch-size "512"
--ubatch-size "256"
KV cache is quantized, but just a bit. It is almost untouched and works perfect.
PS: Don't forget to add the option --reasoning-effort 'medium'.
Do I have to detach the MTP "head" from the GGUF I use? When I offload a separate MTP GGUF into CPU, decode tanks so much, it's not even funny.
It's not required. I did that to prevent OOM.
It's not required. I did that to prevent OOM.
For us 16GB peasants it is tho. Can you share me what you did exactly to make you net speed and not tank
I get ~45tok/s on a 5070ti, with 72k context on q8, which I think is pretty good. This is without doing the separation thing.
[*]
jinja = true
threads = 12
threads-batch = 12
parallel = 1
gpu-layers = auto
ctx-size = 128000
fit-target = 500
flash-attn = on
port = 8065
cache-type-k = f16
cache-type-v = f16
temperature = 0.55
top-p = 0.95
top-k = 20
presence-penalty = 0
min-p = 0.05
reasoning = on
reasoning-format = deepseek
reasoning-preserve = true
mmap = false
direct-io = true
[Qwen3.8-27B-Q3-MTP]
model = /data/models/qwen3.8-27B/Qwen3.8-27B-UD-Q3_K_XL.gguf
chat-template-file = /data/models/qwen_chat_template.jinja
mmap = true
direct-io = true
gpu-layers = 99
ctx-size = 72000
fit-target = 16
batch-size = 1024
ubatch-size = 128
cache-type-k = q8_0
cache-type-v = q8_0
override-tensor = blk\.64\..*=CPU
spec-type = draft-mtp,ngram-mod
spec-draft-p-min = 0.82
spec-draft-n-max = 1
cache-type-k-draft = q4_0
cache-type-v-draft = q4_0
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 8
spec-ngram-mod-n-max = 32
ctx-checkpoints = 128
cache-ram = 6000
temperature = 0.75
min-p = 0.0