Running Qwen3.8-27B Q4_K_S on an RTX 3060 12GB with 96K Context + MTP (~10 t/s)
Hey everyone!
I'm sharing my working llama-server configuration for running Qwen3.8-27B-Q4_K_S on a single RTX 3060 12GB, with a 96K context window, quantized KV cache, selective CPU offloading, and MTP speculative decoding.
I'm using this setup with Pi Agent Harness, and multi-turn agent calls work smoothly thanks to prompt cache reuse and MTP.
The interesting part is that the model, 96K KV cache, and MTP draft buffers can fit within ~11.2 GB of usable VRAM by selectively offloading only FFN tensors to system RAM.
Hardware
- GPU: NVIDIA RTX 3060 12GB (~11.25 GB usable VRAM)
- CPU: AMD Ryzen 7 5700X (8C/16T)
- RAM: 32GB DDR4-3200
- OS: Windows 11
- Backend: llama.cpp
Launch Script
@echo off
title Llama Server - QWEN 3.8
:: Set the model path
set MODEL_PATH=D:\llm\model\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q4_K_S.gguf
llama-server.exe ^
-m "%MODEL_PATH%" ^
--jinja ^
--reasoning-preserve ^
--chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
-fa on ^
-fit off ^
-ngl 99 ^
--override-tensor "blk\.([0-9]|[1-3][0-9]|4[0-5])\.ffn_.*=CPU" ^
-ctk q4_0 ^
-ctv q4_0 ^
--gpu-layers-draft all ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
-lv 4 ^
--no-mmproj ^
-np 1 ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--load-mode none ^
--no-warmup ^
-b 256 ^
-ub 128 ^
-c 98304 ^
--host 0.0.0.0 ^
--port 8033
if %ERRORLEVEL% NEQ 0 (
echo.
echo [ERROR] Llama server exited with error code %ERRORLEVEL%
pause
)
Real Performance
Prompt: HTML animation of a circular speedometer with numbers 0-200 MPH. The needle starts at 0 and smoothly accelerates to 120, then decelerates back to 0 in a loop. The dial glows red when above 100 MPH.
Measured from my actual logs:
| Metric | Result |
|---|---|
| Model | Qwen3.8-27B Q4_K_S |
| Context | 96K tokens |
| KV cache | Q4_0 |
| Prompt processing | ~225 tok/s |
| Generation | ~9.7 tok/s |
| Peak generation | ~10.6 tok/s |
| MTP acceptance | ~88% |
| Mean accepted length | 2.76 |
| VRAM usage | ~11.04 GB |
What makes this work?
1. Q4_K_S instead of lower quants
I tested lower quants such as Q3_K_S, but once combined with the Q4_0 KV cache required for 96K context, the quality degradation became more noticeable.
For my setup, Q4_K_S is the sweet spot between model quality and VRAM usage.
2. Quantized KV cache
The important part for reaching 96K context is:
-ctk q4_0
-ctv q4_0
Without KV cache quantization, the memory requirement becomes too high for a 12GB card.
3. Selective FFN CPU offloading
Instead of offloading entire layers, I'm moving only the FFN tensors of selected layers to system RAM:
--override-tensor "blk\.([0-9]|[1-3][0-9]|4[0-5])\.ffn_.*=CPU"
This frees enough VRAM to keep the 96K KV cache and MTP draft buffers on the GPU.
In my configuration, this means offloading the FFN portions of 46 layers.
If you use a smaller Q4_K_S build, you may be able to keep more of the model on the GPU and potentially improve generation speed.
4. Batch and ubatch sizes
I also found that using relatively conservative batch sizes helped save VRAM:
-b 256
-ub 128
Keeping these values relatively low freed enough VRAM headroom for the 96K Q4_0 KV cache and MTP draft buffers, while still providing around 225 tok/s prompt processing in my setup.
For this configuration, the trade-off was worthwhile to stay within the RTX 3060's ~11.25 GB usable VRAM limit.
5. Standard K-quants vs IQ quants
One interesting result from my testing:
When significant CPU offloading was involved, IQ quants such as IQ3/IQ4 were substantially slower in my setup, apparently due to CPU-side dequantization overhead on DDR4.
Standard K-quants such as Q4_K_S performed much better for this particular RAM/CPU-offload configuration.
This is specific to my hardware and workload, so your results may differ.
6. MTP speculative decoding
I'm using:
--spec-type draft-mtp
--spec-draft-n-max 2
With n-max 2, I observed approximately:
- 88% acceptance
- 2.76 mean accepted tokens
This gives a noticeable improvement to generation throughput compared with normal autoregressive decoding.
Pi Agent Harness
This setup has also been working well with Pi Agent Harness.
I'm using:
--reasoning-preserve
so the <think> blocks remain available, while prompt-cache reuse makes repeated multi-turn agent interactions much cheaper to process.
For agent workloads, this combination of large context + prompt caching + MTP makes the 27B model surprisingly practical on a 12GB GPU.
Final takeaway
The main trick isn't simply running a Q4 model on a 12GB GPU.
It's combining:
Q4_K_S model + Q4_0 KV cache + conservative batch sizes + selective FFN CPU offloading + MTP
That combination lets me run Qwen3.8-27B with 96K context on an RTX 3060 12GB, while maintaining around 9.7 tok/s generation and ~225 tok/s prompt processing.
Hopefully this helps anyone trying to squeeze a larger model onto a 12GB card.
Huge thanks to the r/LocalLLaMA community for all the guides, experiments, and shared knowledge that helped me dial in this setup.
Legacy quant unsloth/Qwen3.8-27B-GGUF:
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/313447f257f7ebde0b968e4778feef774546ed81/Qwen3.8-27B-Q4_K_S.gguf
Please try iq4_nl instead of q4_0, it can provide much better results for quantizing kv cache.
Please try iq4_nl instead of q4_0, it can provide much better results for quantizing kv cache.
Thanks for the suggestion! I tested IQ4_NL as the model quant as well.
Just to clarify, I kept the KV cache at Q4_0 in both tests. The only change was the model quant:
Q4_K_S: 15.74 GB
IQ4_NL: 15.95 GB
I used the same prompt, 96K context, Q4_0 KV cache, -b 256 and -ub 128.
With IQ4_NL, I had to offload 48 FFN layers to RAM instead of 46 with Q4_K_S in order to maintain the same 96K context:
| Metric | Result |
|---|---|
| Model | Qwen3.8-27B IQ4_NL |
| Context | 96K tokens |
| KV cache | Q4_0 |
| Prompt processing | ~220.5 tok/s |
| Generation | ~4.94 t/s |
| Peak generation | 5.10 t/s |
| MTP acceptance | ~85.45% |
| Mean accepted length | 2.71 |
Q4_K_S:
--override-tensor "blk.([0-9]|[1-3][0-9]|4[0-5]).ffn_.*=CPU"
IQ4_NL:
--override-tensor "blk.([0-9]|[1-3][0-9]|4[0-7]).ffn_.*=CPU"
So in this particular setup, IQ4_NL didn't work out better. The slightly larger model required additional FFN offloading to DDR4, and the IQ quantization also appears to have a significant CPU-side dequantization cost on my Ryzen 7 5700X + DDR4 system.
In terms of output quality, the results were quite similar between Q4_K_S and IQ4_NL. I didn't notice any significant quality degradation or obvious errors in the IQ4_NL output with the same prompt and context.
I'm running this on Windows 11, and the RTX 3060 is also driving my display, so I only have around 11.25 GB of VRAM actually available for llama.cpp rather than the full 12 GB.
For this RTX 3060 12GB setup, Q4_K_S currently gives me a much better performance/VRAM balance, while providing essentially the same results in my testing.
Hi it is working. I have the same setup only difference is my processor is R5 5600. I'm getting 6.64 t/s to 7.24 t/s and sometimes it is 8.45 t/s.
Give turboquant a go with:
cache-type-k = q8_0
cache-type-v = turbo3
@Hjx2 thanks for great share. I have similar token numbers with 7900 12 core, 32gb ddr5 6000, and 3060 12gb.. Shouldn't I get more? I added -t 12 btw
@Hjx2 thanks for great share. I have similar token numbers with 7900 12 core, 32gb ddr5 6000, and 3060 12gb.. Shouldn't I get more? I added -t 12 btw
@tamal777 y @Must42 Ideally, you should optimize the configuration to use around 11.6 GB out of the available 12 GB of VRAM. Using -ub 128 helps keep peak VRAM usage under control, reducing the risk of VRAM spikes and OOM while allowing you to make better use of the available VRAM.
You can monitor the actual VRAM usage in Windows Task Manager → Performance → GPU → Dedicated GPU memory.
Since you have the full 12 GB available, you should have more headroom than I do. In my case, I only have around 11.25 GB available because the GPU is also driving my display. So you could potentially reduce the number of CPU-offloaded layers from my 46 layers, or use the additional VRAM to increase the context.
For example, --override-tensor "blk.([0-9]|[1-3][0-9]|4[0-5]).ffn_.*=CPU" offloads 46 layers to the CPU. You can ask ChatGPT to generate the complete list of configurations with fewer CPU-offloaded layers, then test them to find the best VRAM/speed balance.
You can also use --spec-draft-n-max 3 to improve generation speed. It uses slightly more VRAM, but can provide a noticeable increase in tokens/s.
Quant size also matters. For example, the mradermacher Q4_K_S is around 15.45 GB, while the Unsloth Q4_K_S I'm using is 15.74 GB. With only 12 GB of VRAM, those ~300 MB can make a difference in how many layers need to be offloaded to RAM.
Ideally, look for the smallest Q4_K_S build that maintains good quality. In my testing, IQ quants performed worse when CPU/RAM offloading was involved, with noticeably lower generation speed due to the additional CPU-side dequantization overhead.
I only get 2 T/s on a RTX 4070S, 64GB DDR4 3600, Ryzen 7 5800X3D, using latest koboldcpp. 11/12GB VRAM used. That seems super slow...
I only get 2 T/s on a RTX 4070S, 64GB DDR4 3600, Ryzen 7 5800X3D, using latest koboldcpp. 11/12GB VRAM used. That seems super slow...
I’ve only tested this configuration with the official llama.cpp, not KoboldCpp, so I can’t confirm its performance there. Try the same settings with llama.cpp to compare.
I got 32gb ddr4 ram 12gb vram rtx 4070, and i5-12400f using the same parameters, i get around 6tps 7tps max, if only it were around 15 tokens/sec and usable somewhat :(, if only there was a way, as im running q2_x_xl with 100k context
I got 32gb ddr4 ram 12gb vram rtx 4070, and i5-12400f using the same parameters, i get around 6tps 7tps max, if only it were around 15 tokens/sec and usable somewhat :(, if only there was a way, as im running q2_x_xl with 100k context
You should try downloading the Q4_K_S GGUF 2.0. It’s the fastest option, and unlike the 3.0 quants, it doesn’t use IQ quantization.
Also, in the latest llama.cpp update, you can replace the -ot regex approach with the new -ncffn option. For example, -ncffn 40 will offload 40 FFN layers to the CPU.
With those changes, you might be able to get significantly better performance than with Q2_XL, especially with a 100k context.
Versión anterior de quant unsloth/Qwen3.8-27B-GGUF:
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/313447f257f7ebde0b968e4778feef774546ed81/Qwen3.8-27B-Q4_K_S.gguf
I got 32gb ddr4 ram 12gb vram rtx 4070, and i5-12400f using the same parameters, i get around 6tps 7tps max, if only it were around 15 tokens/sec and usable somewhat :(, if only there was a way, as im running q2_x_xl with 100k context
You should try downloading the Q4_K_S GGUF 2.0. It’s the fastest option, and unlike the 3.0 quants, it doesn’t use IQ quantization.
Also, in the latest llama.cpp update, you can replace the -ot regex approach with the new -ncffn option. For example, -ncffn 40 will offload 40 FFN layers to the CPU.
With those changes, you might be able to get significantly better performance than with Q2_XL, especially with a 100k context.
Versión anterior de quant unsloth/Qwen3.8-27B-GGUF:
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/313447f257f7ebde0b968e4778feef774546ed81/Qwen3.8-27B-Q4_K_S.gguf
Gonna try it RIGHT NOW, thank you very much for telling me.
Same exact setup (except for CPU). At first I downloaded the wrong model (Qwen3.8-27B-UD-Q4_K_S (dynamic quant)) — got the same result guys talk about here in the comments — 5-6 t/s. Then I downloaded the right model and got 8-9 t/s, which is about what, 50% faster? Even though the dynamic quant is 700mb smaller.
Thanks!!!
