daksh-neo/Qwen3.6-27B-GGUF

Qwen3.6-27B — GGUF Quantizations

🤖 Made Autonomously Using NEO — Your Autonomous AI Engineering Agent

Quantized GGUF variants of Qwen/Qwen3.6-27B produced with llama.cpp.

Benchmark Results

WikiText-2 perplexity (--ctx-size 512, --parallel 1) and llama-bench pp512 / tg128 throughput on a Tesla V100-SXM2-16GB. -ngl is the number of layers offloaded to GPU; smaller quants fit fully (-ngl 99), larger quants fall back to partial offload because the 16 GB VRAM cap doesn't hold the full model + KV cache.

Model	file_size_mb	perplexity	pp512 t/s	tg128 t/s	-ngl (perp / bench)
Qwen3.6-27B-Q2_K.gguf	10215.44	6.8364 ± —	—	37.03	99 / 99
Qwen3.6-27B-Q4_K_M.gguf	15780.83	5.9013 ± 0.160	360.89	4.88	50 / 50
Qwen3.6-27B-Q5_K_S.gguf	17814.27	5.7555 ± 0.154	402.79	4.98	42 / 53
Qwen3.6-27B-Q8_0.gguf	27271.04	5.7384 ± 0.153	133.71	1.98	28 / 35

Perplexity by quant (lower = better)

File size on disk

Prompt-processing throughput (pp512, tok/s)

Token-generation throughput (tg128, tok/s)

Note: Q2_K's tg128 is high because it's the only quant that fits fully on the V100's 16 GB VRAM (-ngl 99). Q4_K_M / Q5_K_S / Q8_0 fall back to partial offload, so their tok/s reflects the CPU spillover, not a fundamental quality-vs-speed regression. On a 24 GB+ GPU all four would land near Q2_K's headline number.

Pipeline that produced these GGUFs

Quantization Details

Quant Type	Description	Use Case
Q2_K	2-bit K-quants, aggressive compression	Edge / mobile, very limited RAM
Q4_K_M	4-bit K-quants medium, balanced quality / size	General purpose, recommended
Q5_K_S	5-bit K-quants small, better quality	Quality-critical applications
Q8_0	8-bit, near-lossless	Maximum fidelity, reference

Files

• Qwen3.6-27B-Q2_K.gguf
• Qwen3.6-27B-Q4_K_M.gguf
• Qwen3.6-27B-Q5_K_S.gguf
• Qwen3.6-27B-Q8_0.gguf

Usage

Start an OpenAI-compatible server with one of the quants:

llama-server -m Qwen3.6-27B-Q2_K.gguf -c 4096 --host 0.0.0.0 --port 8080
llama-server -m Qwen3.6-27B-Q4_K_M.gguf -c 4096 --host 0.0.0.0 --port 8080
llama-server -m Qwen3.6-27B-Q5_K_S.gguf -c 4096 --host 0.0.0.0 --port 8080
llama-server -m Qwen3.6-27B-Q8_0.gguf -c 4096 --host 0.0.0.0 --port 8080

Or one-shot inference:

llama-cli -m Qwen3.6-27B-Q4_K_M.gguf -p "Hello, world!"

---

Generated by the qwen36gguf pipeline.