TitanMamo10/qwen25-qat-q3-poc

Qwen2.5 QAT 3-bit Proof-of-Concept (consumer hardware)

Quantization-Aware Training artifacts for the custom Q3 GGML grids (Q3_0_G128 / Q3_1_G128 / Q3_1_G64). Training hardware per file is listed below — 0.5B runs fit a GTX 1660 Ti (6GB VRAM, CPU-offloaded optimizer); 1.5B runs used a Modal L4 (batch 2, LR 5e-6). See upstream discussion ikawrakow/ik_llama.cpp#2417 for the type proposal.

Files

File	What	Trained on	PPL (wikitext2-style held-out)
stock-f16.gguf	Qwen2.5-0.5B stock F16 reference (949M)	n/a (stock)	14.74
q3distill8-mixed-Q3_1_G64.gguf	0.5B, LLM-QAT self-distill, mixed grid (embed/lm_head/first+last blocks F16, rest Q3_1_G64), 460M	GTX 1660 Ti 6GB, batch 1, 3300 steps	21.88 (dg+0.0002 vs own F16 — lossless transfer)
qwen15_g64_8000-mixed-Q3_1_G64.gguf	Qwen2.5-1.5B, same recipe, 8000 steps, 1.2G	Modal L4, batch 2, LR 5e-6	14.10

Status (honest)

The format/runtime side is done: these files load and run in the build-qat75 tree (custom Q3 kernels, dp4a MMVQ), greedy generation is token-identical to F16, transfer degrades +0.0002. The training side is still cooking: 0.5B sits at 21.88 vs the 16.5 bar; the loop needs 1.5B scale + more data (100–170k blocks) to close it. PoC model sizes (0.5B, 1.5B) were chosen because they iterate in hours, not weeks (0.5B fits the local 6GB loop; 1.5B runs on a rented L4).

Verify

llama-perplexity -m q3distill8-mixed-Q3_1_G64.gguf -f <your 512-tok chunks> -c 512 -ngl 99

Needs a build with the Q3_1_G64 kernels (see discussion for the patch series once posted).