Upload README.md with huggingface_hub

README.md

@@ -23,6 +23,7 @@ tags:
 | Variant | Context | Generation speed | CPU offload | Fits entirely in 16 GB VRAM |
 |---|---:|---:|---|:---:|
 | **`Q2_K-AllGPU` MTP** | 10K | **~156 tok/s** | **none — whole model + MTP head resident in VRAM** | ✅ |
+| `Q3_K_M` MTP | 10K | ~100 tok/s | 10 expert layers on CPU | ❌ |
 | `Q4_K_M` MTP | 10K | ~65 tok/s | ~20 expert layers on CPU | ❌ |
 
 `Q2_K-AllGPU` is the one to reach for on a single 16 GB consumer card: routed experts quantized to Q2_K (with the more error-sensitive `down` projection bumped one step to Q3_K to claw back quality), zero CPU offload. Expect a real, measured **~10-15% quality drop** versus the Q4 file in exchange for that speed — this is the most aggressive quant in this repo, not a free lunch.
@@ -48,9 +49,10 @@ The head was not trained for KAT-Coder. KAT-Coder is a fine-tune of `Qwen/Qwen3.
 | File | Size | BPW | Recipe | Intended use |
 |---|---:|---:|---|---|
 | `KAT-Coder-V2.5-Dev_Q4_K_M.gguf` | 20.55 GiB | 4.88 | Same as Q4_K_M iMatrix | Higher quality, MTP for a speed bonus, requires CPU offload on 16 GB cards |
+| `KAT-Coder-V2.5-Dev_Q3_K_M_imatrix_MTP.gguf` | 16.89 GiB | 3.98 | Same as Q3_K_M iMatrix | Balanced quality/size, MTP for a speed bonus, lighter CPU offload than Q4 |
 | `KAT-Coder-V2.5-Dev_Q2_K-AllGPU.gguf` | 13.09 GiB | 3.03 (trunk) | Routed-expert `gate`/`up` at Q2_K, `down` bumped to Q3_K, embeddings/output at Q5_K, attention at Q4_K | **AllGPU**: the whole model, MTP head included, fits in a 16 GB card with zero CPU offload — the fastest file in this repo, at a real quality cost from the aggressive routed-expert Q2_K |
 
-Both are calibrated with the same iMatrix as the base release (`calibration_datav5.txt`, 802 chunks). `Q4_K_M.gguf` is byte-identical to the non-MTP release except for the added MTP block; `Q2_K-AllGPU.gguf` has a changed base recipe (see above).
+All are calibrated with the same iMatrix as the base release (`calibration_datav5.txt`, 802 chunks). `Q4_K_M.gguf` and `Q3_K_M_imatrix_MTP.gguf` are byte-identical to the non-MTP release except for the added MTP block; `Q2_K-AllGPU.gguf` has a changed base recipe (see above).
 
 ## MTP quantization
 
@@ -90,6 +92,23 @@ llama-server \
   --batch-size 262144 --ubatch-size 1024 --cache-reuse 256
 ```
 
+### Q3_K_M MTP — 10 expert layers on CPU
+
+```bash
+llama-server \
+  --jinja --host 0.0.0.0 --port 8080 \
+  -m ~/models/gguf/KAT-Coder-V2.5-Dev_Q3_K_M_imatrix_MTP.gguf \
+  --spec-type draft-mtp --spec-draft-n-max 3 \
+  --n-gpu-layers 99 --n-cpu-moe 0 \
+  -ot "blk\.(3[0-9])\.ffn_.*_exps\.weight=CPU" \
+  --ctx-size 131072 --parallel 1 \
+  --flash-attn on \
+  --cache-type-k turbo3 --cache-type-v turbo3 \
+  --batch-size 8192 --ubatch-size 512 --cache-reuse 256
+```
+
+Measured **~100 tok/s at 10K context**, only the last 10 expert layers (`blk.30`-`blk.39`) offloaded to CPU — a lighter trunk than Q4_K_M needs fewer layers moved off GPU.
+
 ### Context-compaction agent — prompt-throughput profile (MTP disabled)
 
 Not every workload wants MTP. Context-compaction / summarization agents feed the model a huge prompt and care about *ingestion* speed, not generation speed — MTP's draft-then-verify overhead only adds latency there for no benefit, so this profile leaves `--spec-type` off entirely (the file still has the MTP head baked in, it's just unused). Fewer layers need CPU offload as a result (only 4, vs. 20 for the Q4 generation profile), and `--ubatch-size` is pushed up to 2048 to maximize prefill throughput instead of favoring low-latency decode.