Upload README.md with huggingface_hub

README.md

@@ -22,6 +22,10 @@ itself an abliterated derivative of `Qwen/Qwen3.8-27B`.
 
 Produced and verified on **2x AMD MI210 (gfx90a / CDNA2)**.
 
+> **On this hardware int8 is SLOWER than BF16** — roughly half the decode rate —
+> and MTP speculative decoding does not work. Both are measured below. Use these
+> only if you need the smaller memory footprint.
+
 ## Scheme
 
 Weights 8-bit, **group size 128**, symmetric. Activations are **not** quantized (`pack-quantized`), so this is weight-only int8.
@@ -29,6 +33,72 @@ Weights 8-bit, **group size 128**, symmetric. Activations are **not** quantized
 Quantized with `llm-compressor` 0.12.1a20260701, `QuantizationModifier`
 (round-to-nearest). **No calibration data was used** and none is needed: the scheme is weight-only, so scales come straight from the weights.
 
+## Measured performance — read this before choosing int8
+
+Benchmarked on 2x AMD MI210 (gfx90a), vLLM `0.27.2rc0+mi210.1`, TP=2,
+**prefix caching disabled** (leaving it on inflates decode, because the
+harness derives decode time by subtracting a prefill that the second, cached
+request never performs).
+
+Decode, tokens/s, at three context depths:
+
+| config | @0 | @8k | @32k | weights |
+|---|---|---|---|---|
+| **BF16 + MTP** | **54.1** | **33.3** | **16.0** | 52 GB |
+| BF16 | 31.3 | 17.5 | 7.7 | 52 GB |
+| **W8A8** (this family) | 17.3 | 12.0 | 6.4 | 34 GB |
+| **W8A16** (this family) | 16.1 | 11.3 | 6.2 | 34 GB |
+
+**INT8 is roughly half the decode rate of BF16 on this hardware, not faster.**
+
+gfx90a has no fused dequantization path and its INT8 peak equals its BF16 peak
+(181 TOPS vs 181 TFLOP/s), so every quantized weight is converted to bf16 before
+the MFMA units can use it, and there is no compute headroom to recover that cost
+from. The same pattern held for int8 activations and for an int8 KV cache, which
+lost 38-48% of throughput at depth.
+
+What int8 does buy is **memory**: 34 GB against 52 GB, and roughly 830k KV
+tokens against 578k at 64K context (12.7x vs 8.8x concurrency). If you are
+capacity-bound rather than latency-bound, that is the trade on offer.
+
+Of the two, **W8A8 is the faster** — about 6% on decode and 36% on prefill —
+despite quantizing more. Prefer it over W8A16 unless you specifically need
+weight-only.
+
+On a CUDA GPU with real INT8 tensor-core paths the picture is likely different;
+none of the above should be read as a claim about other hardware.
+
+## MTP does not work on these quants
+
+The base model ships an MTP draft head and `mtp_num_hidden_layers: 1`, and vLLM
+registers `qwen3_5_mtp` as a speculative method. On BF16 it is a large win —
+**86-100% draft acceptance, ~2x decode**.
+
+On any compressed-tensors quantization of this model it produces **0.0%
+acceptance**: the draft runs every step and every token is rejected, so it is
+pure overhead and leaves you slower than not using it.
+
+Isolated by elimination, all on the same hardware and vLLM build:
+
+| target | acceptance |
+|---|---|
+| official BF16 | 86-100% |
+| abliterated BF16 | 83.8% |
+| W8A8, `mtp.*` kept BF16 | 0.0% |
+| W8A8, `mtp.*` also quantized | 0.0% |
+| W8A16 (weight-only) | 0.0% |
+| W8A8 target, draft pointed at a BF16 checkpoint | 0.0% |
+
+So it is not abliteration, not the quantization scheme, not how the MTP head
+itself is treated, and not the draft checkpoint path — vLLM reads `mtp.*` from
+the target regardless. Exactly 0.0% rather than a degraded rate points to
+something structurally broken rather than quality loss.
+
+Possibly relevant: vLLM copies the target's quantization into the draft config
+for the `dspark` method and has no equivalent for MTP.
+
+**Do not pass `--speculative-config` with these checkpoints.**
+
 ## What is quantized — and what deliberately is not
 
 Qwen3.8-27B is a **hybrid**: `layer_types` is 48 `linear_attention` (GDN) layers
@@ -47,16 +117,11 @@ Of 985 modules, **256 are quantized** and 729 are left at BF16:
 | `re:.*linear_attn.*` | 384 | all 48 GDN layers, incl. `conv1d` / `in_proj_*` / `out_proj` |
 | `re:.*visual.*` | 167 | vision tower |
 | `re:.*norm.*` | 271 | norms |
-| `re:^mtp\..*` | 15 | MTP draft head, kept BF16 for speculative decoding |
+| `re:^mtp\..*` | 15 | MTP draft head |
 | `lm_head`, `embed_tokens` | 2 | |
 
-MLPs dominate parameter count, so quantizing them captures most of the benefit
-while the sensitive linear-attention path stays untouched.
-
-## MTP is intact
-
-The 15 `mtp.*` tensors are preserved at BF16. Note they live in the **top-level**
-`mtp.` namespace, not under `model.`, so naive ignore patterns miss them.
+MLPs dominate parameter count, so quantizing them captures most of the footprint
+reduction while the sensitive linear-attention path stays untouched.
 
 ## Loading gotcha
 
@@ -85,13 +150,11 @@ gate rejects (it accepts only 64 or 128).
 ## Verified
 
 - Loads under vLLM `0.27.2rc0+mi210.1`, TP=2 on 2x MI210
-- KV cache 828,616 tokens, 6.32x concurrency at 128K (BF16 base: 616,189 / 4.70x)
-- Correct at temperature 0: `17*23` -> `391`; 2:15pm–6:40pm -> `265` minutes;
+- Correct at temperature 0: `17*23` -> `391`; 2:15pm-6:40pm -> `265` minutes;
   string-reverse one-liner -> `s = s[::-1]`
 
-**Not measured:** throughput against the BF16 base, and no perplexity or
-benchmark suite has been run. Treat quality claims as unverified beyond the
-spot checks above.
+**Not measured:** perplexity, and no benchmark suite has been run. Quality claims
+beyond the spot checks above are unverified.
 
 ## Note