Upload README.md with huggingface_hub

README.md

@@ -56,23 +56,65 @@ KLD is per-token, 8 samples from `neuralmagic/calibration` (LLM split) at `max_s
 PPL is `wikitext-2-raw` test, non-overlapping 2048-token chunks. Baseline for both is
 **bf16 `Qwen/Qwen3.8-27B` itself**, so what is measured is quantisation loss and nothing else.
 
-| Model | Modules quantised | KLD vs bf16 | max-token KLD | PPL | ΔPPL |
+**Two KLD columns, because one number cannot be fair to both checkpoints.** The official
+release is DeepSeek-format FP8, which has to be dequantised to plain bf16 before it will
+run at all on Blackwell — and a dequantised checkpoint runs with **bf16 activations**. Our
+checkpoint, loaded normally, still applies its scheme's **dynamic FP8 activation quant**.
+Comparing those two directly measures W8A8 against W8A16 and flatters whichever model was
+dequantised. So both are reported both ways.
+
+| Model | Modules quantised | KLD (weight-only) | KLD (as deployed) | PPL | ΔPPL |
 |---|---|---|---|---|---|
-| `Qwen/Qwen3.8-27B` (bf16) | — | 0 | — | 6.9416 | — |
-| [`Qwen/Qwen3.8-27B-FP8`](https://huggingface.co/Qwen/Qwen3.8-27B-FP8) (official) | 407 | **0.0523** | 17.51 | 6.9539 | +0.18 % |
-| **This model** | 192 | 0.0756 | 17.95 | **6.9325** | **-0.13 %** |
-
-### How to read this honestly
-
-The official FP8 has **lower average KLD** than this build, despite quantising more than
-twice as many modules. That is not a contradiction: it uses 128x128 **block** scales,
-whose grid has resolution along the *input* dimension, while `llm-compressor` emits one
-scale per *output channel*. Per-channel scaling has no resolution along the input axis at
-all, so a single outlier column inflates a whole row's scale. Finer element count does not
-imply finer where it matters.
-
-On PPL the two are not meaningfully separable — both sit within noise of the bf16 parent
-(one slightly above, one slightly below).
+| `Qwen/Qwen3.8-27B` (bf16) | — | 0 | 0 | 6.9416 | — |
+| [`Qwen/Qwen3.8-27B-FP8`](https://huggingface.co/Qwen/Qwen3.8-27B-FP8) (official) | 407 | 0.0523 | 0.1001 | 6.9539 | +0.18 % |
+| **This model** | 192 | **0.0362** | **0.0756** | **6.9325** | **-0.13 %** |
+
+*Weight-only* = weights round-tripped through the quantised format, activations bf16.
+*As deployed* = the same weights plus dynamic per-token FP8 activation quantisation on
+exactly the modules each checkpoint actually converts. For this build the two differ because `FP8_DYNAMIC` quantises activations; for the official they differ for the same reason.
+
+**This build has lower divergence from the bf16 parent than the official release on both
+measures** — **31% lower** weight-only and **24% lower** as deployed — while quantising
+192 modules to the official's 407.
+
+On PPL the two are not meaningfully separable; both sit within noise of the bf16 parent.
+A negative ΔPPL does **not** mean this build beats its own parent, only that the loss is
+below measurement noise.
+
+### What does *not* explain the difference
+
+The intuitive story is that scale layout matters — the official uses 128x128 **block**
+scales, `llm-compressor` emits one scale per **output channel**. We tested that directly,
+simulating the FP8 E4M3 round-trip on real weight tensors from this model
+([`experiment_block_scales.py`](https://github.com/huginnfork/llm-quantisation/blob/master/remote/experiment_block_scales.py)):
+
+| scheme | #scales | rel. Frobenius err | max abs err |
+|---|---|---|---|
+| per-tensor | 1 | 0.02648 | 0.02065 |
+| per-output-channel (this build) | 5120 | 0.02643 | 0.02065 |
+| block 128x128 (official) | 5440 | 0.02643 | **0.00426** |
+| *control:* INT8 per-tensor | 1 | 0.0750 | — |
+| *control:* INT8 per-output-channel | 17408 | **0.0101** | — |
+| *control:* INT8 block 128x128 | 5440 | 0.0144 | — |
+
+All FP8 variants land within 0.1 % of each other — **including per-tensor, with a single
+scale for the entire matrix**. The reason is that FP8 is a *floating-point* format: every
+element carries its own exponent, so the scale only has to bring values into range rather
+than resolve dynamic range the way a fixed-point scale must. The INT8 control shows the
+method is sensitive enough to detect layout effects where they exist (a 7.4x spread) — and
+that in *that* regime per-output-channel actually beats 128x128 block, having more scales.
+
+Block scaling does buy a ~5x lower **worst-case** weight error; that is an outlier-tail
+property, not an average one.
+
+We also checked whether the vendor used a smarter rounding than round-to-nearest, by
+comparing their stored weights against an RTN round-trip of the same parent tensors
+([`experiment_rounding.py`](https://github.com/huginnfork/llm-quantisation/blob/master/remote/experiment_rounding.py)):
+their reconstruction error matches RTN to within 0.3 %. They used plain round-to-nearest.
+
+So the gap is not scale layout and not rounding — it is **which modules get quantised**.
+This build leaves the attention path, the SSM block and the MTP head in bf16; the official
+quantises all of them.
 
 So the reasons to prefer this build are **architectural, not average-case accuracy**:
 
@@ -99,6 +141,13 @@ without it, vLLM's compressed-tensors loader treats the bf16 head as a quantised
 finds no scales, mis-loads it, and **every draft is rejected — 0 % acceptance at every
 position** while the head still appears to load correctly in the logs.
 
+Measured on vLLM with `--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'`
+over greedy decoding of mixed chat/code/reasoning prompts:
+
+| | draft tokens | accepted | acceptance |
+|---|---|---|---|
+| This model | 574 | 490 | **85.4%** |
+
 ## Smoke tests
 
 Text generation, vision, and MTP key presence all pass: