Magnus Lundstedt PRO

Confirmed again, you are right that NF4 fails quietly, 2x wrong looks like int8 where 8x wrong looks broken. We verified your numbers, the 4.127, the exact 4.5000 without double quant, the 0.3729 bits it buys, all reproduce, and we've fixed the accounting now! Decoding NF4 values for plots comes later, its codebook belongs with a batch of similar formats we verify together.

Your grouping proposal is adopted: we now key groups off the module path and interpret per family, with anything uninterpreted staying visible as unexplained bytes, which means the next scheme like this should flag itself (we have two thresholds for highlighting unrecognized tensors for yellow and red warning at the bottom). Also taking your denominator point, the UI will state of-group and keep of-bare as a diagnostic. The double quant ordering result (codebook share tracks group granularity, not bit width) is going on the site with credit to you. Thanks for several real findings with measurements attached now. At some point we should have a proper acknowledgements page and you have earned a spot on it. 😄

Let us know if you find more challenging models! 😄

Yes, Tensorlens group qweight/qzeros/scale triples, but for these examples it grouped the two sidecars but left qweight out, reading 32 bits. So thanks for the challenging models! Turns out there were many problems to fix. 😅 Now both Qwen/Qwen2.5-7B-Instruct-AWQ and TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ work!

Now the map will calculate correct bpw value for the group, matching your numbers. I also added your idea on how much of budget is codebook rather than weights (3.8% and 4.1%) in the summary below the map.

A challenge with AWQ was the 4-bit weights stored in a declared I32 in the safetensors header (8 × 4-bit weights), in an unusual order (0,2,4,6,1,3,5,7) as opposed to the more natural assumption that position X is for column X. And with GPTQ it was the descending activation order rather than left to right, so every feature has to be looked up in g_idx to find its group and without that the weights decode subtly wrong.

Worth flagging, there might be similar models (and many others) that still decode wrong, so if you test some other model and numbers look off, that’s another bug I am happy to hear about! 😄