PLE-FP8 Version
First, thank you for doing the uncensoring, great job and super fast too.
I stored the n-gram embeddings as FP8 instead of BF16 with this version, reducing the size to 125GB with very minimal KL divergence.
https://huggingface.co/gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8
This is a great optimization — and I checked it rather than just nodding, so for anyone reading: it's exactly what it says. The n-gram PLE embeddings are repacked BF16→FP8 (F8_E4M3, all 128 shards, ~51.2 B params), the NVFP4 W4A16 experts are untouched, and the MTP head (all 31 tensors) and vision tower (333 tensors) are still bf16 and intact. The size drop checks out to the byte: 186.4 GB → 135.2 GB, and that 51.2 GB is exactly the 51.2 B n-gram params going 2 bytes → 1. Clean, surgical work.
The PLE table was the obvious place to cut — it's the single biggest chunk of the model and it's a lookup, not a matmul, so FP8 there costs far less than FP8 on the compute path would. Nice call.
One honest note on "fits my Spark," since people will ask: measured weights are 125.95 GiB (135.2 GB decimal). On a 128 GB GB10 that's still too tight for real use — even on a 128 GiB part you're left with ~2 GiB before the OS, KV cache and activations, and this is a 512-expert MoE. So it's a big step in the right direction (−51 GB) but not yet a comfortable single-box real-context deployment; the next lever would be the experts or a smaller context target. I haven't verified the KL-divergence claim myself (that needs inference, not metadata) — if you have before/after KL or perplexity numbers I'd love to see them.
Mind if I add a pointer to this from the model card as a community size-optimized variant? Full credit to you — it's a genuinely useful contribution.
Yeah should probably says sparks plural 😀
Sure point to my version as a community effort
And i do have KL diffs, these are measured against your model, so the only thing that differs is the n-gram quanting.
top-1 token agreement 100.0%
mean KL 0.0018
median KL 0.0006
p90 KL 0.0060
Perfect — those numbers settle it. top-1 agreement 100%, median KL 0.0006, p90 0.0060 against the unmodified model with only the n-gram dtype changed is effectively lossless, and exactly what you'd hope for from FP8 on a pure lookup table — no compute path for error to accumulate through. I haven't re-run it myself (that means loading the full 135 GB, which I can't casually do here), so I'm citing it as measured-by-you rather than verified-by-me — but your setup is the right one, same weights everywhere except the PLE, so I'm glad to take it at face value.
Card's updated: I added your version to the "n-gram table stays bf16" section as a community variant, with the size (173.6 → 125.95 GiB) and your KL stats, and full credit — since it's real evidence that the loader-path concern I raised there didn't actually bite in your runtime. Thanks again for the clean work. And yeah — Sparks, plural. 🙂