Qwen3.8-Flash-Next — Abliterated, NVFP4 experts + FP8_PB_WO dense

An abliterated build of Qwen3.8-Flash-Next in the modelopt MIXED_PRECISION format: NVFP4 group-16 experts with FP8_PB_WO (128x128 block, weight-only) dense layers, and the PLE n-gram table in FP8.

This combination had no abliterated build. Every published one is uniform NVFP4 or compressed-tensors, both of which lose the block-scaled FP8 dense path — which on sm120 is what the KV pool depends on. This exists to fill that gap.

It is a transplant, not a new abliteration. The refusal direction was recovered from published weight deltas of an existing abliterated checkpoint and re-applied to this quantization. See METHOD.md.

Attribution


base model	Qwen/Qwen3.8-Flash-Next
quantization these weights are	lovedheart/Qwen3.8-Flash-Next-NVFP4-FP8
the abliteration itself	windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4
PLE fp8 conversion (the shard `r` was recovered from)	gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8
technique	Arditi et al., Refusal in Language Models Is Mediated by a Single Direction

The credit for the ablation belongs upstream. What is new here is the transplant into a different quantization, and the measurement that says it worked.

What was changed

W' = W - 1.5013 * r (r^T W) applied to 24,676 residual-writing tensors:

family	count	orientation
`embed_tokens`	1	dim 1
`linear_attn.out_proj` (GDN)	36	dim 0
`self_attn.o_proj`	13	dim 0
`mlp.shared_expert.down_proj`	49	dim 0
`mlp.experts.*.down_proj`	24,576	dim 0
`ple.value_proj`	1	dim 0

alpha = 1.5013 was measured from the reference build (sd 0.0004 across 12 tensors), not assumed — it is a deliberate over-projection, not a plain rank-1 projection. Attention q/k/v, expert gate_proj/up_proj, norms and the vision tower are untouched.

Evaluation

GSM8K, 1319 problems, temperature 0.6, top_p 0.95, max_tokens 8192, identical harness and settings for both:

	GSM8K	stop rate	error rate
base (`lovedheart` NVFP4+FP8)	0.9666	0.9886	0.0000
this build	0.9666	0.9864	0.0000

No measurable capability cost. Refusal behaviour does change on prompts the base declines. Multimodal transcription is unchanged (same image-token count, same output structure, 4.4 s vs 4.5 s).

Serving

Built and tested on SGLang, one RTX PRO 6000 Blackwell (sm120, 96 GB):

--quantization modelopt --kv-cache-dtype fp8_e4m3 --ple-offload-embedding
--mem-fraction-static 0.975 --context-length 850000
--speculative-algorithm NEXTN --speculative-num-steps 3

Serving profile is identical to the unmodified quantization: KV pool 853,056 tokens, 850k context, 4.07 GB free VRAM. --ple-offload-embedding is required — the n-gram table is ~47.7 GiB and lives in host RAM.

Note reasoning_effort matters a lot on this model. Without it, it will spend an entire token budget thinking and return an empty response; "reasoning_effort": "medium" in chat_template_kwargs fixes it.

Caveats

The quality of the ablation is inherited from the upstream build.
Single-direction abliteration is contested — see There Is More to Refusal in LLMs than a Single Direction.
GSM8K is one benchmark. Unchanged arithmetic does not prove unchanged everything.
Safety alignment has been substantially removed. It will answer things the original refuses. Released for research and for local use on hardware you own.

License follows the base model (other / Qwen terms), as with every checkpoint in the chain above.