Qwen3.5-9B-NVFP4-QAD-s4000

NVFP4 QAD (W4A16-trained (weight-only)) checkpoint of Qwen/Qwen3.5-9B @ c202236235762e1c871ad0ccb60c8ee5ba337b9a, optimizer step 4000 of 4000.

Stage-1 NVFP4 control arm at lr 1e-6 (matched-budget format comparison vs the MXFP4 arm). NVFP4's RTN init is already near-lossless on this model (serving A16 KL ~0.0486, GSM8K ~85 vs BF16 83.8), and at lr 1e-6 training moves almost nothing on top. Superseded by the lr 1e-5 arm (Qwen3.5-9B-NVFP4-QAD-LR1e-5-s4000).

How this checkpoint was produced

item	value
training repo	QATFactory branch weili/mxfp4 @ 86b72a2 (PR) — scripts/train_llm_qat.py
method	QAD (quantization-aware distillation): student trains with fake-quantized forward; frozen BF16 teacher = the base model itself
objective	pure KL at temperature 1.0 (distill_weight 1.0, hard_label_weight 0.0), loss on assistant tokens only
base model / teacher	Qwen/Qwen3.5-9B @ c202236235762e1c871ad0ccb60c8ee5ba337b9a
dataset	openperfectblend_100k_Qwen3.5-9B_think — ~100K ChatML conversations (OpenPerfectBlend-derived prompts with Qwen3.5-9B think-mode responses; native <think> spans in the assistant turns), prepared in the QATFactory project. Train file from togethercomputer/Qwen3.5-9B-reasonmix @ b88c109 (940,793,581 bytes, md5 0406bb3a7a482352360716a1bc5e9e04; ~84.3k train conversations by the trainer's epoch accounting). Held-out eval = a disjoint 256-conversation split (md5 af10c8c304c146a81ddc35e439d7ac4b), 818,944 scored positions — the same corpus used for the serving-KL rows below
preprocessing	model chat template (ChatML), assistant-only loss mask, max_seq_len 8192, right-truncated
this checkpoint	optimizer step 4000 of 4000 (32,000 conversations consumed ~= 0.38 epoch, no data repetition)
batch per weight update	8 sequences = 1/GPU x 8 GPUs x grad-accum 1 (<= 8 x 8192 = 65,536 tokens/update)
learning rate	peak 1.0e-6, cosine decay to 0 over 4000 steps, linear warmup 1% (40 steps) — recipe default; later shown ~10x too small by the 500-step lr sweep (held-out eval -0.3% at 1e-6 vs -46% at 1e-5)
optimizer	AdamW (adamw_torch, beta1 0.9 / beta2 0.999), weight_decay 0.0, max_grad_norm 1.0
precision / parallelism	bf16, FSDP2 full_shard on jbom 8xB200 (single node), gradient checkpointing (non-reentrant), sdpa attention
fake-quantized modules	all linear projections (q/k/v/o_proj, gate/up/down_proj, GatedDeltaNet in/out projections); embeddings, lm_head, norms and the vision tower stay BF16
quantization config	quant_format: nvfp4, fused_runtime_scales: true (exportable native NVFP4)
seed / bookkeeping	seed 42; held-out eval every 100 steps; checkpoint every 1000 steps; step time ~2.0 s/step
in-loop held-out eval KL	0.02903 -> 0.02878 (-0.9% over 4000 steps — flat)
W&B	10wx7xlv (project qatfactory-qat, public)

Training mode: W4A16 (weight-only fake-quant). Weights are fake-quantized in the forward pass; activations stay BF16 during training. The exported artifact can still be served W4A4 — the tables below measure both.

Serving

compressed-tensors NVFP4 artifact carrying the full W4A4 schema (FP4 E2M1 weights, block-16 FP8-E4M3 scales + per-tensor global scales, static input-activation scales learned during training). Default load in vLLM >= 0.25.1 on SM100+ serves W4A4 (measured 12.3-15.2k tok/s single-GPU greedy vs ~7.7-8.3k for W4A16 — ~1.6x). For clean weight-only W4A16 serving use the sibling --weight-only export: Qwen3.5-9B-NVFP4-QAD-s4000-weight-only.

vllm serve weili-0234/Qwen3.5-9B-NVFP4-QAD-s4000 --max-model-len 24576

Evaluation (step 4000)

model	KL vs BF16	GSM8K	GPQA-D	MMLU-Pro
BF16 teacher	— (reference)	83.8	67.68	77.0
this ckpt, A16 (W4A16 serving)	0.0486	83.8	63.1	76.7
this ckpt, A4 (W4A4 serving)	0.0751	85.8	61.6	71.8

All rows come from one frozen harness: vLLM 0.25.1 on B200 (SM100), temperature 0; KL = top-20 + residual-bucket KL vs the BF16 teacher's next-token distribution over 818,944 held-out positions (nats, lower is better; pipeline validated by a BF16-self run at 0.0002); GSM8K test[:500] 0-shot CoT; GPQA-Diamond n=198 (single seed — treat deltas < ~6 points as noise); MMLU-Pro 1000-question subset.

Related checkpoints

• Other steps of this arm: s1000, s2000, s3000
• QAD-trained BF16 master weights (pre-export): Qwen3.5-9B-NVFP4-QAD-s4000-BF16
• Higher-dose arm: NVFP4-QAD-LR1e-5

---

Part of a monitored QAD experiment series with full bookkeeping (pre-registered predictions, exact SHAs/configs/seeds per run). Produced with AI assistance (Claude).