Qwen3.5-122B-A10B-NVFP4-Full-GB10

All-NVFP4 (W4A4) quantization of Qwen/Qwen3.5-122B-A10B — the maximum-decode-speed variant for NVIDIA DGX Spark (GB10, SM121). Every Linear layer is 4-bit, including the lm_head. Vision encoder preserved (BF16) and verified working.

This is the most aggressively quantized checkpoint of this model: at GB10's memory bandwidth it decodes 33–35 tok/s single-stream, ~45% faster than the mixed-precision FP8Dense sibling (23–24 tok/s).

Quality note

The all-FP4 profile carries a subtle quality degradation relative to the FP8Dense sibling: the dense path (attention + GDN) runs 4-bit activations, and the lm_head's 4-bit weights flatten the output distribution slightly. It does not show up on simple extraction benchmarks (both variants ace detail tests), but in extended real-world use comprehension and nuance are noticeably better on FP8Dense. Pick by priority:

• Maximum tok/s → this checkpoint
• Best quality at ~2/3 the speed → FP8Dense

Layout

	This model	FP8Dense	scottgl
MoE experts (256/layer)	NVFP4	NVFP4	NVFP4
Standard attention	NVFP4	FP8 W8A8	BF16
GDN projections	NVFP4	FP8 W8A8	BF16 stored, FP4/FP8 at SGLang runtime
lm_head	NVFP4	BF16	BF16 stored, FP8 at SGLang runtime
Vision	BF16 (preserved)	BF16 (preserved)	—
Serves on	vLLM (lm_head patch, below)	stock vLLM	custom SGLang fork
Checkpoint	66 GB	74 GB	—
Decode, GB10 single-stream	33–35 tok/s	23–24 tok/s	~46 (SGLang + NEXTN spec-dec)

Excluded from quantization (BF16): router gates (mlp.gate, shared_expert_gate), embeddings, vision encoder + merger, norms. No MTP weights in this checkpoint (the config's MTP declaration has no corresponding tensors — speculative decoding is not available).

Quantization details

• Method: llm-compressor oneshot(), NVFP4 scheme (W4A4, group 16, FP8-E4M3 scales)
• Calibration: 512 samples, HuggingFaceH4/ultrachat_200k, seq_len 2048
• Quantized from the full multimodal model (AutoModelForImageTextToText) so the vision tower survives — earlier text-only exports of this model lost it

Benchmarks (DGX Spark GB10, vLLM 0.19.2 from-source SM121 build)

Sequential decode, gen=500:

Context	TTFT	Decode tok/s
warmup	0.84s	35.2
9K	3.22s	34.6
18K	3.72s	33.8
27K	3.69s	32.9

Concurrent @ 32K context, gen=500:

Concurrency	Aggregate tok/s	Per-request tok/s
8×	78.2	12.0
16×	110.6	8.0
32×	138.2	5.2
64×	166.5	3.3
128×	182.8	2.8

KV cache: 5.94× the full 262K context at --gpu-memory-utilization 0.90 (bf16 KV; page size 2,096 tokens, aligned to the GDN/Mamba state size).

Serving (vLLM)

vLLM's stock ParallelLMHead cannot load NVFP4-packed lm_head weights (it inherits VocabParallelEmbedding's loader). The one-file patch in vllm-patches/patch_nvfp4_lm_head.py swaps the lm_head to ReplicatedLinear, which routes through the standard quantized-linear loading path:

docker run -d --name vllm --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  vllm/vllm-openai:latest \
  --model demon-zombie/Qwen3.5-122B-A10B-NVFP4-Full-GB10 \
  --served-model-name Qwen3.5-122B-A10B \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

# apply the lm_head patch, then restart
docker cp patch_nvfp4_lm_head.py vllm:/tmp/
docker exec vllm python3 /tmp/patch_nvfp4_lm_head.py
docker restart vllm

Verified configuration: DGX Spark GB10, from-source vLLM 0.19.2 (SM121, TORCH_CUDA_ARCH_LIST 12.1) with the patch applied — all benchmark numbers above are from that build. Newer stock images have not been re-verified with this checkpoint's NVFP4 lm_head; if loading fails around lm_head/ParallelLMHead, apply the included patch.

Architecture

Qwen3.5-122B-A10B is a hybrid-attention MoE model: 48 layers (36 GDN/Gated-DeltaNet linear attention + 12 full attention, interval 4), 256 routed experts + 1 shared expert per layer (8 active), 3,072 hidden, 248,320 vocab, 262K context, plus a ViT vision encoder.

Credits

• Base model: Qwen/Qwen3.5-122B-A10B
• Quantized with llm-compressor
• Sibling checkpoint: FP8Dense (mixed precision, recommended for quality)