wyattearp/Nemotron-Cascade-2-30B-A3B-NVFP4

Nemotron-Cascade-2-30B-A3B — NVFP4 (Mixed Precision)

Self-quantized on an NVIDIA DGX Spark (GB10 Blackwell) using NVIDIA TensorRT Model Optimizer (main branch, 0.47.0.dev41+g43e9d15e0) PTQ, from the original BF16 checkpoint nvidia/Nemotron-Cascade-2-30B-A3B.

Quantization Recipe

Adapted from ModelOpt's huggingface/models/nvidia/Nemotron-3-Super-120B-A12B/ptq/nvfp4-mse.yaml (a sibling NemotronH-family model's recipe):

Component	Precision	Rationale
MoE routed experts (mixer.experts.*)	NVFP4 (W4A4, group_size 16)	MSE-calibrated static block scales.
MoE shared experts (mixer.shared_experts.*)	FP8 per-tensor	Shared MLP throughput path.
Mamba mixer linears (mixer.{in,out}_proj)	FP8 per-tensor	Preserves linear projections.
Mamba SSM Kernels & RMSNorms	BF16 / FP32	Preserves continuous recurrence scan state (A_log, D, conv1d, dt_bias).
Attention linears & KV Cache	Native BF16	Preserves RoPE geometry and eliminates uncalibrated FP8 attention q_scale degradation.

Calibration: 4,000 samples (1,000 each from nemotron-sft-agentic-v2, nemotron-sft-swe-v2, nemotron-math-v2, nemotron-science-v1), weight-MSE calibration, batch size 8, 500 steps.

---

🛠️ Serving Instructions with vLLM (GB10 / GB200 Blackwell)

Use qwen3_xml, NOT qwen3_coder: Empirical evaluation on multi-turn benchmarks demonstrated that qwen3_coder uses a fragile Python regex parser that fragments streaming buffers and drops reasoning tokens during multi-line shell execution (<, >, |). Switching to qwen3_xml (Python's C-based streaming Expat XML state machine) increases agentic exploit derivation hit rates from 20% to 50%.

docker run -d --gpus all --network inference-net --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  -v /path/to/this/checkpoint:/model \
  vllm/vllm-openai:latest \
  /model \
  --served-model-name nvidia/nemotron-cascade-2-30b-nvfp4 \
  --trust-remote-code \
  --mamba-ssm-cache-dtype float32 \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --gpu-memory-utilization 0.75 \
  --max-model-len 262144

---

🔬 Empirical Findings & Benchmarks

1. Generation Throughput (DGX Spark GB10, vLLM 0.20.0)

Prompt	BF16	NVFP4 (This Checkpoint)	Speedup
General Knowledge	28.66 tok/s	35.36 tok/s	1.23×
Math Reasoning	27.77 tok/s	45.26 tok/s	1.63×
Coding Task	28.52 tok/s	53.47 tok/s	1.87×
Agentic Tool-Call	27.87 tok/s	51.13 tok/s	1.83×
Average	~28.2 tok/s	~46.3 tok/s	~1.64×

2. Multi-Turn Tool Parser Impact (qwen3_coder vs. qwen3_xml)

Configuration	Hit Rate (n=10)	Behavior
Cascade-2 NVFP4 + qwen3_coder	20%	Regex slices streaming chunks; drops <think> reasoning and emits orphaned character debris (e, s, r).
Cascade-2 NVFP4 + qwen3_xml	40% – 50%	Expat XML streaming parser maintains full state; parses heredocs, bash redirects, and reasoning streams cleanly.

---

🔧 Upstream Configuration Fixes Applied to this Checkpoint

1. Native KV Cache Stability: Stripped uncalibrated kv_cache_scheme: fp8 declaration so KV cache defaults to native BF16, preventing the Using uncalibrated q_scale 1.0 with fp8 attention accuracy degradation.
2. Topology Metadata: Corrected layers_block_type, removed orphaned mtp_layers_block_type, and explicitly declared num_hidden_layers: 52.