NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Int4 weight-only quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, in compressed-tensors format for vLLM. 20.49 GB, down from 65.83 GB — a 69% reduction.

Unofficial and unaffiliated with NVIDIA. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

NVIDIA's own optimized deployment build is NVFP4, which needs a Blackwell GPU. This repository is for everything else: int4 W4A16 runs on Marlin kernels from compute capability 7.5 up, so Ampere, Ada and Hopper cards get a 4-bit build too. The base model card lists "building quantized variants" as one of the BF16 repo's intended uses; this is that.

What was changed

Weights were quantized from bfloat16 to int4, group size 64, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer and chat template are the vendor's, unmodified.

6,004 Linear modules were converted, covering 80.0% of the output's bytes:

component	precision	source	quantized
routed experts (128 per MoE layer × 23 layers)	int4 g64	58.75 GB	15.61 GB
Mamba in/out projections + attention (70 modules)	int4 g64	2.06 GB	0.55 GB
shared expert (46 modules)	int4 g64	0.92 GB	0.24 GB
MTP head (mtp.*)	bfloat16	2.67 GB	2.67 GB
backbone.embeddings + lm_head (untied)	bfloat16	1.41 GB	1.41 GB
MoE routers (.gate)	bfloat16 / fp32	0.02 GB	0.02 GB
Mamba conv1d, norms, SSM params	bfloat16	0.001 GB	0.001 GB
total		65.83 GB	20.49 GB

Group size 64, not the usual 128. The expert down_proj takes an 1856-wide input, and 128 does not divide it. At the default group size 2,944 expert tensors cannot be quantized; at 64 nothing is misaligned.

Left at bfloat16:

• mtp.* — the multi-token-prediction head (one attention + one moe block, num_nextn_predict_layers: 1). vLLM loads it through the speculative-decoding path rather than the main stack. At 2.67 GB it is the largest 16-bit component here, so there is real headroom for anyone who measures that vLLM accepts a quantized one.
• .gate — the 23 MoE routers plus their fp32 e_score_correction_bias. Routing decides which experts run at all; 0.02 GB is not worth the risk.
• backbone.embeddings and lm_head — precision-sensitive, and untied in this model.
• conv1d — Mamba causal-convolution kernels, shape (6144, 1, 4). Not Linear layers, and quantizers reject them outright.
• backbone.norm_f and the Mamba A_log / dt_bias / D state-space parameters — 1-D, never quantizable.

The architecture is a 52-block hybrid: 23 Mamba-2 blocks, 23 MoE blocks and 6 attention blocks. Only 6 blocks carry a KV cache, which is what makes long context cheap here (see below). Of ~30B total parameters, ~3B are active per token — 6 of 128 routed experts plus one shared expert.

Checkpoint layout

Experts ship as per-expert 2-D weights in the source already (…mixer.experts.{id}.up_proj, down_proj), so no fused-3-D splitting was involved and the naming carries straight through as …experts.{id}.up_proj.weight_packed. vLLM's NemotronH loader builds its expert mapping with ckpt_gate_proj_name="up_proj" and ckpt_down_proj_name="down_proj", matching this layout.

Usage

Requires vLLM >= 0.25.1, where NemotronHForCausalLM and the nemotron_v3 reasoning parser are both present. No nightly build needed.

vllm serve GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16 \
  --max-model-len 262144 \
  --mamba-backend flashinfer \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Do not pass --quantization; compressed-tensors is detected from config.json. The int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

• --tool-call-parser qwen3_coder is what the base model card specifies — Nemotron 3.5 emits the same XML tool-call framing as Qwen3-Coder. qwen3_xml is an alias for the same parser class in current vLLM.
• --reasoning-parser nemotron_v3 splits thinking into reasoning_content.
• --mamba-ssm-cache-dtype float16 halves the Mamba state cache if you are tight on memory; the base card pairs it with --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5.
• Speculative decoding: the base card uses a separate DSpark checkpoint. This build also still carries the vendor's MTP head, which vLLM can route through its nemotron_h_mtp path — --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'. Neither path is smoke-tested here.

Fitting the card

Only 6 of 52 blocks use attention, with 2 KV heads at head_dim 128 — about 6 KB/token, an order of magnitude cheaper than a conventional 30B. The 23 Mamba blocks hold a fixed-size recurrent state instead, roughly 48 MB per concurrent sequence at the default fp32 SSM cache (half that at float16), independent of sequence length.

context	KV cache	+ weights
32k	~0.2 GB	~20.7 GB
128k	~0.8 GB	~21.3 GB
256k (native max)	~1.6 GB	~22.1 GB

A 32 GB card is comfortable at the full native context with room for concurrency; 24 GB is workable at moderate context and low concurrency. max_position_embeddings is 262144 — the base card's validated 1M-token configurations use VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --max-model-len 1048576 on 8×H100 or GB200, which is not what this single-card build is for. This is arithmetic from config.json, not a measured deployment.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile nemotron-3.5-lightning-30b-a3b

which is equivalent to:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
    save_directory="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16",
    scheme="W4A16",
    group_size=64,
    ignore=["re:.*\\.gate$", "re:.*\\.conv1d$", "re:.*mtp.*",
            "lm_head", "re:.*\\.embeddings$", "re:.*\\.norm_f$"],
    device="cuda:0",
)

The source ships as 14 shards of ~5 GB and a job holds one shard at a time, so no re-sharding is needed and the build peaks at a few GB of VRAM — it took about a minute on one consumer GPU.

Two of those ignore patterns are easy to miss. re:.*\.embeddings$ is needed because this model calls its embedding table backbone.embeddings, not embed_tokens. And re:.*\.norm_f$ is needed because compressed-tensors auto-skips norms with a literal module_name.endswith("norm") test, which the final norm — backbone.norm_f, 1-D (2688,) — misses; without it the run aborts with expected 2D linear weight.

Evaluation

No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Treat the published Nemotron 3.5 Lightning numbers as describing the bfloat16 model, not this one.

Two reasons to be more careful than usual here. The routers stay 16-bit, so expert selection is unchanged, but all 2,944 routed experts are quantized without calibration and rarely-activated experts get no more attention than hot ones. And Mamba blocks carry state across the whole sequence, so projection error has a longer path to accumulate than in a pure attention stack — long-context behaviour is the thing worth checking on your own traffic.

License

OpenMDW License Agreement, version 1.1, inherited from the base model — the vendor's LICENSE is included unmodified. OpenMDW is permissive: it grants use without restriction and places no conditions on model outputs. It does require that any distribution retain a copy of the agreement and all notices of origin, so NVIDIA's LICENSE and its accompanying safety.md, bias.md, privacy.md and explainability.md are carried through here. It also terminates the grant for anyone who brings patent or copyright litigation over the model. Read it before redistributing a derivative.

"Nemotron" and "NVIDIA" are NVIDIA's marks; this repository is not endorsed by or affiliated with NVIDIA.