--- license: other license_name: openmdw-1.1 license_link: https://openmdw.ai/license/1-1/ base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 base_model_relation: quantized pipeline_tag: text-generation library_name: transformers language: - en - es - fr - de - it - ja tags: - compressed-tensors - w4a16 - int4 - vllm - nemotron-3.5 - mamba - moe --- # NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16 Int4 weight-only quantization of [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16), in compressed-tensors format for vLLM. **20.49 GB**, down from 65.83 GB — a 69% reduction. Unofficial and unaffiliated with NVIDIA. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those. NVIDIA's own optimized deployment build is [NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4), which needs a **Blackwell** GPU. This repository is for everything else: int4 W4A16 runs on Marlin kernels from compute capability 7.5 up, so Ampere, Ada and Hopper cards get a 4-bit build too. The base model card lists "building quantized variants" as one of the BF16 repo's intended uses; this is that. ## What was changed Weights were quantized from bfloat16 to **int4, group size 64, symmetric, weight-only** (activations stay 16-bit) using `llmcompressor.model_free_ptq`. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer and chat template are the vendor's, unmodified. 6,004 Linear modules were converted, covering 80.0% of the output's bytes: | component | precision | source | quantized | |---|---|---|---| | routed experts (128 per MoE layer × 23 layers) | int4 g64 | 58.75 GB | **15.61 GB** | | Mamba in/out projections + attention (70 modules) | int4 g64 | 2.06 GB | 0.55 GB | | shared expert (46 modules) | int4 g64 | 0.92 GB | 0.24 GB | | MTP head (`mtp.*`) | bfloat16 | 2.67 GB | 2.67 GB | | `backbone.embeddings` + `lm_head` (untied) | bfloat16 | 1.41 GB | 1.41 GB | | MoE routers (`.gate`) | bfloat16 / fp32 | 0.02 GB | 0.02 GB | | Mamba `conv1d`, norms, SSM params | bfloat16 | 0.001 GB | 0.001 GB | | **total** | | **65.83 GB** | **20.49 GB** | **Group size 64, not the usual 128.** The expert `down_proj` takes an 1856-wide input, and 128 does not divide it. At the default group size 2,944 expert tensors cannot be quantized; at 64 nothing is misaligned. Left at bfloat16: - **`mtp.*`** — the multi-token-prediction head (one `attention` + one `moe` block, `num_nextn_predict_layers: 1`). vLLM loads it through the speculative-decoding path rather than the main stack. At 2.67 GB it is the largest 16-bit component here, so there is real headroom for anyone who measures that vLLM accepts a quantized one. - **`.gate`** — the 23 MoE routers plus their fp32 `e_score_correction_bias`. Routing decides which experts run at all; 0.02 GB is not worth the risk. - **`backbone.embeddings` and `lm_head`** — precision-sensitive, and untied in this model. - **`conv1d`** — Mamba causal-convolution kernels, shape `(6144, 1, 4)`. Not Linear layers, and quantizers reject them outright. - **`backbone.norm_f`** and the Mamba `A_log` / `dt_bias` / `D` state-space parameters — 1-D, never quantizable. The architecture is a 52-block hybrid: **23 Mamba-2 blocks, 23 MoE blocks and 6 attention blocks**. Only 6 blocks carry a KV cache, which is what makes long context cheap here (see below). Of ~30B total parameters, ~3B are active per token — 6 of 128 routed experts plus one shared expert. ### Checkpoint layout Experts ship as per-expert 2-D weights in the source already (`…mixer.experts.{id}.up_proj`, `down_proj`), so no fused-3-D splitting was involved and the naming carries straight through as `…experts.{id}.up_proj.weight_packed`. vLLM's NemotronH loader builds its expert mapping with `ckpt_gate_proj_name="up_proj"` and `ckpt_down_proj_name="down_proj"`, matching this layout. ## Usage Requires **vLLM >= 0.25.1**, where `NemotronHForCausalLM` and the `nemotron_v3` reasoning parser are both present. No nightly build needed. ```bash vllm serve GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16 \ --max-model-len 262144 \ --mamba-backend flashinfer \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` Do not pass `--quantization`; compressed-tensors is detected from `config.json`. The int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above. - **`--tool-call-parser qwen3_coder`** is what the base model card specifies — Nemotron 3.5 emits the same XML tool-call framing as Qwen3-Coder. `qwen3_xml` is an alias for the same parser class in current vLLM. - **`--reasoning-parser nemotron_v3`** splits thinking into `reasoning_content`. - **`--mamba-ssm-cache-dtype float16`** halves the Mamba state cache if you are tight on memory; the base card pairs it with `--enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5`. - **Speculative decoding**: the base card uses a separate DSpark checkpoint. This build also still carries the vendor's MTP head, which vLLM can route through its `nemotron_h_mtp` path — `--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'`. Neither path is smoke-tested here. ### Fitting the card Only **6 of 52 blocks use attention**, with 2 KV heads at `head_dim` 128 — about **6 KB/token**, an order of magnitude cheaper than a conventional 30B. The 23 Mamba blocks hold a fixed-size recurrent state instead, roughly **48 MB per concurrent sequence** at the default fp32 SSM cache (half that at float16), independent of sequence length. | context | KV cache | + weights | |---|---|---| | 32k | ~0.2 GB | ~20.7 GB | | 128k | ~0.8 GB | ~21.3 GB | | 256k (native max) | ~1.6 GB | ~22.1 GB | A 32 GB card is comfortable at the full native context with room for concurrency; 24 GB is workable at moderate context and low concurrency. `max_position_embeddings` is 262144 — the base card's validated 1M-token configurations use `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --max-model-len 1048576` on 8×H100 or GB200, which is not what this single-card build is for. This is arithmetic from `config.json`, not a measured deployment. ## Reproducing this checkpoint Built with [llm-quantizer](https://github.com/gotoai/llm-quantizer): ```bash ./llmq.py run --profile nemotron-3.5-lightning-30b-a3b ``` which is equivalent to: ```python # llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818, # transformers==5.15.1, torch==2.13.0 from llmcompressor import model_free_ptq model_free_ptq( model_stub="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", save_directory="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16", scheme="W4A16", group_size=64, ignore=["re:.*\\.gate$", "re:.*\\.conv1d$", "re:.*mtp.*", "lm_head", "re:.*\\.embeddings$", "re:.*\\.norm_f$"], device="cuda:0", ) ``` The source ships as 14 shards of ~5 GB and a job holds one shard at a time, so no re-sharding is needed and the build peaks at a few GB of VRAM — it took about a minute on one consumer GPU. Two of those ignore patterns are easy to miss. `re:.*\.embeddings$` is needed because this model calls its embedding table `backbone.embeddings`, not `embed_tokens`. And `re:.*\.norm_f$` is needed because compressed-tensors auto-skips norms with a literal `module_name.endswith("norm")` test, which the final norm — `backbone.norm_f`, 1-D `(2688,)` — misses; without it the run aborts with `expected 2D linear weight`. ## Evaluation **No benchmarks have been run.** Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Treat the published Nemotron 3.5 Lightning numbers as describing the bfloat16 model, not this one. Two reasons to be more careful than usual here. The routers stay 16-bit, so expert *selection* is unchanged, but all 2,944 routed experts are quantized without calibration and rarely-activated experts get no more attention than hot ones. And Mamba blocks carry state across the whole sequence, so projection error has a longer path to accumulate than in a pure attention stack — long-context behaviour is the thing worth checking on your own traffic. ## License **OpenMDW License Agreement, version 1.1**, inherited from the base model — the vendor's `LICENSE` is included unmodified. OpenMDW is permissive: it grants use without restriction and places no conditions on model outputs. It does require that any distribution retain a copy of the agreement and all notices of origin, so NVIDIA's `LICENSE` and its accompanying `safety.md`, `bias.md`, `privacy.md` and `explainability.md` are carried through here. It also terminates the grant for anyone who brings patent or copyright litigation over the model. Read it before redistributing a derivative. "Nemotron" and "NVIDIA" are NVIDIA's marks; this repository is not endorsed by or affiliated with NVIDIA.