README.md

---
license: other
license_name: openmdw-1.1
license_link: https://openmdw.ai/license/1-1/
base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
language:
  - en
  - es
  - fr
  - de
  - it
  - ja
tags:
  - compressed-tensors
  - w8a16
  - int8
  - vllm
  - nemotron-3.5
  - mamba
  - moe
---

# NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W8A16

Int8 weight-only quantization of
[nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16),
in compressed-tensors format for vLLM. **35.93 GB**, down from 65.83 GB — a 45% reduction.
Its KV cache is unusually cheap, so a 48 GB card serves it at the full native 256k context.

This is the fidelity-first build. Int8 round-to-nearest stays far closer to the bfloat16
weights than int4 does, at ~1.8x the footprint of the
[int4 W4A16 sibling](https://huggingface.co/GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16)
(20.49 GB). If you are targeting a 24 or 32 GB card, use that one; use this one when you
have the VRAM and want the least quality loss quantization can give without calibration.
The gap matters more than usual here — see [Evaluation](#evaluation).

Unofficial and unaffiliated with NVIDIA. All model capabilities, evaluations and
limitations belong to the original model card — see the base model for those.

NVIDIA's own optimized deployment build is
[NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4), which
needs a **Blackwell** GPU. This repository is for everything else: int8 W8A16 runs on
Marlin kernels from compute capability 7.5 up, so Ampere, Ada and Hopper cards get a
weight-quantized build too. The base model card lists "building quantized variants" as one
of the BF16 repo's intended uses; this is that.

## What was changed

Weights were quantized from bfloat16 to **int8, group size 64, symmetric, weight-only**
(activations stay 16-bit) using `llmcompressor.model_free_ptq`. No calibration data was
used and the model was never loaded — the quantizer operates directly on the safetensors.
Architecture, tokenizer and chat template are the vendor's, unmodified.

6,004 Linear modules were converted, covering 88.6% of the output's bytes:

| component | precision | source | quantized |
|---|---|---|---|
| routed experts (128 per MoE layer × 23 layers) | int8 g64 | 58.75 GB | **30.29 GB** |
| Mamba in/out projections + attention (70 modules) | int8 g64 | 2.06 GB | 1.06 GB |
| shared expert (46 modules) | int8 g64 | 0.92 GB | 0.47 GB |
| MTP head (`mtp.*`) | bfloat16 | 2.67 GB | 2.67 GB |
| `backbone.embeddings` + `lm_head` (untied) | bfloat16 | 1.41 GB | 1.41 GB |
| MoE routers (`.gate`) | bfloat16 / fp32 | 0.02 GB | 0.02 GB |
| Mamba `conv1d`, norms, SSM params | bfloat16 | 0.001 GB | 0.001 GB |
| **total** | | **65.83 GB** | **35.93 GB** |

Of the quantized share, 30.86 GB is packed int8 and 0.96 GB is bf16 group scales. The
scales are the same count as in the W4A16 sibling — both builds use group size 64 and
convert the same 6,004 modules — so the int8 payload is almost exactly twice the int4 one
and everything else is unchanged.

**Group size 64, not the usual 128.** The expert `down_proj` takes an 1856-wide input, and
128 does not divide it. At the default group size 2,944 expert tensors cannot be
quantized; at 64 nothing is misaligned.

Left at bfloat16:

- **`mtp.*`** — the multi-token-prediction head (one `attention` + one `moe` block,
  `num_nextn_predict_layers: 1`). vLLM loads it through the speculative-decoding path
  rather than the main stack. At 2.67 GB it is the largest 16-bit component here, so
  there is real headroom for anyone who measures that vLLM accepts a quantized one.
- **`.gate`** — the 23 MoE routers plus their fp32 `e_score_correction_bias`. Routing
  decides which experts run at all; 0.02 GB is not worth the risk.
- **`backbone.embeddings` and `lm_head`** — precision-sensitive, and untied in this model.
- **`conv1d`** — Mamba causal-convolution kernels, shape `(6144, 1, 4)`. Not Linear
  layers, and quantizers reject them outright.
- **`backbone.norm_f`** and the Mamba `A_log` / `dt_bias` / `D` state-space parameters —
  1-D, never quantizable.

The architecture is a 52-block hybrid: **23 Mamba-2 blocks, 23 MoE blocks and 6 attention
blocks**. Only 6 blocks carry a KV cache, which is what makes long context cheap here (see
below). Of ~30B total parameters, ~3B are active per token — 6 of 128 routed experts plus
one shared expert.

### Checkpoint layout

Experts ship as per-expert 2-D weights in the source already
(`…mixer.experts.{id}.up_proj`, `down_proj`), so no fused-3-D splitting was involved and
the naming carries straight through as `…experts.{id}.up_proj.weight_packed`. vLLM's
NemotronH loader builds its expert mapping with `ckpt_gate_proj_name="up_proj"` and
`ckpt_down_proj_name="down_proj"`, matching this layout.

## Usage

Requires **vLLM >= 0.25.1**, where `NemotronHForCausalLM` and the `nemotron_v3` reasoning
parser are both present. No nightly build needed.

```bash
vllm serve GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W8A16 \
  --max-model-len 262144 \
  --mamba-backend flashinfer \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

Do not pass `--quantization`; compressed-tensors is detected from `config.json`. The int8
W8A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

- **`--tool-call-parser qwen3_coder`** is what the base model card specifies — Nemotron 3.5
  emits the same XML tool-call framing as Qwen3-Coder. `qwen3_xml` is an alias for the same
  parser class in current vLLM.
- **`--reasoning-parser nemotron_v3`** splits thinking into `reasoning_content`.
- **`--mamba-ssm-cache-dtype float16`** halves the Mamba state cache if you are tight on
  memory; the base card pairs it with
  `--enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5`.
- **Speculative decoding**: the base card uses a separate DSpark checkpoint. This build
  also still carries the vendor's MTP head, which vLLM can route through its
  `nemotron_h_mtp` path — `--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'`.
  Neither path is smoke-tested here.

### Fitting the card

Only **6 of 52 blocks use attention**, with 2 KV heads at `head_dim` 128 — about
**6 KB/token**, an order of magnitude cheaper than a conventional 30B. The 23 Mamba blocks
hold a fixed-size recurrent state instead, roughly **48 MB per concurrent sequence** at the
default fp32 SSM cache (half that at float16), independent of sequence length.

| context | KV cache | + weights |
|---|---|---|
| 32k | ~0.2 GB | ~36.1 GB |
| 128k | ~0.8 GB | ~36.7 GB |
| 256k (native max) | ~1.6 GB | ~37.5 GB |

The weights, not the context, are the binding constraint on this build: going from 32k to
the full 256k costs only ~1.4 GB. A 48 GB card is comfortable at the native maximum with
room for concurrency. A 40 GB card fits but with little slack once activations and CUDA
graphs are counted — lower `--max-model-len` before lowering
`--gpu-memory-utilization`. On 32 GB or less, use the W4A16 sibling instead.

`max_position_embeddings` is 262144 — the base card's validated 1M-token configurations use
`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --max-model-len 1048576` on 8×H100 or GB200, which is not
what this single-card build is for. This is arithmetic from `config.json`, not a measured
deployment.

## Reproducing this checkpoint

Built with [llm-quantizer](https://github.com/gotoai/llm-quantizer):

```bash
./llmq.py run --profile nemotron-3.5-lightning-30b-a3b --scheme W8A16
```

which is equivalent to:

```python
# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
    save_directory="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W8A16",
    scheme="W8A16",
    group_size=64,
    ignore=["re:.*\\.gate$", "re:.*\\.conv1d$", "re:.*mtp.*",
            "lm_head", "re:.*\\.embeddings$", "re:.*\\.norm_f$"],
    device="cuda:0",
)
```

The source ships as 14 shards of ~5 GB and a job holds one shard at a time, so no
re-sharding is needed and the build peaks at a few GB of VRAM — it took about a minute on
one consumer GPU.

Two of those ignore patterns are easy to miss. `re:.*\.embeddings$` is needed because this
model calls its embedding table `backbone.embeddings`, not `embed_tokens`. And
`re:.*\.norm_f$` is needed because compressed-tensors auto-skips norms with a literal
`module_name.endswith("norm")` test, which the final norm — `backbone.norm_f`, 1-D
`(2688,)` — misses; without it the run aborts with `expected 2D linear weight`.

## Evaluation

**No benchmarks have been run.** Data-free round-to-nearest quantization degrades quality
more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured
here. Int8 degrades far less than int4 — that is the reason this build exists — but "less"
is not "none". Treat the published Nemotron 3.5 Lightning numbers as describing the
bfloat16 model, not this one.

Two properties of this architecture are why the int8 build is worth its extra 15 GB. The
routers stay 16-bit, so expert *selection* is identical in both builds, but all 2,944
routed experts are quantized without calibration and rarely-activated experts get no more
attention than hot ones — with only 6 of 128 experts active per token, a cold expert's
error is never averaged away. And Mamba blocks carry state across the whole sequence, so
projection error has a longer path to accumulate than in a pure attention stack. Both are
error-accumulation stories, and int8's smaller per-weight error compounds less. Long-context
behaviour on your own traffic is still the thing worth checking.

## License

**OpenMDW License Agreement, version 1.1**, inherited from the base model — the vendor's
`LICENSE` is included unmodified. OpenMDW is permissive: it grants use without restriction
and places no conditions on model outputs. It does require that any distribution retain a
copy of the agreement and all notices of origin, so NVIDIA's `LICENSE` and its
accompanying `safety.md`, `bias.md`, `privacy.md` and `explainability.md` are carried
through here. It also terminates the grant for anyone who brings patent or copyright
litigation over the model. Read it before redistributing a derivative.

"Nemotron" and "NVIDIA" are NVIDIA's marks; this repository is not endorsed by or
affiliated with NVIDIA.