impacte/NVIDIA-Nemotron-3.5-Lightning-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B (GGUF)

GGUF conversion of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, an open 30B mixture-of-experts (MoE) model with ~3B active parameters, built by NVIDIA for the execution layer of always-on agents. Converted for use with llama.cpp using imatrix-calibrated quantization.

One quantization is provided:

Quantization	File	Size	Use case
IQ4_XS	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf	18.7 GB	Fits entirely on a 24 GB GPU (e.g. 16 GB + 8 GB dual) with room for a 256K KV cache

Model Summary

Property	Value
Base model	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Architecture	NemotronHForCausalLM (nemotron_h_moe) — hybrid Mamba-2 + MoE + Attention with Multi-Token Prediction (MTP)
Parameters	~30B total / ~3B active per token
Experts	128 routed + 1 shared, 6 routed active per token
Layers	52 (interleaved; ~6 attention, rest Mamba-2/MoE)
Hidden size	2,688
Context length	262,144 (256K native, up to ~1M via rope scaling)
Vocab size	131,072
Reasoning	Yes (thinking mode, <think> blocks)
Tool calling	Native (<tool_call> XML format)
Modalities	Text
Languages	en, es, fr, de, it, ja

Files

impacte/NVIDIA-Nemotron-3.5-Lightning-GGUF/
├── NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf   # IQ4_XS imatrix GGUF (18.7 GB)
└── .gitattributes                                        # LFS tracking

Usage

llama.cpp (local inference)

# IQ4_XS (24 GB GPU, full 256K context)
llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf \
  --ctx-size 262144 \
  --port 8080

For the full 256K context, use a quantized KV cache (--cache-type-k q4_0 --cache-type-v q4_0) to keep the KV cache footprint minimal:

llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf \
  --ctx-size 262144 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --port 8080

Then call the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS",
    "messages": [
      {"role": "user", "content": "Explain what a Tauri v2 app is."}
    ]
  }'

Ollama

# Pull the pre-built Ollama tag (256K context, q4_0 KV cache)
ollama run oamazonasgabriel/nemotron-3.5-lightning:iq4-xs-256k-24gbGPU

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf",
    n_ctx=262144,
    n_gpu_layers=-1,  # offload all layers to GPU
)

About the base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B is an open 30B-parameter mixture-of-experts (MoE) model with ~3B active parameters, built by NVIDIA for the execution layer of always-on agents. It uses a hybrid Mamba-2 + MoE + attention architecture with only ~6 attention layers out of 52, so the KV cache stays tiny even at very long contexts. The model supports native tool calling and thinking mode, and is multilingual (en, es, fr, de, it, ja).

License & Attribution

• Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — released under the OpenMDW-1.1 license. Review before commercial use.
• GGUF conversion: performed with llama.cpp's convert_hf_to_gguf.py (bf16) and llama-quantize (IQ4_XS with imatrix calibration).

Note: This is a GGUF conversion of a model under the OpenMDW-1.1 license. Ensure your use complies with that license.

Limitations

• The IQ4_XS GGUF (18.7 GB) fits on a 24 GB GPU (16 GB + 8 GB dual) with room for a 256K KV cache at q4_0, but trades some precision vs higher quants (Q6_K / Q8_0).
• The base model's general capabilities are retained; this is a direct conversion with no additional fine-tuning.