Instructions to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
./llama-cli -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
./build/bin/llama-cli -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Use Docker

docker model run hf.co/tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

LM Studio
Jan

vLLM

How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Ollama
How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with Ollama:
```
ollama run hf.co/tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4
```
Unsloth Desktop

How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4
```

Lemonade

How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Run and chat with the model

lemonade run user.NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF-NVFP4

List all available models

lemonade list

Hermes Agent

How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF:NVFP4" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

NVIDIA Nemotron 3.5 Lightning 30B-A3B - NVFP4 GGUF for llama.cpp

A GGUF repack of NVIDIA's Nemotron 3.5 Lightning 30B-A3B (NVFP4) so it runs on llama.cpp's native Blackwell (sm_120) FP4 tensor-core path. The 4-bit NVFP4 expert weights are copied bit-exact into GGUF; the remaining tensors (attention, token embedding, Mamba-2 projections, and the MTP head) are stored at Q8_0. The 4-bit weights are never re-quantized, so the result is numerically faithful to NVIDIA's checkpoint.

About the model

Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + attention model: 30B total parameters (~3B active), 52 layers of which only 6 use full attention, 128 routed experts (6 active per token) plus 1 shared expert, and a native context length of 1,048,576 tokens.

Because only 6 of the 52 layers keep a growing KV cache (the Mamba-2 and MoE layers carry a fixed-size state), the model holds its full 1M-token context on a single 24 GB GPU - something a dense model of this size cannot do.

Files

File	Size	What it is
`Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf`	18.5 GB	Base model. Runs standalone; full 1M context on a 24 GB card.
`mtp-Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf`	2.0 GB	Optional MTP head. A speculative-decoding draft, loaded via `-md` alongside the base.

Grab the base model for normal use; add the MTP head only if you want speculative decoding.

Running it (llama.cpp)

Base model, full 1M context on a 24 GB card:

llama-server -m Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -ngl 99 -c 1048576 -fa on --cache-type-k q4_0 --cache-type-v q4_0

With the MTP head for speculative decoding (draft depth 2 is the sweet spot; it uses more memory, so reduce the context on a 24 GB card):

llama-server -m Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -md mtp-Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -ngl 99 -c 524288 -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-ngl 99 \
    --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

The MTP head is a modest, consistent speedup - roughly +7% on fast GPUs and low-double-digit percent on slower hardware (see the chart above) - at the cost of some context headroom on 24 GB. It affects throughput only, never output quality: the draft's tokens are always verified by the base model.

Quality

Evaluated on three benchmarks from the TNG benchmark suite (MMLU-Pro and terminal-bench 2.0 on distilled subsets, IFBench in full), for the released configuration (base model + external MTP head):

Benchmark	Score
MMLU-Pro	79.2%
IFBench	70.7%
terminal-bench 2.0	26.7%

The repack is numerically faithful to NVIDIA's NVFP4 checkpoint by construction, and these scores are consistent with the source model within measurement noise. The benchmarks use distilled subsets with small sample counts, so single-point differences are noisy - terminal-bench in particular varies substantially run to run; treat the numbers as indicative rather than precise.

How it was made

Converted with llama.cpp's convert_hf_to_gguf.py: the routed experts, shared expert, and LM head stay NVFP4 (repacked bit-exact); the token embedding, attention projections, Mamba-2 in/out projections, and the MTP head are stored at Q8_0 (near-lossless). No dequantize-and-requantize step, so the vendor's calibrated 4-bit weights are preserved exactly.

License and attribution

This is a format repackage (not a retrain) of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4. The only changes are quantization / format conversion to GGUF, documented under How it was made.

Attribution: please credit NVIDIA for the Nemotron 3.5 Lightning model and its NVFP4 (TensorRT Model Optimizer) export, alongside this repackage.

Trademarks: "NVIDIA" and "Nemotron" are trademarks of NVIDIA Corporation. The OpenMDW license does not grant trademark rights; these names are used here only to identify the upstream model. This repackage is not affiliated with, sponsored by, or endorsed by NVIDIA.

Downloads last month: 329

GGUF

Model size

32B params

Architecture

nemotron_h_moe

Hardware compatibility

4-bit

Model tree for tngtech/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

Base model

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Quantized

(5)

this model