Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF

Nemotron-3-Nano-Omni-30B-A3B-Reasoning — Text-Only GGUF

Text-only GGUF conversion of nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, released by NVIDIA on 2026-04-28.

This conversion strips the vision (CRADIO v4-H) and audio (Parakeet) encoders and packages only the language model core for use with llama.cpp and Ollama. The text core is a Mamba2-Transformer hybrid Mixture-of-Experts (30B total, 3B active) with reasoning tuning.

What this is — and what it isn't

• ✅ The full text-reasoning capability of Nemotron-3-Nano-Omni, in a single text-only GGUF.
• ✅ Identical text weights to the omni release (extracted from language_model.* tensors, prefix stripped, vision/audio dropped).
• ❌ Not multimodal. The GGUF cannot accept images, audio, or video. For full omni capability, use NVIDIA's official BF16 / FP8 / NVFP4 weights with a transformers-compatible runtime — multimodal heads in llama.cpp would require an upstream PR adding support for CRADIO + Parakeet.
• ⚠ Distinct from nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16. That's NVIDIA's text-only sister model. Our weights come from the omni variant's text encoder, which was co-trained with the multimodal heads. Behavior may differ slightly from the standalone text-only release.

Architecture

Field	Value
Architecture	Mamba2-Transformer hybrid MoE (NemotronHForCausalLM)
Hybrid pattern	MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME
Parameters	~30B total, ~3B active per token
Hidden size	2688
Layers	52
Mamba heads	64
Attention heads	32 (head_dim 128)
Routed experts	128
Shared experts	1
Top-k routing	6
Vocab	131,072
Context	32K (per chat_template.jinja)

Quants

This is a 30B-A3B MoE — only 3B params active per token. MoE architectures hold quality well at lower bit widths because routing isolates each token's compute to a small fraction of the model. The Q4_K_M default is the accessible end here; sub-Q4 quants would be wasted disk for quality lost on a model this sparse.

Quant	Size	Use case
Q4_K_M	~17 GB	recommended default — accessible end, runs on consumer hardware
Q5_K_M	~21 GB	bump quality if you have headroom
Q6_K	~25 GB	near-lossless reasoning
Q8_0	~32 GB	reference quality
F16	~60 GB	full precision (uploaded on request — useful for further quantization)

(Sizes approximate — actual sizes confirmed once conversion completes. 30B MoE means total params, not active — disk size scales with total.)

Usage — Ollama

# Pull a quant
huggingface-cli download Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF \
    Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf \
    --local-dir ./nemotron-omni

# Build local Ollama model
cd ./nemotron-omni
cat > Modelfile <<EOF
FROM ./Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf
PARAMETER num_ctx 32768
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1.05
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
EOF
ollama create nemotron-3-omni-text:Q4_K_M -f Modelfile

# Use
ollama run nemotron-3-omni-text:Q4_K_M

A Modelfile is included in this repo.

Reasoning toggle

The chat template supports thinking-mode toggles via tokens in user messages:

• /think — enable thinking
• /no_think — suppress thinking

You can also pass "think": false at the top level of /api/chat (NOT inside options) to suppress thinking via the API.

Usage — llama.cpp

./build/bin/llama-cli \
    -m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf \
    -c 32768 \
    -p "Explain MoE routing in three sentences." \
    -n 256 \
    --temp 0.6

License

Use is governed by the NVIDIA Open Model Agreement. Commercial use is permitted under the agreement's terms. This conversion is a derivative work — same license applies.

Conversion details

• Source: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 downloaded 2026-04-28
• Tools: llama.cpp (commit 1a635cde0) convert_hf_to_gguf.py with the existing NemotronHForCausalLM registration (no patches required to the converter)
• Steps: extracted language_model.* tensors → stripped prefix → wrote clean NemotronHForCausalLM config → ran converter → quantized

Limitations & caveats

• Text-only. Drop the omni weights if you need vision/audio.
• Hybrid Mamba2-Attention layers. Some llama.cpp/Ollama features that assume pure-attention models may behave unexpectedly (e.g. context shifting). Standard generation works fine.
• Brand-new architecture. Released the same day as this conversion. Expect rough edges; please open issues at the discussions tab.
• Quant quality on hybrid models. Mamba2 layers may be more quant-sensitive than pure attention. If you see degraded reasoning at Q4_K_M, try Q5_K_M or Q6_K.

Acknowledgments

• NVIDIA for the open release of Nemotron-3-Nano-Omni
• The llama.cpp team for NemotronHForCausalLM support
• This conversion produced for the Zenith swarm — autonomous engineering collective project

Citation

If you use this GGUF, please cite NVIDIA's original release:

@misc{nvidia2026nemotron3nanoomni,
  title  = {Nemotron-3-Nano-Omni-30B-A3B-Reasoning},
  author = {NVIDIA},
  year   = {2026},
  month  = {April},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16}
}