Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+accuracy_plot.png filter=lfs diff=lfs merge=lfs -text
+agentic_coding_benchmarks.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,51 @@
+Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)
+
+By exercising rights granted to you under this agreement, you accept and agree
+to its terms.
+
+As used in this agreement, "Model Materials" means the materials provided to
+you under this agreement, consisting of: (1) one or more machine learning
+models (including architecture and parameters); and (2) all related artifacts
+(including associated data, documentation and software) that are provided to
+you hereunder.
+
+Subject to your compliance with this agreement, permission is hereby granted,
+free of charge, to deal in the Model Materials without restriction, including
+under all copyright, patent, database, and trade secret rights included or
+embodied therein.
+
+If you distribute any portion of the Model Materials, you shall retain in your
+distribution (1) a copy of this agreement, and (2) all copyright notices and
+other notices of origin included in the Model Materials that are applicable to
+your distribution.
+
+If you file, maintain, or voluntarily participate in a lawsuit against any
+person or entity asserting that the Model Materials directly or indirectly
+infringe any patent or copyright, then all rights and grants made to you
+hereunder are terminated, unless that lawsuit was in response to a
+corresponding lawsuit first brought against you.
+
+This agreement does not impose any restrictions or obligations with respect to
+any use, modification, or sharing of any outputs generated by using the Model
+Materials.
+
+THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
+ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
+TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
+
+YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
+APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
+ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
+MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
+REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
+DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
+ANYTHING INCORPORATED OR EMBODIED THEREIN.
+
+IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
+USE THEREOF OR OTHER DEALINGS THEREIN.

README.md

@@ -0,0 +1,222 @@
+---
+license: other
+license_name: openmdw-1.1
+license_link: https://openmdw.ai/license/1-1/
+base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
+base_model_relation: quantized
+pipeline_tag: text-generation
+library_name: transformers
+language:
+  - en
+  - es
+  - fr
+  - de
+  - it
+  - ja
+tags:
+  - compressed-tensors
+  - w8a16
+  - int8
+  - vllm
+  - nemotron-3.5
+  - mamba
+  - moe
+---
+
+# NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W8A16
+
+Int8 weight-only quantization of
+[nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16),
+in compressed-tensors format for vLLM. **35.93 GB**, down from 65.83 GB — a 45% reduction.
+Its KV cache is unusually cheap, so a 48 GB card serves it at the full native 256k context.
+
+This is the fidelity-first build. Int8 round-to-nearest stays far closer to the bfloat16
+weights than int4 does, at ~1.8x the footprint of the
+[int4 W4A16 sibling](https://huggingface.co/GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16)
+(20.49 GB). If you are targeting a 24 or 32 GB card, use that one; use this one when you
+have the VRAM and want the least quality loss quantization can give without calibration.
+The gap matters more than usual here — see [Evaluation](#evaluation).
+
+Unofficial and unaffiliated with NVIDIA. All model capabilities, evaluations and
+limitations belong to the original model card — see the base model for those.
+
+NVIDIA's own optimized deployment build is
+[NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4), which
+needs a **Blackwell** GPU. This repository is for everything else: int8 W8A16 runs on
+Marlin kernels from compute capability 7.5 up, so Ampere, Ada and Hopper cards get a
+weight-quantized build too. The base model card lists "building quantized variants" as one
+of the BF16 repo's intended uses; this is that.
+
+## What was changed
+
+Weights were quantized from bfloat16 to **int8, group size 64, symmetric, weight-only**
+(activations stay 16-bit) using `llmcompressor.model_free_ptq`. No calibration data was
+used and the model was never loaded — the quantizer operates directly on the safetensors.
+Architecture, tokenizer and chat template are the vendor's, unmodified.
+
+6,004 Linear modules were converted, covering 88.6% of the output's bytes:
+
+| component | precision | source | quantized |
+|---|---|---|---|
+| routed experts (128 per MoE layer × 23 layers) | int8 g64 | 58.75 GB | **30.29 GB** |
+| Mamba in/out projections + attention (70 modules) | int8 g64 | 2.06 GB | 1.06 GB |
+| shared expert (46 modules) | int8 g64 | 0.92 GB | 0.47 GB |
+| MTP head (`mtp.*`) | bfloat16 | 2.67 GB | 2.67 GB |
+| `backbone.embeddings` + `lm_head` (untied) | bfloat16 | 1.41 GB | 1.41 GB |
+| MoE routers (`.gate`) | bfloat16 / fp32 | 0.02 GB | 0.02 GB |
+| Mamba `conv1d`, norms, SSM params | bfloat16 | 0.001 GB | 0.001 GB |
+| **total** | | **65.83 GB** | **35.93 GB** |
+
+Of the quantized share, 30.86 GB is packed int8 and 0.96 GB is bf16 group scales. The
+scales are the same count as in the W4A16 sibling — both builds use group size 64 and
+convert the same 6,004 modules — so the int8 payload is almost exactly twice the int4 one
+and everything else is unchanged.
+
+**Group size 64, not the usual 128.** The expert `down_proj` takes an 1856-wide input, and
+128 does not divide it. At the default group size 2,944 expert tensors cannot be
+quantized; at 64 nothing is misaligned.
+
+Left at bfloat16:
+
+- **`mtp.*`** — the multi-token-prediction head (one `attention` + one `moe` block,
+  `num_nextn_predict_layers: 1`). vLLM loads it through the speculative-decoding path
+  rather than the main stack. At 2.67 GB it is the largest 16-bit component here, so
+  there is real headroom for anyone who measures that vLLM accepts a quantized one.
+- **`.gate`** — the 23 MoE routers plus their fp32 `e_score_correction_bias`. Routing
+  decides which experts run at all; 0.02 GB is not worth the risk.
+- **`backbone.embeddings` and `lm_head`** — precision-sensitive, and untied in this model.
+- **`conv1d`** — Mamba causal-convolution kernels, shape `(6144, 1, 4)`. Not Linear
+  layers, and quantizers reject them outright.
+- **`backbone.norm_f`** and the Mamba `A_log` / `dt_bias` / `D` state-space parameters —
+  1-D, never quantizable.
+
+The architecture is a 52-block hybrid: **23 Mamba-2 blocks, 23 MoE blocks and 6 attention
+blocks**. Only 6 blocks carry a KV cache, which is what makes long context cheap here (see
+below). Of ~30B total parameters, ~3B are active per token — 6 of 128 routed experts plus
+one shared expert.
+
+### Checkpoint layout
+
+Experts ship as per-expert 2-D weights in the source already
+(`…mixer.experts.{id}.up_proj`, `down_proj`), so no fused-3-D splitting was involved and
+the naming carries straight through as `…experts.{id}.up_proj.weight_packed`. vLLM's
+NemotronH loader builds its expert mapping with `ckpt_gate_proj_name="up_proj"` and
+`ckpt_down_proj_name="down_proj"`, matching this layout.
+
+## Usage
+
+Requires **vLLM >= 0.25.1**, where `NemotronHForCausalLM` and the `nemotron_v3` reasoning
+parser are both present. No nightly build needed.
+
+```bash
+vllm serve GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W8A16 \
+  --max-model-len 262144 \
+  --mamba-backend flashinfer \
+  --reasoning-parser nemotron_v3 \
+  --enable-auto-tool-choice --tool-call-parser qwen3_coder
+```
+
+Do not pass `--quantization`; compressed-tensors is detected from `config.json`. The int8
+W8A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.
+
+- **`--tool-call-parser qwen3_coder`** is what the base model card specifies — Nemotron 3.5
+  emits the same XML tool-call framing as Qwen3-Coder. `qwen3_xml` is an alias for the same
+  parser class in current vLLM.
+- **`--reasoning-parser nemotron_v3`** splits thinking into `reasoning_content`.
+- **`--mamba-ssm-cache-dtype float16`** halves the Mamba state cache if you are tight on
+  memory; the base card pairs it with
+  `--enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5`.
+- **Speculative decoding**: the base card uses a separate DSpark checkpoint. This build
+  also still carries the vendor's MTP head, which vLLM can route through its
+  `nemotron_h_mtp` path — `--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'`.
+  Neither path is smoke-tested here.
+
+### Fitting the card
+
+Only **6 of 52 blocks use attention**, with 2 KV heads at `head_dim` 128 — about
+**6 KB/token**, an order of magnitude cheaper than a conventional 30B. The 23 Mamba blocks
+hold a fixed-size recurrent state instead, roughly **48 MB per concurrent sequence** at the
+default fp32 SSM cache (half that at float16), independent of sequence length.
+
+| context | KV cache | + weights |
+|---|---|---|
+| 32k | ~0.2 GB | ~36.1 GB |
+| 128k | ~0.8 GB | ~36.7 GB |
+| 256k (native max) | ~1.6 GB | ~37.5 GB |
+
+The weights, not the context, are the binding constraint on this build: going from 32k to
+the full 256k costs only ~1.4 GB. A 48 GB card is comfortable at the native maximum with
+room for concurrency. A 40 GB card fits but with little slack once activations and CUDA
+graphs are counted — lower `--max-model-len` before lowering
+`--gpu-memory-utilization`. On 32 GB or less, use the W4A16 sibling instead.
+
+`max_position_embeddings` is 262144 — the base card's validated 1M-token configurations use
+`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --max-model-len 1048576` on 8×H100 or GB200, which is not
+what this single-card build is for. This is arithmetic from `config.json`, not a measured
+deployment.
+
+## Reproducing this checkpoint
+
+Built with [llm-quantizer](https://github.com/gotoai/llm-quantizer):
+
+```bash
+./llmq.py run --profile nemotron-3.5-lightning-30b-a3b --scheme W8A16
+```
+
+which is equivalent to:
+
+```python
+# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
+# transformers==5.15.1, torch==2.13.0
+from llmcompressor import model_free_ptq
+
+model_free_ptq(
+    model_stub="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
+    save_directory="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W8A16",
+    scheme="W8A16",
+    group_size=64,
+    ignore=["re:.*\\.gate$", "re:.*\\.conv1d$", "re:.*mtp.*",
+            "lm_head", "re:.*\\.embeddings$", "re:.*\\.norm_f$"],
+    device="cuda:0",
+)
+```
+
+The source ships as 14 shards of ~5 GB and a job holds one shard at a time, so no
+re-sharding is needed and the build peaks at a few GB of VRAM — it took about a minute on
+one consumer GPU.
+
+Two of those ignore patterns are easy to miss. `re:.*\.embeddings$` is needed because this
+model calls its embedding table `backbone.embeddings`, not `embed_tokens`. And
+`re:.*\.norm_f$` is needed because compressed-tensors auto-skips norms with a literal
+`module_name.endswith("norm")` test, which the final norm — `backbone.norm_f`, 1-D
+`(2688,)` — misses; without it the run aborts with `expected 2D linear weight`.
+
+## Evaluation
+
+**No benchmarks have been run.** Data-free round-to-nearest quantization degrades quality
+more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured
+here. Int8 degrades far less than int4 — that is the reason this build exists — but "less"
+is not "none". Treat the published Nemotron 3.5 Lightning numbers as describing the
+bfloat16 model, not this one.
+
+Two properties of this architecture are why the int8 build is worth its extra 15 GB. The
+routers stay 16-bit, so expert *selection* is identical in both builds, but all 2,944
+routed experts are quantized without calibration and rarely-activated experts get no more
+attention than hot ones — with only 6 of 128 experts active per token, a cold expert's
+error is never averaged away. And Mamba blocks carry state across the whole sequence, so
+projection error has a longer path to accumulate than in a pure attention stack. Both are
+error-accumulation stories, and int8's smaller per-weight error compounds less. Long-context
+behaviour on your own traffic is still the thing worth checking.
+
+## License
+
+**OpenMDW License Agreement, version 1.1**, inherited from the base model — the vendor's
+`LICENSE` is included unmodified. OpenMDW is permissive: it grants use without restriction
+and places no conditions on model outputs. It does require that any distribution retain a
+copy of the agreement and all notices of origin, so NVIDIA's `LICENSE` and its
+accompanying `safety.md`, `bias.md`, `privacy.md` and `explainability.md` are carried
+through here. It also terminates the grant for anyone who brings patent or copyright
+litigation over the model. Read it before redistributing a derivative.
+
+"Nemotron" and "NVIDIA" are NVIDIA's marks; this repository is not endorsed by or
+affiliated with NVIDIA.

bias.md

@@ -0,0 +1,10 @@
+| Field | Response |
+| :---- | :---- |
+| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None |
+| Bias Metric (If Measured): | [BBQ Accuracy Scores in Ambiguous Contexts](https://github.com/nyu-mll/BBQ/) |
+| Which characteristic (feature) show(s) the greatest difference in performance?: | The model shows high variance in the characteristics when it is used with a high temperature. |
+| Measures taken to mitigate against unwanted bias: | Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) employed to calibrate the model’s reasoning capabilities to maintain logical consistency and appropriate complexity when interacting with or interpreting data from diverse age demographics. |
+| If using internal data, description of methods implemented in data acquisition or processing, if any, to address the prevalence of identifiable biases in the training, testing, and validation data: | The training datasets contain a large amount of synthetic data generated by LLMs. We manually curated prompts. |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | [BBQ](https://github.com/nyu-mll/BBQ/) |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | These datasets, such as web-scraped finance reasoning data derived from SEC EDGAR filings, science and math problem datasets, OpenResearcher/source-document datasets, Common Crawl, CC-News, Wikimedia, and long-context document datasets, do not collectively or exhaustively represent all demographic groups (and proportionally therein). For instance, these datasets do not contain explicit mentions of demographic classes such as age, gender, or ethnicity in approximately 97% to 99.9% of finance reasoning samples and in over 85% of samples across the broader assessed datasets. In the subset where such terms are present, these datasets contain notable representational skews. For example, ethnicity mentions are often dominated by Middle Eastern contexts (found in finance documents) or "White," "Two or more," and "Black or African American" as the most frequent ethnic identifiers, while references categorized as male-only significantly outnumber those categorized as female-only. Furthermore, gender is explicitly mentioned in approximately 12% of samples across the broader dataset assessment, yet in only 0.9% of finance-specific samples. Dataset-level results vary by source type, with long-context/source-document datasets containing higher explicit demographic mention rates compared to certain web-scraped sources. To mitigate these imbalances, we recommend considering evaluation techniques such as bias audits, fine-tuning with demographically balanced datasets, and mitigation strategies such as counterfactual data augmentation to align with the desired model behavior. This evaluation used a 3,000-sample subset per dataset, identified as the optimal threshold for maximizing embedder accuracy. |
+| Unwanted Bias Testing: | Constrained to English-language inputs. Multi-lingual parity is not currently claimed or guaranteed. |

chat_template.jinja

@@ -0,0 +1,190 @@
+{% macro render_extra_keys(json_dict, handled_keys) %}
+    {%- if json_dict is mapping %}
+        {%- for json_key in json_dict if json_key not in handled_keys %}
+            {%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
+                {{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
+            {%- else %}
+                {{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
+            {%- endif %}
+        {%- endfor %}
+    {%- endif %}
+{% endmacro %}
+{%- set enable_thinking = enable_thinking if enable_thinking is defined else True %}
+{%- set truncate_history_thinking = truncate_history_thinking if truncate_history_thinking is defined else True %}
+{%- set ns = namespace(last_user_idx = -1) %}
+{%- set loop_messages = messages %}
+{%- for m in loop_messages %}
+  {%- if m["role"] == "user" %}
+    {%- set ns.last_user_idx = loop.index0 %}
+  {%- endif %}
+{%- endfor %}
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = [] %}
+{%- endif %}
+{%- set ns = namespace(last_user_idx = -1) %}
+{%- for m in loop_messages %}
+  {%- if m["role"] == "user" %}
+    {%- set ns.last_user_idx = loop.index0 %}
+  {%- endif %}
+{%- endfor %}
+{%- if system_message is defined %}
+    {{- "<|im_start|>system\n" + system_message }}
+{%- else %}
+    {%- if tools is iterable and tools | length > 0 %}
+        {{- "<|im_start|>system\n" }}
+    {%- endif %}
+{%- endif %}
+{%- if tools is iterable and tools | length > 0 %}
+    {%- if system_message is defined and system_message | length > 0 %}
+        {{- "\n\n" }}
+    {%- endif %}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n" }}
+    {{- "<tools>" }}
+    {%- for tool in tools %}
+        {%- if tool.function is defined %}
+            {%- set tool = tool.function %}
+        {%- endif %}
+        {{- "\n<function>\n<name>" ~ tool.name ~ "</name>" }}
+        {%- if tool.description is defined %}
+            {{- '\n<description>' ~ (tool.description | trim) ~ '</description>' }}
+        {%- endif %}
+        {{- '\n<parameters>' }}
+        {%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
+            {%- for param_name, param_fields in tool.parameters.properties|items %}
+                {{- '\n<parameter>' }}
+                {{- '\n<name>' ~ param_name ~ '</name>' }}
+                {%- if param_fields.type is defined %}
+                    {{- '\n<type>' ~ (param_fields.type | string) ~ '</type>' }}
+                {%- endif %}
+                {%- if param_fields.description is defined %}
+                    {{- '\n<description>' ~ (param_fields.description | trim) ~ '</description>' }}
+                {%- endif %}
+                {%- if param_fields.enum is defined %}
+                    {{- '\n<enum>' ~ (param_fields.enum | tojson | safe) ~ '</enum>' }}
+                {%- endif %}
+                {%- set handled_keys = ['name', 'type', 'description', 'enum'] %}
+                {{- render_extra_keys(param_fields, handled_keys) }}
+                {{- '\n</parameter>' }}
+            {%- endfor %}
+        {%- endif %}
+        {% set handled_keys = ['type', 'properties', 'required'] %}
+        {{- render_extra_keys(tool.parameters, handled_keys) }}
+        {%- if tool.parameters is defined and tool.parameters.required is defined %}
+            {{- '\n<required>' ~ (tool.parameters.required | tojson | safe) ~ '</required>' }}
+        {%- endif %}
+        {{- '\n</parameters>' }}
+        {%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
+        {{- render_extra_keys(tool, handled_keys) }}
+        {{- '\n</function>' }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+{%- endif %}
+{%- if system_message is defined %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if tools is iterable and tools | length > 0 %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in loop_messages %}
+    {%- if message.role == "assistant" %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
+            {%- set content = "<think>\n" ~ message.reasoning_content ~ "</think>" ~ (message.content | default('', true)) %}
+        {%- else %}
+            {%- set content = message.content | default('', true) %}
+            {%- if content is string -%}
+                {%- if '<think>' not in content and '</think>' not in content -%}
+                    {%- set content = "<think></think>" ~ content -%}
+                {%- endif -%}
+            {%- else -%}
+                {%- set content = content -%}
+            {%- endif -%}
+        {%- endif %}
+        {%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
+            {{- '<|im_start|>assistant\n' }}
+                {%- set include_content = not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
+                {%- if content is string and content | trim | length > 0 %}
+                    {%- if include_content %}
+                        {{- (content | trim) ~ '\n' -}}
+                    {%- else %}
+                        {%- set c = (content | string) %}
+                        {%- if '</think>' in c %}
+                            {%- set c = c.split('</think>')[-1] %}
+                        {%- elif '<think>' in c %}
+                            {%- set c = c.split('<think>')[0] %}
+                        {%- endif %}
+                        {%- set c = "<think></think>" ~ c %}
+                        {%- if c | length > 0 %}
+                            {{- c ~ '\n' -}}
+                        {%- endif %}
+                    {%- endif %}
+                {%- else %}
+                    {{- "<think></think>" -}}
+                {%- endif %}
+                {%- for tool_call in message.tool_calls %}
+                    {%- if tool_call.function is defined %}
+                        {%- set tool_call = tool_call.function %}
+                    {%- endif %}
+                    {{- '<tool_call>\n<function=' ~ tool_call.name ~ '>\n' -}}
+                        {%- if tool_call.arguments is defined %}
+                            {%- for args_name, args_value in tool_call.arguments|items %}
+                                {{- '<parameter=' ~ args_name ~ '>\n' -}}
+                                    {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                                {{- args_value ~ '\n</parameter>\n' -}}
+                            {%- endfor %}
+                        {%- endif %}
+                    {{- '</function>\n</tool_call>\n' -}}
+                {%- endfor %}
+                {{- '<|im_end|>\n' }}
+        {%- else %}
+            {%- if not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
+                {{- '<|im_start|>assistant\n' ~ (content | default('', true) | string | trim) ~ '<|im_end|>\n' }}
+            {%- else %}
+                {%- set c = (content | default('', true) | string) %}
+                {%- if '<think>' in c and '</think>' in c %}
+                    {%- set c = "<think></think>" ~ c.split('</think>')[-1] %}
+                {%- endif %}
+                {%- set c = c | trim %}
+                {%- if c | length > 0 %}
+                    {{- '<|im_start|>assistant\n' ~ c ~ '<|im_end|>\n' }}
+                {%- else %}
+                    {{- '<|im_start|>assistant\n<|im_end|>\n' }}
+                {%- endif %}
+            {%- endif %}
+        {%- endif %}
+    {%- elif message.role == "user" or message.role == "system" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- set content = message.content | string %}
+        {{- content }}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user\n' }}
+        {%- endif %}
+        {{- '<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>\n' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {%- if enable_thinking %}
+        {{- '<|im_start|>assistant\n<think>\n' }}
+    {%- else %}
+        {{- '<|im_start|>assistant\n<think></think>' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,162 @@
+{
+  "architectures": [
+    "NemotronHForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "chunk_size": 128,
+  "conv_kernel": 4,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "expand": 2,
+  "head_dim": 128,
+  "hidden_dropout": 0.0,
+  "hidden_size": 2688,
+  "initializer_range": 0.02,
+  "intermediate_size": 1856,
+  "layer_norm_epsilon": 1e-05,
+  "layers_block_type": [
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "attention",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "attention",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "attention",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "attention",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "attention",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "attention",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe",
+    "mamba",
+    "moe"
+  ],
+  "mamba_head_dim": 64,
+  "mamba_hidden_act": "silu",
+  "mamba_num_heads": 64,
+  "mamba_proj_bias": false,
+  "mamba_ssm_cache_dtype": "float32",
+  "max_position_embeddings": 262144,
+  "mlp_bias": false,
+  "mlp_hidden_act": "relu2",
+  "model_type": "nemotron_h",
+  "moe_intermediate_size": 1856,
+  "moe_latent_size": null,
+  "moe_shared_expert_intermediate_size": 3712,
+  "moe_shared_expert_overlap": true,
+  "mtp_layers_block_type": [
+    "attention",
+    "moe"
+  ],
+  "n_group": 1,
+  "n_groups": 8,
+  "n_routed_experts": 128,
+  "n_shared_experts": 1,
+  "norm_eps": 1e-05,
+  "norm_topk_prob": true,
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 52,
+  "num_key_value_heads": 2,
+  "num_logits_to_keep": 1,
+  "num_nextn_predict_layers": 1,
+  "pad_token_id": 0,
+  "partial_rotary_factor": 1.0,
+  "quantization_config": {
+    "config_groups": {
+      "config_group_0": {
+        "format": "pack-quantized",
+        "input_activations": null,
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": 64,
+          "num_bits": 8,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "group",
+          "symmetric": true,
+          "type": "int",
+          "zp_dtype": null
+        }
+      }
+    },
+    "format": "pack-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "re:.*\\.gate$",
+      "re:.*\\.conv1d$",
+      "re:.*mtp.*",
+      "lm_head",
+      "re:.*\\.embeddings$",
+      "re:.*\\.norm_f$"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "transform_config": {},
+    "version": "0.18.1.a20260818"
+  },
+  "rescale_prenorm_residual": true,
+  "residual_in_fp32": false,
+  "rope_theta": 10000,
+  "routed_scaling_factor": 2.5,
+  "sliding_window": null,
+  "ssm_state_size": 128,
+  "tie_word_embeddings": false,
+  "time_step_floor": 0.0001,
+  "time_step_max": 0.1,
+  "time_step_min": 0.001,
+  "topk_group": 1,
+  "transformers_version": "4.57.6",
+  "use_bias": false,
+  "use_cache": true,
+  "use_conv_bias": true,
+  "use_mamba_kernels": true,
+  "vocab_size": 131072
+}

explainability.md

@@ -0,0 +1,14 @@
+| Field | Response |
+| :---- | :---- |
+| Intended Task/Domain: | Text generation, reasoning, and chat |
+| Model Type: | Text-to-text Mamba2-Transformer Hybrid |
+| Intended Users: | Generative AI creators working with conversational AI models and image content. |
+| Output: | Text |
+| Tools used to evaluate datasets to identify synthetic data and ensure data authenticity. | We used a Gemma-3 4B-based filtering model fine-tuned on [Nemotron Content Safety Dataset v2](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) to ensure the quality of synthetic data. |
+| Describe how the model works: | Generates text by predicting the next word or token based on the context provided in the input sequence using multiple self-attention layers. |
+| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Age, Disability Status, Gender Identity, Nationality, Physical Appearance, Ethnicity, Socioeconomic Status, Sexual Orientation, Religion |
+| Technical Limitations & Mitigation: | This model performs particularly well in instruction following regimes, as such may be strongly influenced by untrusted inputs and should be paired with appropriate guardrails and data filtering to better align use-case behaviors when exposed to such data. |
+| Verified to have met prescribed NVIDIA quality standards: | Yes |
+| Performance Metrics: | Accuracy, Throughput, and User-side throughput |
+| Potential Known Risks: | The model was optimized explicitly for instruction following and as such is more susceptible to prompt injection and jailbreaking in various forms as a result of its instruction tuning. This means that the model should be paired with additional rails or system filtering to limit exposure to instructions from malicious sources -- either directly or indirectly by retrieval (e.g. via visiting a website) -- as they may yield outputs that can lead to harmful, system-level outcomes up to and including remote code execution in agentic systems when effective security controls including guardrails are not in place. The model may generate answers that may be inaccurate, omit key information, include irrelevant or redundant text, or produce socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. The model may exhibit self-anthropomorphism (e.g., displaying human-like characteristics in dialogue, such as expressing preferences and emotions). |
+| Licensing: | Use of this model is governed by the [OpenMDW License Agreement, version 1.1](https://raw.githubusercontent.com/OpenMDW/OpenMDW/refs/heads/main/1.1/LICENSE.OpenMDW-1.1) (OpenMDW-1.1). |

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": [
+    2,
+    11
+  ],
+  "pad_token_id": 0,
+  "top_p": 0.95,
+  "transformers_version": "4.57.6",
+  "temperature": 1.0
+}

model-00001-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79c9ef6cddd3779fec55991c07d954543084656ae96635b8a1f82e92a911f29a
+size 2915453848

model-00002-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3740c71bc76b18a4b8013c8bc97e5dd06d871ffb0e0fa0169c01c638abf343ab
+size 2575218432

model-00003-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b933d41543434ed4e802aad2f910bf2c798410059b5789192c04335bce4c0644
+size 2575219440

model-00004-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39ab7ff029bb0d743c740983ac11238bdce5a39af72f2a9526b40a5928ded615
+size 2576811656

model-00005-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19d03020dfcc47fa5377159933bdde2f38dd2f61ff4fdd782ed4bb5079f788ed
+size 2569005224

model-00006-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2ead9f0aa074d3a9fd40efad24a3d0e6a5fd2f6c6811d99eeced896082f295e
+size 2573210456

model-00007-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28bcb1b29ca06282b89bbb012f065c54685eb68c2a3ef9369f5d8f3978a27a51
+size 2575219816

model-00008-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9458cae347dea65c34e1ed202a0b773965f872b307ae8d6137791352708f396
+size 2575219904

model-00009-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3daf90edb5e0de300242854ad55f533a9d3605fc59c719e2bb65ae68e521483f
+size 2576811656

model-00010-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a871d444acfc8285a011a43ca2720af25cfdbb9752c5b20575d6abd32cd49674
+size 2575219904

model-00011-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c351426be56cee7adf20e76b75114034c01df7079014d949025bd233b7d382ee
+size 2576811664

model-00012-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29c93fd77b3fe9c93a858916324ae3b7b5a89f3928c066a549dba3297d92a01f
+size 2576811680

model-00013-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9618c85646c96977a8a0a5b307dec07315c81653c72f08a2480633202eea18f0
+size 2017370440

model-00014-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64577b275ca4e7e5266eae0903674f7f46ec2a8cbf4f4f1a3207f80d503cd1d0
+size 2670685240

privacy.md

@@ -0,0 +1,5 @@
+| Privacy Information |
+| :--- |
+| Nemotron 3.5 Lightning was trained on large-scale publicly available data that may contain images, audio-video, and text relating to people. NVIDIA collected and used this data in compliance with applicable data protection and privacy laws. This model was not designed to derive insights or otherwise learn from any personal data contained in the datasets. |
+| NVIDIA uses a combination of filters, data minimization techniques, and other guardrails to help prevent personal data from being recited by our models. We employ automated tools and data processing techniques during pre-training or training to identify and filter certain categories of personal data. |
+| Please review NVIDIA's [Privacy Policy](https://www.nvidia.com/en-us/about-nvidia/privacy-policy/) for more information. |

safety.md

@@ -0,0 +1,9 @@
+| Field | Response |
+| :---- | :---- |
+| Model Application Field(s): | Chat, Instruction Following, Chatbot Development, Code Generation, Reasoning, Customer Service |
+| Describe the life critical impact (if present). | Not Applicable |
+| Description of methods implemented in data acquisition or processing, if any, to address other types of potentially harmful data in the training, testing, and validation data: | We used a guard model for content safety to exclude potentially harmful data from training.  |
+| Description of any methods implemented in data acquisition or processing, if any, to address illegal or harmful content in the training data, including, but not limited to, child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII) | We used a Gemma-3 4B-based guard model trained on [Nemotron Content Safety Dataset v2](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) for content safety to exclude potentially illegal or harmful content from the training. |
+| Use Case Restrictions: | Use of this model is governed by the [OpenMDW License Agreement, version 1.1](https://raw.githubusercontent.com/OpenMDW/OpenMDW/refs/heads/main/1.1/LICENSE.OpenMDW-1.1) (OpenMDW-1.1).|
+| Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development.  Restrictions enforce dataset access during training, and dataset license constraints adhered to. |
+| This AI model was developed based on our policies to ensure responsible data handling and risk mitigation. The datasets used for training have been scanned for harmful content and illegal content, consistent with our policies including scanning for Child Sexual Abuse Material (CSAM). Ongoing review and monitoring mechanisms are in place based on our policies and to maintain data integrity. | True. We use [Nemotron Content Safety Dataset V2](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) and an internal safety dataset specialized for minority sexuality for content safety evaluation to ensure the safety of this model. |

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:623c34567aebb18582765289fbe23d901c62704d6518d71866e0e58db892b5b7
+size 17077484