README.md

---
license: apache-2.0
base_model: IFM/K2-Horizon-MoVA-36B-A4B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
- vllm
- compressed-tensors
- nvfp4
- fp8
- mixed-precision
- quantized
- moe
- mova
- k2-horizon
- reasoning
- tool-calling
- blackwell
thumbnail: https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8/resolve/main/assets/banner.png
---

<p align="center">
  <img src="assets/banner.png" alt="Primitive — K2-Horizon-MoVA-36B-A4B mixed NVFP4/FP8" width="100%">
</p>

<p align="center">
  <img src="https://img.shields.io/badge/size-44.48%20GB-F98424?style=flat-square" alt="size 44.48 GB">
  <img src="https://img.shields.io/badge/vs%20BF16-1.7×%20smaller-F98424?style=flat-square" alt="1.7x smaller than BF16">
  <img src="https://img.shields.io/badge/vs%20official%20FP8-8%25%20smaller-F98424?style=flat-square" alt="8 percent smaller than the official FP8">
  <img src="https://img.shields.io/badge/knowledge-level%20with%20BF16%20%26%20FP8-3DD13D?style=flat-square" alt="knowledge level with BF16 and FP8">
  <img src="https://img.shields.io/badge/call%20accuracy-1.4%20below%20FP8-lightgrey?style=flat-square" alt="call accuracy 1.4 below the official FP8">
  <img src="https://img.shields.io/badge/vs%20our%20NVFP4-%2B1.7%20call%20accuracy-F98424?style=flat-square" alt="plus 1.7 call accuracy versus our pure NVFP4 build">
  <img src="https://img.shields.io/badge/requires-Blackwell-lightgrey?style=flat-square" alt="requires Blackwell">
  <a href="https://primitive.com"><img src="https://img.shields.io/badge/web-primitive.com-lightgrey?style=flat-square" alt="primitive.com"></a>
</p>

<h3 align="center">8% smaller than the official FP8, level on knowledge, 1.4 behind on call accuracy.</h3>

<p align="center">
Mixed-precision quantization of <a href="https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B">IFM/K2-Horizon-MoVA-36B-A4B</a> to
<b>44.48 GB</b>, on one 96 GB card.<br>
15 of the 45 sparse layers carry NVFP4 experts; the other 30 carry FP8 with block scales.
</p>

> **This is the better of our two builds on call accuracy, and it needs Blackwell.** There is also a
> [pure NVFP4 build](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) at
> 36.74 GB that is 7.74 GB smaller, 1.21× faster at concurrency 32, and **the only one of the two
> that runs on Ampere**. Take this one if tool calling matters most; take that one for pre-Blackwell
> hardware, the smallest footprint, or throughput under load.

---

## Why this quant

* 🗜️ **8% smaller than IFM's own FP8, 1.7× smaller than BF16.** 44.48 GB against 48.36 GB and
  74.89 GB.
* 🎯 **Knowledge is level with both references.** 87.3 against the official FP8's 86.8 and BF16's
  85.9 over 1,170 items. It is the highest number in that column and it is still a tie.
* 🧭 **The better of our two builds on tool calls.** 73.5 on call accuracy against the pure NVFP4
  build's 71.8, for 7.74 GB more — the one axis that separates the two (t = +2.4). Against the
  official FP8 the gap is 1.4 and neither half of the suite separates on its own.
* 🧠 **Block scales, not bit width, are what moved tool calling here.** Two thirds of the expert
  layers carry FP8 with `[128,128]` block scales rather than group-wise 4-bit.
* 🖥️ **One card, no tensor parallelism.** Both official recipes use TP=2; this fits TP=1 with
  231,488 tokens of KV at 32K context.
* ⚠️ **Blackwell.** Two thirds of the expert layers carry FP8 block scales and sm_80 has no FP8
  tensor cores, so we neither built nor tested this one for Ampere. The pure NVFP4 build is verified
  there.
* 🔧 **Serves on stock vLLM `main`, no patches and no custom kernels.** Plain `compressed-tensors`.

---

## Serve it

K2-Horizon support merged into vLLM `main` on 3 September 2026
([#55063](https://github.com/vllm-project/vllm/pull/55063)) and is **not in a release yet** —
0.28.0 predates it. Use a `main`/nightly build:

```bash
hf download primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8 \
  --local-dir ./K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
  vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c \
  --model /models/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8 \
  --trust-remote-code --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser k2_horizon
```

That image tag is the nightly every number on this page was measured on (vllm
`0.28.1rc1.dev388`). On one RTX PRO 6000 Blackwell it leaves 43.02 GiB resident for weights and
42.39 GiB for KV — **231,488 tokens at 32K context, 7.06× concurrency**. Expect a slow first boot:
engine init takes 198 s, of which 90 s is `torch.compile`, because the FP8 block-scale path compiles
more kernels than a pure NVFP4 one (97 s and 3 s respectively for our other build).

**Reasoning effort is a server-level choice on this architecture, not a per-request one.** The
template emits a different tag pair per effort and `K2HorizonReasoningParser` picks one pair once
at startup, defaulting to `high`:

- `high` (the default) — `<ifm|think>` … `</ifm|think>`
- `medium` — `<ifm|think_fast>` … `</ifm|think_fast>`
- `low` — `<ifm|think_faster>` … `</ifm|think_faster>`

Send a per-request `reasoning_effort: low` against a server parsing `high` and the tags never
match, so the whole reasoning trace stays in `content` instead of being separated out. Set the
effort on the server. Split reasoning is returned in `message.reasoning`.

---

## Measured

**1,370 items across fourteen public benchmarks.** A 1,170-item knowledge suite — MMLU-Pro,
ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500 — and a
200-item tool-calling suite — BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, When2Call — under one fixed
protocol for every row: [PQE-1.2](https://huggingface.co/datasets/primitive-ai/pqe),
`temperature 0.6` / `top_p 0.95` / `top_k 20`, thinking forced on, a 16,384-token budget,
concurrency 32, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.

Auto-scored, no LLM judge. A knowledge item is scored on the last `ANSWER:` in the reply; a
tool-calling item counts only if every expected call is present with acceptable arguments and no
call is invented. Sizes are the safetensors weights, and `±` is the sample standard deviation over
the runs counted in the `runs k/a` column.

| build | size | knowledge | call | abstain | tool calling | runs k/a | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| [IFM BF16](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B) | 74.89 GB | 85.9 ±0.7 | 74.6 ±1.3 | 60.8 ±2.9 | 71.8 ±1.0 | 2/3 | 95.8% | 1206 |
| [IFM FP8](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-FP8) | 48.36 GB | 86.8 ±0.5 | 74.9 ±1.5 | 60.4 ±2.9 | 72.0 ±0.8 | 2/6 | 96.1% | 1176 |
| **this repo** | **44.48 GB** | 87.3 ±1.0 | 73.5 ±1.7 | 58.9 ±3.1 | 70.6 ±1.4 | 2/9 | 96.3% | 1118 |
| [our pure NVFP4](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) | 36.74 GB | 86.5 ±0.5 | 71.8 ±1.0 | 60.4 ±3.7 | 69.5 ±1.4 | 2/6 | 95.6% | 1234 |

`call` is accuracy on the 160 tool-calling items that require a call; `abstain` is the 40 whose
correct action is to call **nothing**; `tool calling` is the two pooled as 200. We report the halves
separately because pooling hides which one moved. `runs k/a` is the number of knowledge and agentic
runs behind each row. `finished` is the share of the 1,170 knowledge items that answered inside the
token budget; `out/answer` is mean output tokens.

**Knowledge is one group.** This build's 87.3 is the highest number in the column and it does not
separate from anything: against the official FP8 it is t = +0.6, at n=2 per side. Don't rank on this
column.

**Tool calling: where the 1.4 sits.** The pooled 200-item number separates from the official FP8
(t = −2.6 over nine runs here and six there), but neither half does on its own: `call` is 1.4 low at
t = −1.7 and `abstain` is 1.5 low at t = −1.0. So the loss is small, real in aggregate, and spread
across both halves of the suite rather than concentrated in one. Our pure NVFP4 build behaves
differently — there the loss is 3.1 on `call` alone (t = −4.2) and abstention is untouched.

It is not a formatting failure. Unparsed replies run at 14 per 1,800 here (0.78%) against 13 per
1,200 for the official FP8 (1.08%) and 6 per 600 for BF16 (1.00%) — cleaner than either IFM build,
and our pure NVFP4 build is cleaner still at 0.75%. Thinking fired on 99–100% of items in every row.

<details>
<summary>Per-benchmark detail, both suites</summary>

| benchmark | items | IFM BF16 | IFM FP8 | **this repo** | our pure NVFP4 |
|---|---|---|---|---|---|
| **tool calling** | | | | | |
| bfcl_v4 | 30 | 83.3 | 81.7 | 81.9 | 80.6 |
| xlam / APIGen | 45 | 44.4 | 44.8 | 40.2 | 37.8 |
| toolace | 45 | 76.3 | 78.1 | 77.8 | 75.2 |
| glaive v2 | 40 | 100.0 | 100.0 | 99.7 | 99.6 |
| when2call | 40 | 60.8 | 60.4 | 58.9 | 60.4 |
| **knowledge** | | | | | |
| mmlu_pro | 200 | 85.5 | 86.5 | 87.5 | 84.5 |
| math_500 | 100 | 89.5 | 88.5 | 89.5 | 89.0 |
| gsm8k | 100 | 97.5 | 97.0 | 97.5 | 97.5 |
| arc_challenge | 150 | 98.7 | 97.0 | 98.0 | 98.0 |
| hellaswag | 150 | 68.0 | 73.3 | 74.7 | 70.0 |
| winogrande | 150 | 80.7 | 81.3 | 78.7 | 84.3 |
| commonsenseqa | 120 | 75.4 | 79.2 | 80.0 | 77.5 |
| boolq | 100 | 90.0 | 91.0 | 92.0 | 91.0 |
| openbookqa | 100 | 96.0 | 94.0 | 94.5 | 94.5 |

Pooled over every run behind the row, so a cell rests on `runs × items`. `when2call` supplies the
40 abstention items; the other four tool-calling benchmarks supply the 160 call items.

**Where the tool-calling difference sits.** xlam/APIGen is 40.2 here against the official FP8's
44.8, 4.6 points on 45 items and this build's largest gap against that reference. The pure NVFP4
build gives up 7.0 points on the same benchmark, so this build recovers about a third of that loss.
xlam items carry the longest argument lists in the suite.

</details>

> **The noise floor, and why the run counts are in the table.** Every build compared here has at
> least six agentic runs, the one exception being the BF16 reference at three. It needs to be six.
> The official FP8's tool-calling figure read 72.7 ±0.3 after three runs and 72.0 ±0.8 after six;
> this build's read 70.3 ±1.6 after three and 70.6 ±1.4 after nine. On the 1,170-item knowledge
> suite, repeat runs of a single checkpoint span up to 1.9 points, so **treat a knowledge gap of a
> point or so as unresolved.** The widest gap in that column is 1.4, between this build and BF16,
> and even that is only t = +1.6. We publish ties as ties.

Two footnotes on the table. The BF16 row was served at 20,480 context against the quants' 32,768,
because 74.89 GB of weights leaves only 81,664 tokens of KV on a 96 GB card; the generation budget
was the same 16,384 tokens for every row. And `out/answer` is a mean over a long tail: median output
is 204 tokens against a mean of 1,118, a ratio of 5.5×. That shape usually means a model that
reaches its answer and cannot stop. Here it does not — the official FP8 shows 1,176 against a
median of 211 on the same items, a ratio of 5.6×, with the same ~4% of items hitting the budget.
It is how K2-Horizon reasons, and quantization did not cause it.

---

## Throughput

Cache-free, 8,192 tokens in and 512 out, one RTX PRO 6000 Blackwell, same nightly and same serve
flags as the accuracy rows. Prefix caching is disabled and every measurement runs on its own seed,
so no row is reading a cache the previous one filled. These come from one sitting and drift a few
percent between sittings, so read them as a bracket. BF16 is absent because at 74.89 GB it only
serves 20,480 context on this card, which would not be the same measurement.

| build | tok/s @ conc 1 | TPOT @ conc 1 | tok/s @ conc 32 | TPOT @ conc 32 | median TTFT @ conc 32 |
|---|---|---|---|---|---|
| [IFM FP8](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-FP8) | 29.74 | 32.83 ms | 240.6 | 89.2 ms | 4.46 s |
| **this repo** | 29.99 | 32.56 ms | 248.4 | 89.5 ms | 3.59 s |
| [our pure NVFP4](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) | **30.45** | **32.10 ms** | **301.5** | 93.1 ms | **1.54 s** |

**Quantizing this model buys throughput under load, not single-stream latency.** Those three builds
span 11.6 GB and land within 0.71 tok/s of each other at concurrency 1. At concurrency 32 their
per-token decode cost is nearly flat too — 89.2, 89.5 and 93.1 ms, a 4.4% spread — so the aggregate
differences are not a decode-rate effect. They show up in prefill and in scheduling under load,
where median time to first token falls from 4.46 s to 1.54 s across the range. Decode stays pinned
because vLLM restacks the 15.1 GB of BF16 MoVA value-experts on every forward pass whatever the
routed experts are quantized to, and that fixed cost dominates a single stream.

This build sits between the two on all five columns, which is what its size predicts. If throughput
under load matters more than call accuracy, the pure NVFP4 build is 1.21× this one at concurrency 32
and reaches first token in 1.54 s against 3.59 s.

---

## Choosing between our two builds

| | **this repo** | [pure NVFP4](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) |
|---|---|---|
| size | 44.48 GB | **36.74 GB** |
| knowledge | 87.3 ±1.0 | 86.5 ±0.5 |
| call | **73.5 ±1.7** | 71.8 ±1.0 |
| abstain | 58.9 ±3.1 | 60.4 ±3.7 |
| tool calling, pooled | 70.6 ±1.4 | 69.5 ±1.4 |
| tok/s @ conc 32 | 248.4 | **301.5** |
| median TTFT @ conc 32 | 3.59 s | **1.54 s** |
| Blackwell | yes | yes |
| **Ampere (A100)** | **no** | **yes** |

This build spends 7.74 GB, +21%, and call accuracy is the one axis that pays it back: +1.7 at
t = +2.4. The pooled tool-calling lead of 1.1 does not separate (t = +1.5), nor does knowledge
(t = +0.9). **If tool calling dominates your workload and you have 48 GB, IFM's own FP8 is still the
most accurate option** — it leads this build by 1.4 on call accuracy. What this build offers against
it is 8% less memory at a gap that neither half of the suite resolves on its own.

**On Ampere.** We have not tested this build on an A100 and do not expect FP8 block scales to load
on sm_80, which has no FP8 tensor cores. The pure NVFP4 build does run there through Marlin
dequantization, verified on an A100 40 GB: loading, serving and generating correctly with the
reasoning parser working.

---

## Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different `temperature`, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above — the
same 1,370 items, auto-scored, no LLM judge.

| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| [Laguna-XS-2.1](https://huggingface.co/primitive-ai/Laguna-XS-2.1-mixed-NVFP4-MXFP8) | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
| [Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8) | 30 B MoE+Mamba | 19.2 GiB | 87.1 | 87.9 | 85.4 | 70.5 | 97.9% | 1429 |
| [Ornith-1.5-35B-A3B](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8) | 35 B MoE | 22.6 GiB | 88.7 | 91.7 | 74.4 | 60.0 | 99.3% | 760 |
| [Muse-Glimmer-30B](https://huggingface.co/primitive-ai/Muse-Glimmer-30B-mixed-NVFP4) | 30 B MoE | 20.4 GiB | 86.6 | 88.8 | 78.6 | 54.5 | 99.7% | 800 |
| [Qwen3.8-27B](https://huggingface.co/primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8) | 27 B dense | 20.7 GiB | 88.8 | 90.4 | 85.5 | 54.5 | 99.7% | 651 |
| [Granite-4.2-30B](https://huggingface.co/primitive-ai/granite-4.2-30b-mixed-NVFP4-FP8) | 30 B dense | 18.1 GB | 85.5 | 86.2 | 85.8 | 60.8 | 98.5% | 1502 |
| [K2-Horizon-MoVA-36B-A4B NVFP4](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) | 37 B MoE+MoVA, 4 B active | 36.7 GB | 84.1 | 86.5 | 71.8 | 60.4 | 95.6% | 1234 |
| **K2-Horizon-MoVA-36B-A4B mixed** (this repo) | 37 B MoE+MoVA, 4 B active | **44.5 GB** | **84.9** | 87.3 | 73.5 | 58.9 | 96.3% | 1118 |
| [Laguna-S-2.1](https://huggingface.co/primitive-ai/Laguna-S-2.1-mixed-NVFP4-MXFP8) | 110 B MoE | 64.0 GiB | 84.3 | 87.1 | 64.6 | 81.0 | 97.3% | 995 |
| [Qwen3.8-Flash-Next](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8) | 180 B MoE, 6 B active | 183.7 GB | 90.3 | 92.2 | 84.8 | 56.7 | 99.5% | 686 |

`overall` pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. **Read it with `finished`**: `overall` scores an answer that overran the token budget as
wrong, but it cannot tell you whether the model needed the room or failed to stop — `finished` and
`out/answer` separate those, and this model's 96.3% is among the lowest `finished` in the table
because it reasons in long chains. A gap under **1.0** in `overall` is a tie. Sizes are as each card
reports them, which mixes GB and GiB.

---

## What's quantized to what

The model is 37.4 B parameters in five very unequal pieces:

| | params | share |
|---|---|---|
| routed MoE experts, 45 layers × 100 × 3 × 2560×768 | 26.54 B | 71.0% |
| MoVA value-experts, 45 layers × 64 × 2560×1024 | 7.55 B | 20.2% |
| attention `qk_proj` / `o_proj` / `gate_proj` | 1.61 B | 4.3% |
| `embed_tokens` + `lm_head`, untied, vocab 250624 | 1.28 B | 3.4% |
| shared experts + the three dense MLP layers | 0.41 B | 1.1% |

Which of those pieces got which format:

| tensors | count | format |
|---|---|---|
| routed experts on 30 of the 45 sparse layers | 9,000 | FP8 E4M3, block scales `[128,128]` |
| routed experts on the other 15 sparse layers | 4,500 | NVFP4, group 16 |
| MoVA `v_experts` and `v_router`; `qk_proj`/`o_proj`/`gate_proj`; `mlp.gate`; shared experts; dense layers 0–2; `lm_head`; embeddings; norms | 3,408 | BF16 |

`compressed-tensors`, top-level format `mixed-precision`, two config groups. The NVFP4 group is W4A4
and carries observed activation scales plus a tensor-level `weight_global_scale`; the FP8 group is
weight-only with block scales.

**The second-largest block cannot be quantized at all.** vLLM's MoVA attention does this on every
forward pass:

```python
w1 = torch.stack([expert.weight for expert in self.v_experts], dim=0).contiguous()
v  = fused_mova_impl(..., w1=w1, ...)
```

It stacks the raw `.weight` of all 64 value-experts and hands `fused_mova_impl` a dense tensor. A
quantized `Linear` carries `weight_packed` plus scales and no `weight`, so a quantized v_expert
either fails at load or feeds packed bytes to a kernel that reads them as bfloat16. IFM's own FP8
ignores all 2,880 of them for the same reason. That is 20% of the model pinned to BF16 in any build,
which is why lower precision buys less here than the parameter count suggests: those 7.55 B
parameters are 15.1 GB of BF16 that every forward pass moves regardless. Per the throughput section,
that is why decode speed barely moves.

Two engine constraints shape the rest. MoVA fuses `q` and `k` into a single `qk_proj`, so those two
would have to share one scheme. And a layer's experts are quantized as a unit: vLLM's fused MoE
requires one scheme across an expert's `gate`/`up`/`down`, so the mixture here is per layer, never
per projection.

---

## Attribution

- Base: `IFM/K2-Horizon-MoVA-36B-A4B`, Apache-2.0, BF16 weights unmodified.
- Built with `llm-compressor` 0.13.1 and `compressed-tensors` 0.18.1.
- Calibration: 496 conversations at seqlen 2048, used to observe activation scales.
- The checkpoint carries IFM's `modeling_k2_horizon.py` and `configuration_k2_horizon.py`
  unchanged, so `--trust-remote-code` works for tooling that wants the reference implementation.
- Audited before publication: every fused NVFP4 group shares one `weight_global_scale`, every FP8
  block-scale tensor has the shape its group declares, no activation scale is zero or NaN, and
  `config.json` agrees with the tensors actually present.

---

<p align="center">
  <br>
  <img src="assets/primitive-logo.png" alt="Primitive" width="34"><br>
  <sub>
    <a href="https://primitive.com"><b>primitive</b></a> ·
    <a href="https://huggingface.co/primitive-ai">more models</a> ·
    inference economics for production LLM systems
  </sub>
</p>