---
license: apache-2.0
base_model: IFM/K2-Horizon-MoVA-36B-A4B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
- vllm
- compressed-tensors
- nvfp4
- quantized
- moe
- mova
- k2-horizon
- reasoning
- tool-calling
- blackwell
thumbnail: https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4/resolve/main/assets/banner.png
---
Half the size of BF16. Level on knowledge and abstention, 3.1 behind on call accuracy.
4-bit quantization of IFM/K2-Horizon-MoVA-36B-A4B to
36.74 GB, on one 96 GB card.
The 13,500 routed expert projections are NVFP4; everything else stays BF16.
> **This is the small, portable one.** There is also a
> [mixed NVFP4/FP8 build](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8)
> at 44.48 GB that is 1.7 higher on call accuracy but needs Blackwell. **This build is the one that
> also runs on Ampere**, verified generating on an A100 40 GB. Take it for pre-Blackwell hardware,
> for the smallest footprint, or for throughput under load; take the mixed one if call accuracy
> matters more than any of those.
---
## Why this quant
* 🗜️ **2.0× smaller than BF16, 24% smaller than IFM's own FP8.** 36.74 GB against 74.89 GB and
48.36 GB.
* 🎯 **Knowledge is level with both references.** 86.5 against the official FP8's 86.8 and BF16's
85.9 over 1,170 items.
* 🧭 **Abstention is level too, and that is the useful half of the tool-calling result.** 60.4
against the official FP8's 60.4, the same number. What 4 bits cost is argument accuracy on calls
the model decides to make: 71.8 against 74.9, a real gap at t = −4.2, and two of the five
benchmarks carry 90% of it.
* ⚡ **1.25× the official FP8's throughput at concurrency 32, and first token in a third the time.**
301.5 against 240.6 tok/s, median TTFT 1.54 s against 4.46 s. Single-stream is a wash, and the
throughput section explains why that is architectural.
* 🖥️ **One card, no tensor parallelism.** Both official recipes use TP=2; this fits TP=1 with
275,952 tokens of KV at 32K context.
* 🧩 **3,408 modules stay at BF16.** IFM's own FP8 leaves the same set alone. The reason is
architectural and spelled out below.
* 🔧 **Serves on stock vLLM `main`, no patches and no custom kernels.** Plain `compressed-tensors`.
When this repo first went up, on 4 September 2026, no NVFP4 build of this model served on vLLM.
---
## Serve it
K2-Horizon support merged into vLLM `main` on 3 September 2026
([#55063](https://github.com/vllm-project/vllm/pull/55063)) and is **not in a release yet** —
0.28.0 predates it. Use a `main`/nightly build:
```bash
hf download primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4 \
--local-dir ./K2-Horizon-MoVA-36B-A4B-NVFP4
docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c \
--model /models/K2-Horizon-MoVA-36B-A4B-NVFP4 \
--trust-remote-code --max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--reasoning-parser k2_horizon
```
That image tag is the nightly every number on this page was measured on (vllm
`0.28.1rc1.dev388`). On one RTX PRO 6000 Blackwell it leaves 34.87 GiB resident for weights and
50.53 GiB for KV — **275,952 tokens at 32K context, 8.42× concurrency** — and the engine is up in
97 s, of which 3 s is `torch.compile`.
**Reasoning effort is a server-level choice on this architecture, not a per-request one.** The
template emits a different tag pair per effort and `K2HorizonReasoningParser` picks one pair once
at startup, defaulting to `high`:
- `high` (the default) — `` … ``
- `medium` — `` … ``
- `low` — `` … ``
Send a per-request `reasoning_effort: low` against a server parsing `high` and the tags never
match, so the whole reasoning trace stays in `content` instead of being separated out. Set the
effort on the server. Split reasoning is returned in `message.reasoning`.
---
## Measured
**1,370 items across fourteen public benchmarks.** A 1,170-item knowledge suite — MMLU-Pro,
ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500 — and a
200-item tool-calling suite — BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, When2Call — under one fixed
protocol for every row: [PQE-1.2](https://huggingface.co/datasets/primitive-ai/pqe),
`temperature 0.6` / `top_p 0.95` / `top_k 20`, thinking forced on, a 16,384-token budget,
concurrency 32, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.
Auto-scored, no LLM judge. A knowledge item is scored on the last `ANSWER:` in the reply; a
tool-calling item counts only if every expected call is present with acceptable arguments and no
call is invented. Sizes are the safetensors weights, and `±` is the sample standard deviation over
the runs counted in the `runs k/a` column.
| build | size | knowledge | call | abstain | tool calling | runs k/a | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| [IFM BF16](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B) | 74.89 GB | 85.9 ±0.7 | 74.6 ±1.3 | 60.8 ±2.9 | 71.8 ±1.0 | 2/3 | 95.8% | 1206 |
| [IFM FP8](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-FP8) | 48.36 GB | 86.8 ±0.5 | 74.9 ±1.5 | 60.4 ±2.9 | 72.0 ±0.8 | 2/6 | 96.1% | 1176 |
| [our mixed NVFP4/FP8](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8) | 44.48 GB | 87.3 ±1.0 | 73.5 ±1.7 | 58.9 ±3.1 | 70.6 ±1.4 | 2/9 | 96.3% | 1118 |
| **this repo** | **36.74 GB** | 86.5 ±0.5 | 71.8 ±1.0 | 60.4 ±3.7 | 69.5 ±1.4 | 2/6 | 95.6% | 1234 |
`call` is accuracy on the 160 tool-calling items that require a call; `abstain` is the 40 whose
correct action is to call **nothing**; `tool calling` is the two pooled as 200. We report the halves
separately because pooling hides which one moved. `runs k/a` is the number of knowledge and agentic
runs behind each row. `finished` is the share of the 1,170 knowledge items that answered inside the
token budget; `out/answer` is mean output tokens.
**Knowledge is one group.** No pair separates: against the official FP8 this build is t = −0.6, the
mixed build t = +0.6, BF16 t = −1.5, all at n=2 per side. Don't rank on this column.
**Call accuracy is what 4 bits cost this model.** 71.8 against the official FP8's 74.9 — 3.1 points,
t = −4.2 over six runs each. Abstention is untouched at 60.4 against 60.4, the same number to two
decimals, so the model has not become trigger-happy or timid. It picks arguments wrong slightly
more often.
Nor is it a formatting failure. Unparsed replies run at 9 per 1,200 here (0.75%) against 13 per
1,200 for the official FP8 (1.08%) and 6 per 600 for BF16 (1.00%) — the lowest rate of any build
measured. Thinking fired on 99–100% of items in every row.
Per-benchmark detail, both suites
| benchmark | items | IFM BF16 | IFM FP8 | our mixed | **this repo** |
|---|---|---|---|---|---|
| **tool calling** | | | | | |
| bfcl_v4 | 30 | 83.3 | 81.7 | 81.9 | 80.6 |
| xlam / APIGen | 45 | 44.4 | 44.8 | 40.2 | 37.8 |
| toolace | 45 | 76.3 | 78.1 | 77.8 | 75.2 |
| glaive v2 | 40 | 100.0 | 100.0 | 99.7 | 99.6 |
| when2call | 40 | 60.8 | 60.4 | 58.9 | 60.4 |
| **knowledge** | | | | | |
| mmlu_pro | 200 | 85.5 | 86.5 | 87.5 | 84.5 |
| math_500 | 100 | 89.5 | 88.5 | 89.5 | 89.0 |
| gsm8k | 100 | 97.5 | 97.0 | 97.5 | 97.5 |
| arc_challenge | 150 | 98.7 | 97.0 | 98.0 | 98.0 |
| hellaswag | 150 | 68.0 | 73.3 | 74.7 | 70.0 |
| winogrande | 150 | 80.7 | 81.3 | 78.7 | 84.3 |
| commonsenseqa | 120 | 75.4 | 79.2 | 80.0 | 77.5 |
| boolq | 100 | 90.0 | 91.0 | 92.0 | 91.0 |
| openbookqa | 100 | 96.0 | 94.0 | 94.5 | 94.5 |
Pooled over every run behind the row, so a cell rests on `runs × items`. `when2call` supplies the
40 abstention items; the other four tool-calling benchmarks supply the 160 call items.
**Two benchmarks carry the tool-calling gap.** xlam/APIGen falls 44.8 → 37.8 against the official
FP8, 7.0 points on 45 items, and toolace 78.1 → 75.2, 2.9 points on 45. Weighted by item count
those two are 64% and 26% of the 2.5-point pooled gap; bfcl contributes 0.17 of a point, glaive
0.08, and abstention does not move at all. Both carry the longest argument lists in the suite,
which fits 4 bits costing argument accuracy rather than call structure.
> **The noise floor, and why the run counts are in the table.** Every build compared here has at
> least six agentic runs, the one exception being the BF16 reference at three. It needs to be six.
> The official FP8's tool-calling figure read 72.7 ±0.3 after three runs and 72.0 ±0.8 after six;
> this build's read 69.0 ±1.5 after three and 69.5 ±1.4 after six. On the 1,170-item knowledge
> suite, repeat runs of a single checkpoint span up to 1.9 points, so **treat a knowledge gap of a
> point or so as unresolved.** The widest gap in that column is 1.4, between the mixed build and
> BF16, and even that is only t = +1.6. We publish ties as ties.
Two footnotes on the table. The BF16 row was served at 20,480 context against the quants' 32,768,
because 74.89 GB of weights leaves only 81,664 tokens of KV on a 96 GB card; the generation budget
was the same 16,384 tokens for every row. And `out/answer` is a mean over a long tail: median output
is 208 tokens against a mean of 1,234, a ratio of 5.9×. That shape usually means a model that
reaches its answer and cannot stop. Here it does not — the official FP8 shows 1,176 against a
median of 211 on the same items, a ratio of 5.6×, with the same ~4% of items hitting the budget.
It is how K2-Horizon reasons, and quantization did not cause it.
---
## Throughput
Cache-free, 8,192 tokens in and 512 out, one RTX PRO 6000 Blackwell, same nightly and same serve
flags as the accuracy rows. Prefix caching is disabled and every measurement runs on its own seed,
so no row is reading a cache the previous one filled. These come from one sitting and drift a few
percent between sittings, so read them as a bracket. BF16 is absent because at 74.89 GB it only
serves 20,480 context on this card, which would not be the same measurement.
| build | tok/s @ conc 1 | TPOT @ conc 1 | tok/s @ conc 32 | TPOT @ conc 32 | median TTFT @ conc 32 |
|---|---|---|---|---|---|
| [IFM FP8](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-FP8) | 29.74 | 32.83 ms | 240.6 | 89.2 ms | 4.46 s |
| [our mixed NVFP4/FP8](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8) | 29.99 | 32.56 ms | 248.4 | 89.5 ms | 3.59 s |
| **this repo** | **30.45** | **32.10 ms** | **301.5** | 93.1 ms | **1.54 s** |
**Quantizing this model buys throughput under load, not single-stream latency.** Those three builds
span 11.6 GB and land within 0.71 tok/s of each other at concurrency 1. At concurrency 32 their
per-token decode cost is nearly flat too — 89.2, 89.5 and 93.1 ms, a 4.4% spread — so the 25%
aggregate gain is not a decode-rate effect. It shows up in prefill and in scheduling under load,
where median time to first token falls from 4.46 s to 1.54 s. Decode stays pinned because vLLM
restacks the 15.1 GB of BF16 MoVA value-experts on every forward pass whatever the routed experts
are quantized to, and that fixed cost dominates a single stream.
---
## Choosing between our two builds
| | **this repo** | [mixed NVFP4/FP8](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8) |
|---|---|---|
| size | **36.74 GB** | 44.48 GB |
| knowledge | 86.5 ±0.5 | 87.3 ±1.0 |
| call | 71.8 ±1.0 | **73.5 ±1.7** |
| abstain | 60.4 ±3.7 | 58.9 ±3.1 |
| tool calling, pooled | 69.5 ±1.4 | 70.6 ±1.4 |
| tok/s @ conc 32 | **301.5** | 248.4 |
| median TTFT @ conc 32 | **1.54 s** | 3.59 s |
| Blackwell | yes | yes |
| **Ampere (A100)** | **yes** | **no** |
One axis separates our two builds: call accuracy, where the mixed build leads by 1.7 at t = +2.4.
Its pooled tool-calling lead of 1.1 does not (t = +1.5), and neither does its knowledge (t = +0.9).
So its extra 7.74 GB, +21%, buys better arguments on calls. This one is the faster and more portable
artifact. **If tool calling dominates your workload and you have 48 GB, IFM's own FP8 is still the
most accurate option** — it leads this build by 3.1 on call accuracy.
**On Ampere.** NVFP4 has no native kernel on sm_80; vLLM dequantizes through Marlin. Verified on an
A100 40 GB: loads, serves and generates correctly with the reasoning parser working. That was a
load-and-generate probe at 2,048 context, not a scored run — a 40 GB card leaves too little room for
the 16,384-token protocol — so every number on this page comes from Blackwell. We have not tested
the mixed build on Ampere and do not expect its FP8 block scales to load there.
---
## Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different `temperature`, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above — the
same 1,370 items, auto-scored, no LLM judge.
| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| [Laguna-XS-2.1](https://huggingface.co/primitive-ai/Laguna-XS-2.1-mixed-NVFP4-MXFP8) | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
| [Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8) | 30 B MoE+Mamba | 19.2 GiB | 87.1 | 87.9 | 85.4 | 70.5 | 97.9% | 1429 |
| [Ornith-1.5-35B-A3B](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8) | 35 B MoE | 22.6 GiB | 88.7 | 91.7 | 74.4 | 60.0 | 99.3% | 760 |
| [Muse-Glimmer-30B](https://huggingface.co/primitive-ai/Muse-Glimmer-30B-mixed-NVFP4) | 30 B MoE | 20.4 GiB | 86.6 | 88.8 | 78.6 | 54.5 | 99.7% | 800 |
| [Qwen3.8-27B](https://huggingface.co/primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8) | 27 B dense | 20.7 GiB | 88.8 | 90.4 | 85.5 | 54.5 | 99.7% | 651 |
| [Granite-4.2-30B](https://huggingface.co/primitive-ai/granite-4.2-30b-mixed-NVFP4-FP8) | 30 B dense | 18.1 GB | 85.5 | 86.2 | 85.8 | 60.8 | 98.5% | 1502 |
| **K2-Horizon-MoVA-36B-A4B** (this repo) | 37 B MoE+MoVA, 4 B active | **36.7 GB** | **84.1** | 86.5 | 71.8 | 60.4 | 95.6% | 1234 |
| [K2-Horizon-MoVA-36B-A4B mixed](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8) | 37 B MoE+MoVA, 4 B active | 44.5 GB | 84.9 | 87.3 | 73.5 | 58.9 | 96.3% | 1118 |
| [Laguna-S-2.1](https://huggingface.co/primitive-ai/Laguna-S-2.1-mixed-NVFP4-MXFP8) | 110 B MoE | 64.0 GiB | 84.3 | 87.1 | 64.6 | 81.0 | 97.3% | 995 |
| [Qwen3.8-Flash-Next](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8) | 180 B MoE, 6 B active | 183.7 GB | 90.3 | 92.2 | 84.8 | 56.7 | 99.5% | 686 |
`overall` pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. **Read it with `finished`**: `overall` scores an answer that overran the token budget as
wrong, but it cannot tell you whether the model needed the room or failed to stop — `finished` and
`out/answer` separate those, and this model's 95.6% is the lowest `finished` in the table because it
reasons in long chains. A gap under **1.0** in `overall` is a tie. Sizes are as each card reports
them, which mixes GB and GiB.
---
## What's quantized to what
The model is 37.4 B parameters in five very unequal pieces:
| | params | share |
|---|---|---|
| routed MoE experts, 45 layers × 100 × 3 × 2560×768 | 26.54 B | 71.0% |
| MoVA value-experts, 45 layers × 64 × 2560×1024 | 7.55 B | 20.2% |
| attention `qk_proj` / `o_proj` / `gate_proj` | 1.61 B | 4.3% |
| `embed_tokens` + `lm_head`, untied, vocab 250624 | 1.28 B | 3.4% |
| shared experts + the three dense MLP layers | 0.41 B | 1.1% |
Which of those pieces got which format:
| tensors | count | format |
|---|---|---|
| `mlp.experts.{0..99}.{gate,up,down}_proj` on all 45 sparse layers | 13,500 | NVFP4, group 16 |
| MoVA `v_experts` and `v_router`; `qk_proj`/`o_proj`/`gate_proj`; `mlp.gate`; shared experts; dense layers 0–2; `lm_head`; embeddings; norms | 3,408 | BF16 |
`compressed-tensors`, one config group. NVFP4 is W4A4, so the checkpoint carries observed activation
scales alongside the weights as well as a tensor-level `weight_global_scale`.
**The second-largest block cannot be quantized at all.** vLLM's MoVA attention does this on every
forward pass:
```python
w1 = torch.stack([expert.weight for expert in self.v_experts], dim=0).contiguous()
v = fused_mova_impl(..., w1=w1, ...)
```
It stacks the raw `.weight` of all 64 value-experts and hands `fused_mova_impl` a dense tensor. A
quantized `Linear` carries `weight_packed` plus scales and no `weight`, so a quantized v_expert
either fails at load or feeds packed bytes to a kernel that reads them as bfloat16. IFM's own FP8
ignores all 2,880 of them for the same reason. That is 20% of the model pinned to BF16 in any build,
which is why 4 bits buys less here than the parameter count suggests: those 7.55 B parameters are
15.1 GB of BF16 that every forward pass moves regardless. Per the throughput section, that is why
decode speed barely moves.
Attention is left alone too: MoVA fuses `q` and `k` into a single `qk_proj`, so those two would have
to share one scheme and, under NVFP4, one `weight_global_scale`.
---
## Attribution
- Base: `IFM/K2-Horizon-MoVA-36B-A4B`, Apache-2.0, BF16 weights unmodified.
- Built with `llm-compressor` 0.13.1 and `compressed-tensors` 0.18.1.
- Calibration: 496 conversations at seqlen 2048, used to observe activation scales.
- The checkpoint carries IFM's `modeling_k2_horizon.py` and `configuration_k2_horizon.py`
unchanged, so `--trust-remote-code` works for tooling that wants the reference implementation.
- Audited before publication: every fused NVFP4 group shares one `weight_global_scale`, no
activation scale is zero or NaN, and `config.json` agrees with the tensors actually present.
---
primitive ·
more models ·
inference economics for production LLM systems