Instructions to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Ollama
How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with Ollama:
```
ollama run hf.co/NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with Docker Model Runner:
```
docker model run hf.co/NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M
```

Lemonade

How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.K2-Horizon-MoVA-36B-A4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

K2-Horizon-MoVA-36B-A4B — GGUF

GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B, produced with the MBZUAI-IFM fork of llama.cpp (model/K2Horizon branch) that adds native k2-horizon architecture support.

Architecture note: Upstream llama.cpp does not yet support the k2_horizon / K2HorizonForCausalLM architecture (PR in progress as of September 2026). You must use the MBZUAI-IFM fork linked above — or upstream once the PR merges — to load these GGUFs.

Model Overview

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family released by the Institute of Foundation Models (IFM) at MBZUAI on 3 September 2026. It is a Mixture-of-Experts (MoE) model augmented with a novel Mixture-of-Values (MoVA) attention mechanism, storing ~36 B parameters while activating only ~4 B per token.

Property	Value
Architecture	`k2_horizon` (`K2HorizonForCausalLM`)
Total parameters	~36 B
Active parameters / token	~4 B
Layers	48 (3 leading dense + 45 MoE)
Hidden size	2 560
MoE routing	100 routed experts / 8 active + 1 shared expert per MoE layer
MoVA attention	64 value experts / 4 active per attention layer
Context window	512 K (524 288 tokens, native)
Embedding tying	Untied (separate input / output embeddings)
Precision (source)	BF16
License	Apache 2.0

What Is MoVA?

Mixture-of-Values (MoVA) applies the same sparse routing philosophy as MoE, but inside the attention mechanism: the value projections are replaced by a learned pool of value experts, and only a small subset is activated per token. This lets the model store a much richer set of representational "styles" in the value space without paying for all of them at inference time, complementing the feed-forward MoE sparsity.

Quantization Details

File	Quant type	Approx. size	Notes
`K2-Horizon-MoVA-36B-A4B-BF16.gguf`	BF16	~70 GiB	Original precision; quantization source
`K2-Horizon-MoVA-36B-A4B-Q8_0.gguf`	Q8_0	~35 GiB	Near-lossless
`K2-Horizon-MoVA-36B-A4B-Q6_K.gguf`	Q6_K	~27 GiB	Very high quality
`K2-Horizon-MoVA-36B-A4B-Q5_K_M.gguf`	Q5_K_M	~23 GiB	High quality, recommended
`K2-Horizon-MoVA-36B-A4B-Q5_K_S.gguf`	Q5_K_S	~22 GiB
`K2-Horizon-MoVA-36B-A4B-Q5_1.gguf`	Q5_1	~22 GiB
`K2-Horizon-MoVA-36B-A4B-Q5_0.gguf`	Q5_0	~21 GiB
`K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf`	Q4_K_M	~20 GiB	Good balance of size and quality
`K2-Horizon-MoVA-36B-A4B-Q4_K_S.gguf`	Q4_K_S	~19 GiB
`K2-Horizon-MoVA-36B-A4B-Q4_1.gguf`	Q4_1	~20 GiB
`K2-Horizon-MoVA-36B-A4B-Q4_0.gguf`	Q4_0	~19 GiB
`K2-Horizon-MoVA-36B-A4B-Q3_K_L.gguf`	Q3_K_L	~16 GiB
`K2-Horizon-MoVA-36B-A4B-Q3_K_M.gguf`	Q3_K_M	~15 GiB
`K2-Horizon-MoVA-36B-A4B-Q3_K_S.gguf`	Q3_K_S	~14 GiB
`K2-Horizon-MoVA-36B-A4B-Q2_K.gguf`	Q2_K	~10 GiB	Smallest; used as imatrix source
`K2-Horizon-MoVA-36B-A4B-IQ4_NL.gguf`	IQ4_NL	~20 GiB	imatrix-calibrated
`K2-Horizon-MoVA-36B-A4B-IQ4_XS.gguf`	IQ4_XS	~19 GiB	imatrix-calibrated
`K2-Horizon-MoVA-36B-A4B-IQ3_M.gguf`	IQ3_M	~16 GiB	imatrix-calibrated
`K2-Horizon-MoVA-36B-A4B-IQ3_XS.gguf`	IQ3_XS	~14 GiB	imatrix-calibrated
`K2-Horizon-MoVA-36B-A4B-IQ3_XXS.gguf`	IQ3_XXS	~13 GiB	imatrix-calibrated
`K2-Horizon-MoVA-36B-A4B-IQ2_M.gguf`	IQ2_M	~11 GiB	imatrix-calibrated

IQ quants (those beginning with IQ) were generated using an importance matrix computed on the Q2_K quant with 8 × 512-token chunks of WikiText-2 calibration data (Salesforce/wikitext).

All quants were produced from the BF16 GGUF published by IFM at IFM/K2-Horizon-MoVA-36B-A4B-GGUF. No safetensors conversion was performed; the IFM BF16 GGUF already includes tokenizer metadata and a llama.cpp-compatible chat template.

Hardware Requirements (Rough Guide)

These figures are weight-only estimates and exclude KV cache, runtime overhead, and context length. They are not minimum VRAM/RAM requirements.

Quant	Est. weight size	Practical minimum RAM/VRAM
BF16	~70 GiB	2 × 48 GB GPU (A6000 / H100 80 GB)
Q8_0	~35 GiB	2 × 24 GB GPU
Q5_K_M	~23 GiB	1 × 24 GB GPU or 2 × 16 GB
Q4_K_M	~20 GiB	1 × 24 GB GPU
Q3_K_M	~15 GiB	1 × 16 GB GPU
Q2_K / IQ2_M	~10–11 GiB	1 × 12 GB GPU

Memory caveat: "4B active" means 4B parameters are computed per token, not that the model fits in 4B-worth of memory. All 36B weights must be resident in RAM/VRAM.

Running with llama.cpp

Prerequisite: build from the MBZUAI-IFM fork (model/K2Horizon branch) until upstream llama.cpp merges the PR.

git clone --depth 1 --branch model/K2Horizon \
    https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-k2horizon
cd llama.cpp-k2horizon
cmake -B build -DGGML_NATIVE=OFF
cmake --build build -j4

Basic inference:

./build/bin/llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf \
  -p "Explain the difference between MoE and MoVA architectures." \
  -n 1024 \
  --temp 1.0 \
  --top-p 0.95 \
  -c 8192

Serving (Original BF16 Model)

The original BF16 safetensors checkpoint can be served with vLLM or SGLang.

vLLM:

vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice

SGLang (validated on 2 × H200):

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tp 2 \
  --ep 2 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

Best Practices

Always use reasoning_effort="high" — all reported benchmark scores were obtained at high reasoning effort.
Recommended sampling: temperature=1.0, top_p=0.95, max_new_tokens ≥ 32 768.
Tool use / agent tasks: enable the k2_horizon tool-call parser when using the vLLM or SGLang API.
llama.cpp context: the model supports 512 K natively, but local inference is memory-constrained — use -c 8192 or -c 32768 unless you have sufficient RAM.

Benchmark Results

Results from the IFM model card. All scores in %. Bold = best in row. K2-Horizon-MoVA-36B-A4B uses high reasoning effort; other open models are in their respective reasoning modes.

Benchmark	K2-Horizon-MoVA-36B-A4B (4B active)	Nemotron 3 Ultra (55B active)	Nemotron 3 Super (12B active)	G9v3-39A5B (5B active)	Qwen3.6-35B-A3B (3B active)	Muse Glimmer-30B (Dense)	Gemma 4 31B-it (Dense)
Agents
tau3-Banking (tool use)	26.8	14.2	10.3	22.1	9.3	23.5	14.8
Coding
Terminal-Bench 2.1 (agentic)	58.6	53.9	38.6	32.6	44.9	51.7	43.4
SciCode (scientific coding)	38.9	39.9	36.0	34.0	35.8	43.6	43.4
Scientific Reasoning
Humanity's Last Exam	25.2	28.4	20.8	17.5	22.2	22.0	23.6
GPQA Diamond	80.8	86.7	80.0	80.5	84.1	83.5	85.7
CritPt (frontier physics)	2.1	3.1	3.1	0.3	0.3	2.6	1.4
General
AA-LCR (long-context)	66.3	71.0	60.3	62.0	66.7	80.0	68.3
AA-Omniscience Accuracy	18.8	22.6	24.3	14.9	18.8	27.0	20.0
AA-Omniscience Non-Hallucination	69.2	70.3	13.0	87.0	49.5	18.1	15.0

Source & Links

Original model: IFM/K2-Horizon-MoVA-36B-A4B
Official BF16 GGUF: IFM/K2-Horizon-MoVA-36B-A4B-GGUF
llama.cpp fork (K2-Horizon support): MBZUAI-IFM/llama.cpp @ model/K2Horizon
IFM blog post: https://ifm.ai/blog/k2/
vLLM recipe: https://recipes.vllm.ai/IFM/K2-Horizon-MoVA-36B-A4B
SGLang cookbook: https://docs.sglang.io/cookbook/autoregressive/IFM/K2-Horizon

Quantization Pipeline

These GGUFs were generated with the following pipeline, mirroring the approach used for other large MoE checkpoints in this workspace:

Fork checkout & build — the MBZUAI-IFM model/K2Horizon branch is cloned to a dedicated directory and compiled with CMake (GGML_NATIVE=OFF for portability).
Source download — the IFM BF16 GGUF (~70 GiB) is downloaded from IFM/K2-Horizon-MoVA-36B-A4B-GGUF instead of converting from safetensors (a direct BF16 or Q8_0 conversion would exceed the ~109 GiB free on /tmp).
Imatrix — computed on the Q2_K quant (~10 GiB, fits in 15 GB RAM) with 8 calibration chunks of WikiText-2 to produce an importance matrix for the IQ quants.
Quantization — all standard and IQ quants are produced from the BF16 GGUF source; each file is deleted immediately after a verified upload to keep disk usage under BF16 (~70 GiB) + one quant at any given moment.
Resume support — a status.json file tracks which quants have been generated and uploaded so interrupted runs resume from where they left off.

License

The original model and these GGUF quantizations are released under the Apache 2.0 License. See LICENSE on the upstream repository for full terms.

Citation

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}

Downloads last month: 25,114

GGUF

Model size

37B params

Architecture

k2-horizon

Hardware compatibility

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for NANI-Nithin/K2-Horizon-MoVA-36B-A4B-GGUF

Base model

IFM/K2-Horizon-MoVA-36B-A4B

Quantized

(18)

this model