---
license: apache-2.0
base_model:
- IFM/K2-Horizon-MoVA-36B-A4B-GGUF
- IFM/K2-Horizon-MoVA-36B-A4B
base_model_relation: quantized
tags:
- gguf
- moe
- k2-horizon
- mova
- apex-quant
pipeline_tag: text-generation
---

# K2-Horizon-MoVA-36B-A4B — Apex Quant GGUF Models

> # 🛑 READ BEFORE YOU RUN THIS MODEL
>
> **The "Use this model" quick-start buttons above (llama serve -hf ..., winget, Docker, Ollama, etc.) point to *vanilla/upstream* llama.cpp. They will NOT work correctly with these files.**
>
> Upstream llama.cpp has no support for the `k2-horizon` architecture yet, and even the community fork that adds support has two known bugs. If you use the auto-generated buttons above, you will get one of:
> - a load failure (`Failed to process regex ... regex_error`) on Windows, or
> - garbage/repeated output (`> > > ...`) on any OS.
>
> **You must build the patched fork described in [Building the Runtime](#️-building-the-runtime-patched-llamacpp) below.** There is currently no other correct way to run this GGUF with llama.cpp.

This repository contains 3 quantized GGUF profiles of the **K2-Horizon-MoVA-36B-A4B** model, produced using **Apex-Quant** technology.

## 📦 Model Profile Summary

| Profile | Size | BPW | Use Case |
|---------|------|-----|----------|
| **i-quality** | 23.8 GB | 5.47 | Highest quality, production environments |
| **i-balanced** | 26.2 GB | 6.02 | Maximum expert precision (densest expert quant) |
| **i-compact** | 17.6 GB | 4.04 | Compact deployment, low VRAM/RAM |

> **BPW** = Bits Per Weight. Higher value = better quality.
>
> Note: unlike most model families, here **i-balanced is larger than i-quality** — it keeps all routed experts at Q5_K or higher, while i-quality compresses mid-layer experts to IQ4_XS. Choose based on your memory budget vs. precision preference.

## 📁 File Structure

```
models/
├── K2-Horizon-36B-i-quality.gguf    # 23.8 GB — highest quality
├── K2-Horizon-36B-i-balanced.gguf   # 26.2 GB — max expert precision
└── K2-Horizon-36B-i-compact.gguf    # 17.6 GB — compact
```

## 🔗 Source Model

These models were created based on the **K2-Horizon-MoVA-36B-A4B** model.

- **Model Page:** https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
- **Architecture:** K2Horizon MoE with MoVA (Mixture-of-Value Attention)
- **Total Parameters:** ~36B (~4B active per token)
- **Context Length:** 524,288 tokens (training context; usable range depends on KV-cache RAM)
- **Reference Implementation:** vLLM (`K2HorizonForCausalLM`)
- **Official llama.cpp fork:** https://github.com/MBZUAI-IFM/llama.cpp

## 🛠️ Technology

These quantized models were produced using **Apex-Quant** technology.

- **Apex-Quant:** MoE-aware mixed-precision quantization (layer-band importance: edge / near / mid)
- **Infrastructure:** llama.cpp (`llama-quantize`, `--tensor-type-file`)
- **Config Generator:** `apex-quant/scripts/generate_config.sh --profile <profile> --layers 48`

## ⚙️ Building the Runtime (patched llama.cpp)

### Why a patched fork is required

The K2-Horizon support fork ([MBZUAI-IFM/llama.cpp](https://github.com/MBZUAI-IFM/llama.cpp)) provides the base K2-Horizon support. On top of the pinned commit below, **two fixes** are required for reliable operation (both verified empirically in our test setup):

| # | Issue | Observed Symptom | Fix |
|---|-------|------------------|-----|
| 1 | Tokenizer pre-regex contains `\u200C`/`\u200D` escapes unsupported by MSVC `std::regex` | Model fails to load on **Windows**: `Failed to process regex ... regex_error(error_escape)` | Remove the `\u200C\|\u200D` alternation from the K2_HORIZON pre-tokenization regex |
| 2 | Decoder residual/norm wiring diverged from the vLLM reference behavior | Model loads fine but generates **garbage/repeated tokens** (`> > > ...`), possible crashes during graph reservation | Rewrite residual flow in `k2-horizon.cpp` forward pass following vLLM semantics (see note below) |

Both fixes are provided as ready-to-apply patches below.

### Step-by-step

```bash
# 1. Clone the official K2-Horizon fork and pin the exact commit these models were validated against
git clone https://github.com/MBZUAI-IFM/llama.cpp llama-cpp-k2horizon
cd llama-cpp-k2horizon
git checkout 35999d101cf2233fc54f09c3c8d599da7303ce02

# 2. Apply the two patches (from this repo's patches/ directory)
git apply /path/to/patches/0001-fix-k2-horizon-msvc-regex.patch
git apply /path/to/patches/0002-fix-k2-horizon-parallel-norm.patch
```

#### Patch 1 — MSVC tokenizer regex fix (`src/llama-vocab.cpp`)

MSVC's `std::regex` rejects `\uXXXX` escape sequences, so loading any K2-Horizon model crashes on Windows before inference even starts. The zero-width joiner/non-joiner alternatives are irrelevant for BPE pre-tokenization quality and are removed:

```diff
             case LLAMA_VOCAB_PRE_TYPE_K2_HORIZON:
                 regex_exprs = {
-                    "...(?:\\p{L}|\\p{M}|\\u200C|\\u200D)+...",
+                    "...(?:\\p{L}|\\p{M})+...",
                 };
```

*(Full, exact diff: [`patches/0001-fix-k2-horizon-msvc-regex.patch`](patches/0001-fix-k2-horizon-msvc-regex.patch))*

#### Patch 2 — Parallel norm residual fix (`src/models/k2-horizon.cpp`)

This patch rewires the decoder residual path to follow the vLLM reference semantics (`K2HorizonRMSNorm.forward(x, residual)`), where the running sum of sublayer outputs is tracked across layers and folded in *before* normalization.

> **Status note:** With this patch applied, the repeated-token (`> > >`) failure mode and the graph-reservation crash disappeared and generation became coherent. The exact causal mechanism has **not** been independently verified against the upstream weights — treat this patch as a validated fix rather than a proven diagnosis.

The new residual flow tracks an accumulated `residual` tensor across layers:

```cpp
ggml_tensor * residual = nullptr;          // outside the layer loop

for (int il = 0; il < n_layer; ++il) {
    // --- parallel norm BEFORE attention ---
    if (il == 0) {
        residual = inpL;                   // layer 0: residual = embedding
        cur = group_rms_norm(inpL, attn_norm);
    } else {
        cur      = ggml_add(ctx0, inpL, residual);  // prev MLP out + accumulated sum
        residual = cur;
        cur      = group_rms_norm(cur, attn_norm);
    }
    /* ...attention... */

    // --- parallel norm BEFORE FFN ---
    cur      = ggml_add(ctx0, cur, residual);       // attn out + accumulated sum
    residual = cur;
    cur      = group_rms_norm(cur, ffn_norm);
    /* ...MoE FFN... */

    cur  = build_cvec(cur, il);
    inpL = cur;                                 // NO residual add here (deferred)
}

// final layer: fold last MLP output into the accumulated sum, then normalize
cur = ggml_add(ctx0, inpL, residual);
cur = group_rms_norm(cur, output_norm);
```

*(Full, exact diff: [`patches/0002-fix-k2-horizon-parallel-norm.patch`](patches/0002-fix-k2-horizon-parallel-norm.patch))*

### Compile

**Windows (Visual Studio 2022 + CUDA):**
```bat
cmake -B build -G "Visual Studio 17 2022" -A x64 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
cmake --build build --config Release
```

**Linux (GCC/Clang + CUDA):**
```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
cmake --build build --config Release -j$(nproc)
```

CPU-only builds work too (drop `-DLLAMA_CUDA=ON`).

### Verify your build

```bash
./build/bin/Release/llama-cli -m K2-Horizon-36B-i-compact.gguf -n 30 --temp 0.7 --top-p 0.9 -p "Merhaba dünya"
```

✅ **Working:** coherent natural-language answer (~27–45 tok/s decode with full CUDA offload).
❌ **Broken:** repeated `>` or command-line echo → one of the patches was not applied, or you are running an unpatched (vanilla/upstream) llama.cpp build.

## 📋 Technical Details

### Architecture Information (from GGUF metadata)
- **Architecture:** `k2-horizon`
- **Block Count:** 48 layers (3 leading dense blocks, 45 MoE blocks)
- **Expert Count:** 100 routed experts/layer
- **Expert Used Count:** 8 (top-8 routing)
- **Shared Experts:** 1/layer
- **Expert Gating:** softmax with weight normalization (scale 2.5)
- **Hidden Size:** 2,560
- **Feed Forward Size:** 6,144 (dense) / 768 (per expert)
- **Attention Heads:** 32 (KV heads: 8, grouped query attention)
- **Head Dimension:** 128 (full RoPE rotation)
- **RoPE Frequency Base:** 10,000,000
- **Layer Norm:** Grouped RMSNorm (2 groups), ε = 1e-6
- **MoVA:** Mixture-of-Value Attention — 64 value experts/layer, top-4 routed
- **Residual Scheme:** Parallel (pre-norm), as implemented by Patch 2 above — see the status note for verification caveats.

### Quantize Profile Details

Layer bands (48 layers): **EDGE** = L0–4 & L43–47 · **NEAR** = L5–9 & L38–42 · **MID** = L10–37

#### i-quality (Q6_K/Q5_K/IQ4_XS)
- **Routed Expert FFN:** EDGE Q6_K / NEAR Q5_K / MID IQ4_XS
- **Shared FFN:** Q8_0
- **Attention:** Q6_K
- **BPW:** 5.47
- **File Size:** 23.8 GB

#### i-balanced (Q6_K/Q5_K)
- **Routed Expert FFN:** EDGE Q6_K / NEAR Q5_K / MID Q5_K
- **Shared FFN:** Q8_0
- **Attention:** Q6_K
- **BPW:** 6.02
- **File Size:** 26.2 GB

#### i-compact (Q4_K/Q3_K)
- **Routed Expert FFN:** EDGE Q4_K / NEAR Q3_K / MID Q3_K
- **Shared FFN:** Q6_K
- **Attention:** Q4_K
- **BPW:** 4.04
- **File Size:** 17.6 GB

## 💻 Usage

> ⚠️ Both examples below assume you already built the **patched fork** from the section above. Running these commands against a stock `llama-cli`/`llama-server` build (including the one installed via the "Use this model" buttons at the top of this page) will not work.

### With llama.cpp (patched fork)

```bash
# Interactive chat
./build/bin/Release/llama-cli -m models/K2-Horizon-36B-i-quality.gguf \
    -n 256 --temp 0.7 --top-p 0.9 -p "Hello, how are you?"

# Single-shot completion
./build/bin/Release/llama-completion -m models/K2-Horizon-36B-i-compact.gguf \
    -n 128 -p "Explain mixture-of-experts models:"
```

### With Python (llama-cpp-python)

Build `llama-cpp-python` **against the same patched source tree** (see above), e.g.:

```bash
CMAKE_BUILD_ARGS="-DLLAMA_CURL=OFF" pip install llama-cpp-python \
    --global-option=build_ext \
    --global-option="--library_dir=$(pwd)/build/lib" \
    --global-option="--include_dir=$(pwd)/include"
```

```python
from llama_cpp import Llama

llm = Llama(
    model_path="models/K2-Horizon-36B-i-quality.gguf",
    n_ctx=8192,
    n_threads=8,
)

output = llm("Hello, how are you?", max_tokens=256)
print(output["choices"][0]["text"])
```

## 📊 Model Comparison

| Criterion | i-quality | i-balanced | i-compact |
|-----------|-----------|------------|-----------|
| **Quality** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ |
| **Speed** | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| **RAM/VRAM** | High | Highest | Low |
| **Size** | 23.8 GB | 26.2 GB | 17.6 GB |
| **BPW** | 5.47 | 6.02 | 4.04 |

## 📝 Notes

- All models are in **GGUF** format and were generated from the BF16 master checkpoint.
- The **K2Horizon** architecture combines Mixture-of-Experts (100×8) with **MoVA** (64×4 value experts) and grouped RMSNorm. Residual wiring follows the patched behavior described in [runtime fixes](#️-building-the-runtime-patched-llamacpp) — see the status note there before treating it as a confirmed architectural fact.
- Measured performance (full CUDA offload, 49/49 layers): prompt eval ≈ 28–139 tok/s, generation ≈ 27–45 tok/s (i-compact, consumer GPU).
- Only the 3 APEX profiles are published here; the 70 GB BF16 master stays upstream.
- **If you arrived here via the "Use this model" quick-start buttons at the top of this page:** those commands run vanilla llama.cpp and will not work with this GGUF yet. Please use the patched-fork instructions above instead.

## 📄 License

⚠️ Please review the **original model's license/terms** before commercial use. The GGUF files are repackaged quantizations of the upstream weights.

## 🙏 Acknowledgments

- **[MBZUAI-IFM](https://github.com/MBZUAI-IFM)** — K2-Horizon model and official llama.cpp fork
- **[localai-org/apex-quant](https://github.com/localai-org/apex-quant)** — Apex-Quant MoE-aware mixed-precision quantization framework
- **[ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)** — GGUF format and quantization engine

## 🔗 Related Links

- **Source GGUF:** https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
- **llama.cpp fork:** https://github.com/MBZUAI-IFM/llama.cpp
- **Apex-Quant:** https://github.com/localai-org/apex-quant
- **GGUF Format:** https://github.com/ggerganov/ggml/blob/master/docs/gguf.md