Instructions to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
./llama-cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Use Docker

docker model run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

LM Studio
Jan

vLLM

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Ollama
How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Ollama:
```
ollama run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
```
Unsloth Desktop

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Docker Model Runner:
```
docker model run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
```

Lemonade

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Run and chat with the model

lemonade run user.K2-Horizon-MoVA-36B-A4B-APEX-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

K2-Horizon-MoVA-36B-A4B — Apex Quant GGUF Models

🛑 READ BEFORE YOU RUN THIS MODEL

The "Use this model" quick-start buttons above (llama serve -hf ..., winget, Docker, Ollama, etc.) point to vanilla/upstream llama.cpp. They will NOT work correctly with these files.

Upstream llama.cpp has no support for the k2-horizon architecture yet, and even the community fork that adds support has two known bugs. If you use the auto-generated buttons above, you will get one of:

a load failure (Failed to process regex ... regex_error) on Windows, or

garbage/repeated output (> > > ...) on any OS.

You must build the patched fork described in Building the Runtime below. There is currently no other correct way to run this GGUF with llama.cpp.

This repository contains 3 quantized GGUF profiles of the K2-Horizon-MoVA-36B-A4B model, produced using Apex-Quant technology.

📦 Model Profile Summary

Profile	Size	BPW	Use Case
i-quality	23.8 GB	5.47	Highest quality, production environments
i-balanced	26.2 GB	6.02	Maximum expert precision (densest expert quant)
i-compact	17.6 GB	4.04	Compact deployment, low VRAM/RAM

BPW = Bits Per Weight. Higher value = better quality.

Note: unlike most model families, here i-balanced is larger than i-quality — it keeps all routed experts at Q5_K or higher, while i-quality compresses mid-layer experts to IQ4_XS. Choose based on your memory budget vs. precision preference.

📁 File Structure

models/
├── K2-Horizon-36B-i-quality.gguf    # 23.8 GB — highest quality
├── K2-Horizon-36B-i-balanced.gguf   # 26.2 GB — max expert precision
└── K2-Horizon-36B-i-compact.gguf    # 17.6 GB — compact

🔗 Source Model

These models were created based on the K2-Horizon-MoVA-36B-A4B model.

Model Page: https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
Architecture: K2Horizon MoE with MoVA (Mixture-of-Value Attention)
Total Parameters: ~~36B (~~4B active per token)
Context Length: 524,288 tokens (training context; usable range depends on KV-cache RAM)
Reference Implementation: vLLM (K2HorizonForCausalLM)
Official llama.cpp fork: https://github.com/MBZUAI-IFM/llama.cpp

🛠️ Technology

These quantized models were produced using Apex-Quant technology.

Apex-Quant: MoE-aware mixed-precision quantization (layer-band importance: edge / near / mid)
Infrastructure: llama.cpp (llama-quantize, --tensor-type-file)
Config Generator: apex-quant/scripts/generate_config.sh --profile <profile> --layers 48

⚙️ Building the Runtime (patched llama.cpp)

Why a patched fork is required

The K2-Horizon support fork (MBZUAI-IFM/llama.cpp) provides the base K2-Horizon support. On top of the pinned commit below, two fixes are required for reliable operation (both verified empirically in our test setup):

#	Issue	Observed Symptom	Fix
1	Tokenizer pre-regex contains `\u200C`/`\u200D` escapes unsupported by MSVC `std::regex`	Model fails to load on Windows: `Failed to process regex ... regex_error(error_escape)`	Remove the `\u200C\|\u200D` alternation from the K2_HORIZON pre-tokenization regex
2	Decoder residual/norm wiring diverged from the vLLM reference behavior	Model loads fine but generates garbage/repeated tokens (`> > > ...`), possible crashes during graph reservation	Rewrite residual flow in `k2-horizon.cpp` forward pass following vLLM semantics (see note below)

Both fixes are provided as ready-to-apply patches below.

Step-by-step

# 1. Clone the official K2-Horizon fork and pin the exact commit these models were validated against
git clone https://github.com/MBZUAI-IFM/llama.cpp llama-cpp-k2horizon
cd llama-cpp-k2horizon
git checkout 35999d101cf2233fc54f09c3c8d599da7303ce02

# 2. Apply the two patches (from this repo's patches/ directory)
git apply /path/to/patches/0001-fix-k2-horizon-msvc-regex.patch
git apply /path/to/patches/0002-fix-k2-horizon-parallel-norm.patch

Patch 1 — MSVC tokenizer regex fix (`src/llama-vocab.cpp`)

MSVC's std::regex rejects \uXXXX escape sequences, so loading any K2-Horizon model crashes on Windows before inference even starts. The zero-width joiner/non-joiner alternatives are irrelevant for BPE pre-tokenization quality and are removed:

             case LLAMA_VOCAB_PRE_TYPE_K2_HORIZON:
                 regex_exprs = {
-                    "...(?:\\p{L}|\\p{M}|\\u200C|\\u200D)+...",
+                    "...(?:\\p{L}|\\p{M})+...",
                 };

(Full, exact diff: patches/0001-fix-k2-horizon-msvc-regex.patch)

Patch 2 — Parallel norm residual fix (`src/models/k2-horizon.cpp`)

This patch rewires the decoder residual path to follow the vLLM reference semantics (K2HorizonRMSNorm.forward(x, residual)), where the running sum of sublayer outputs is tracked across layers and folded in before normalization.

Status note: With this patch applied, the repeated-token (> > >) failure mode and the graph-reservation crash disappeared and generation became coherent. The exact causal mechanism has not been independently verified against the upstream weights — treat this patch as a validated fix rather than a proven diagnosis.

The new residual flow tracks an accumulated residual tensor across layers:

ggml_tensor * residual = nullptr;          // outside the layer loop

for (int il = 0; il < n_layer; ++il) {
    // --- parallel norm BEFORE attention ---
    if (il == 0) {
        residual = inpL;                   // layer 0: residual = embedding
        cur = group_rms_norm(inpL, attn_norm);
    } else {
        cur      = ggml_add(ctx0, inpL, residual);  // prev MLP out + accumulated sum
        residual = cur;
        cur      = group_rms_norm(cur, attn_norm);
    }
    /* ...attention... */

    // --- parallel norm BEFORE FFN ---
    cur      = ggml_add(ctx0, cur, residual);       // attn out + accumulated sum
    residual = cur;
    cur      = group_rms_norm(cur, ffn_norm);
    /* ...MoE FFN... */

    cur  = build_cvec(cur, il);
    inpL = cur;                                 // NO residual add here (deferred)
}

// final layer: fold last MLP output into the accumulated sum, then normalize
cur = ggml_add(ctx0, inpL, residual);
cur = group_rms_norm(cur, output_norm);

(Full, exact diff: patches/0002-fix-k2-horizon-parallel-norm.patch)

Compile

Windows (Visual Studio 2022 + CUDA):

cmake -B build -G "Visual Studio 17 2022" -A x64 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
cmake --build build --config Release

Linux (GCC/Clang + CUDA):

cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
cmake --build build --config Release -j$(nproc)

CPU-only builds work too (drop -DLLAMA_CUDA=ON).

Verify your build

./build/bin/Release/llama-cli -m K2-Horizon-36B-i-compact.gguf -n 30 --temp 0.7 --top-p 0.9 -p "Merhaba dünya"

✅ Working: coherent natural-language answer (~27–45 tok/s decode with full CUDA offload). ❌ Broken: repeated > or command-line echo → one of the patches was not applied, or you are running an unpatched (vanilla/upstream) llama.cpp build.

📋 Technical Details

Architecture Information (from GGUF metadata)

Architecture: k2-horizon
Block Count: 48 layers (3 leading dense blocks, 45 MoE blocks)
Expert Count: 100 routed experts/layer
Expert Used Count: 8 (top-8 routing)
Shared Experts: 1/layer
Expert Gating: softmax with weight normalization (scale 2.5)
Hidden Size: 2,560
Feed Forward Size: 6,144 (dense) / 768 (per expert)
Attention Heads: 32 (KV heads: 8, grouped query attention)
Head Dimension: 128 (full RoPE rotation)
RoPE Frequency Base: 10,000,000
Layer Norm: Grouped RMSNorm (2 groups), ε = 1e-6
MoVA: Mixture-of-Value Attention — 64 value experts/layer, top-4 routed
Residual Scheme: Parallel (pre-norm), as implemented by Patch 2 above — see the status note for verification caveats.

Quantize Profile Details

Layer bands (48 layers): EDGE = L0–4 & L43–47 · NEAR = L5–9 & L38–42 · MID = L10–37

i-quality (Q6_K/Q5_K/IQ4_XS)

Routed Expert FFN: EDGE Q6_K / NEAR Q5_K / MID IQ4_XS
Shared FFN: Q8_0
Attention: Q6_K
BPW: 5.47
File Size: 23.8 GB

i-balanced (Q6_K/Q5_K)

Routed Expert FFN: EDGE Q6_K / NEAR Q5_K / MID Q5_K
Shared FFN: Q8_0
Attention: Q6_K
BPW: 6.02
File Size: 26.2 GB

i-compact (Q4_K/Q3_K)

Routed Expert FFN: EDGE Q4_K / NEAR Q3_K / MID Q3_K
Shared FFN: Q6_K
Attention: Q4_K
BPW: 4.04
File Size: 17.6 GB

💻 Usage

⚠️ Both examples below assume you already built the patched fork from the section above. Running these commands against a stock llama-cli/llama-server build (including the one installed via the "Use this model" buttons at the top of this page) will not work.

With llama.cpp (patched fork)

# Interactive chat
./build/bin/Release/llama-cli -m models/K2-Horizon-36B-i-quality.gguf \
    -n 256 --temp 0.7 --top-p 0.9 -p "Hello, how are you?"

# Single-shot completion
./build/bin/Release/llama-completion -m models/K2-Horizon-36B-i-compact.gguf \
    -n 128 -p "Explain mixture-of-experts models:"

With Python (llama-cpp-python)

Build llama-cpp-python against the same patched source tree (see above), e.g.:

CMAKE_BUILD_ARGS="-DLLAMA_CURL=OFF" pip install llama-cpp-python \
    --global-option=build_ext \
    --global-option="--library_dir=$(pwd)/build/lib" \
    --global-option="--include_dir=$(pwd)/include"

from llama_cpp import Llama

llm = Llama(
    model_path="models/K2-Horizon-36B-i-quality.gguf",
    n_ctx=8192,
    n_threads=8,
)

output = llm("Hello, how are you?", max_tokens=256)
print(output["choices"][0]["text"])

📊 Model Comparison

Criterion	i-quality	i-balanced	i-compact
Quality	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐
Speed	⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐⭐
RAM/VRAM	High	Highest	Low
Size	23.8 GB	26.2 GB	17.6 GB
BPW	5.47	6.02	4.04

📝 Notes

All models are in GGUF format and were generated from the BF16 master checkpoint.
The K2Horizon architecture combines Mixture-of-Experts (100×8) with MoVA (64×4 value experts) and grouped RMSNorm. Residual wiring follows the patched behavior described in runtime fixes — see the status note there before treating it as a confirmed architectural fact.
Measured performance (full CUDA offload, 49/49 layers): prompt eval ≈ 28–139 tok/s, generation ≈ 27–45 tok/s (i-compact, consumer GPU).
Only the 3 APEX profiles are published here; the 70 GB BF16 master stays upstream.
If you arrived here via the "Use this model" quick-start buttons at the top of this page: those commands run vanilla llama.cpp and will not work with this GGUF yet. Please use the patched-fork instructions above instead.

📄 License

⚠️ Please review the original model's license/terms before commercial use. The GGUF files are repackaged quantizations of the upstream weights.

🙏 Acknowledgments

MBZUAI-IFM — K2-Horizon model and official llama.cpp fork
localai-org/apex-quant — Apex-Quant MoE-aware mixed-precision quantization framework
ggerganov/llama.cpp — GGUF format and quantization engine

🔗 Related Links

Source GGUF: https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
llama.cpp fork: https://github.com/MBZUAI-IFM/llama.cpp
Apex-Quant: https://github.com/localai-org/apex-quant
GGUF Format: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md

Downloads last month: 1,328

GGUF

Model size

37B params

Architecture

k2-horizon

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Base model

IFM/K2-Horizon-MoVA-36B-A4B

Quantized

(17)

this model

K2-Horizon-MoVA-36B-A4B — Apex Quant GGUF Models

🛑 READ BEFORE YOU RUN THIS MODEL

📦 Model Profile Summary

📁 File Structure

🔗 Source Model

🛠️ Technology

⚙️ Building the Runtime (patched llama.cpp)

Why a patched fork is required

Step-by-step

Patch 1 — MSVC tokenizer regex fix (src/llama-vocab.cpp)

Patch 2 — Parallel norm residual fix (src/models/k2-horizon.cpp)

Compile

Verify your build

📋 Technical Details

Architecture Information (from GGUF metadata)

Quantize Profile Details

i-quality (Q6_K/Q5_K/IQ4_XS)

i-balanced (Q6_K/Q5_K)

i-compact (Q4_K/Q3_K)

💻 Usage

With llama.cpp (patched fork)

With Python (llama-cpp-python)

📊 Model Comparison

📝 Notes

📄 License

🙏 Acknowledgments

🔗 Related Links

Model tree for vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Patch 1 — MSVC tokenizer regex fix (`src/llama-vocab.cpp`)

Patch 2 — Parallel norm residual fix (`src/models/k2-horizon.cpp`)