Upload 3 files

README.md

@@ -1,3 +1,278 @@
 ---
-license: apache-2.0
+license: other
+base_model: IFM/K2-Horizon-MoVA-36B-A4B-GGUF
+base_model_relation: quantized
+tags:
+- gguf
+- moe
+- k2-horizon
+- mova
+- apex-quant
+pipeline_tag: text-generation
 ---
+
+# K2-Horizon-MoVA-36B-A4B — Apex Quant GGUF Models
+
+This repository contains 3 quantized GGUF profiles of the **K2-Horizon-MoVA-36B-A4B** model, produced using **Apex-Quant** technology.
+
+> ⚠️ **Runtime requirement:** These models require the **patched llama.cpp fork** described in [Building the Runtime](#-building-the-runtime-patched-llamacpp). Unpatched builds of the fork either fail to load on Windows (MSVC regex error) or produce **garbage output** (repetitive tokens such as `>`). See [Fixes](#-why-a-patched-fork-is-required).
+
+## 📦 Model Profile Summary
+
+| Profile | Size | BPW | Use Case |
+|---------|------|-----|----------|
+| **i-quality** | 23.8 GB | 5.47 | Highest quality, production environments |
+| **i-balanced** | 26.2 GB | 6.02 | Maximum expert precision (densest expert quant) |
+| **i-compact** | 17.6 GB | 4.04 | Compact deployment, low VRAM/RAM |
+
+> **BPW** = Bits Per Weight. Higher value = better quality.
+>
+> Note: unlike most model families, here **i-balanced is larger than i-quality** — it keeps all routed experts at Q5_K or higher, while i-quality compresses mid-layer experts to IQ4_XS. Choose based on your memory budget vs. precision preference.
+
+## 📁 File Structure
+
+```
+models/
+├── K2-Horizon-36B-i-quality.gguf    # 23.8 GB — highest quality
+├── K2-Horizon-36B-i-balanced.gguf   # 26.2 GB — max expert precision
+└── K2-Horizon-36B-i-compact.gguf    # 17.6 GB — compact
+```
+
+## 🔗 Source Model
+
+These models were created based on the **K2-Horizon-MoVA-36B-A4B** model.
+
+- **Model Page:** https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
+- **Architecture:** K2Horizon MoE with MoVA (Mixture-of-Value Attention)
+- **Total Parameters:** ~36B (~4B active per token)
+- **Context Length:** 524,288 tokens (training context; usable range depends on KV-cache RAM)
+- **Reference Implementation:** vLLM (`K2HorizonForCausalLM`)
+- **Official llama.cpp fork:** https://github.com/MBZUAI-IFM/llama.cpp
+
+## 🛠️ Technology
+
+These quantized models were produced using **Apex-Quant** technology.
+
+- **Apex-Quant:** MoE-aware mixed-precision quantization (layer-band importance: edge / near / mid)
+- **Infrastructure:** llama.cpp (`llama-quantize`, `--tensor-type-file`)
+- **Config Generator:** `apex-quant/scripts/generate_config.sh --profile <profile> --layers 48`
+
+## ⚙️ Building the Runtime (patched llama.cpp)
+
+### Why a patched fork is required
+
+The K2-Horizon support fork ([MBZUAI-IFM/llama.cpp](https://github.com/MBZUAI-IFM/llama.cpp)) provides the base K2-Horizon support. On top of the pinned commit below, **two fixes** are required for reliable operation (both verified empirically in our test setup):
+
+| # | Issue | Observed Symptom | Fix |
+|---|-------|------------------|-----|
+| 1 | Tokenizer pre-regex contains `\u200C`/`\u200D` escapes unsupported by MSVC `std::regex` | Model fails to load on **Windows**: `Failed to process regex ... regex_error(error_escape)` | Remove the `\u200C\|\u200D` alternation from the K2_HORIZON pre-tokenization regex |
+| 2 | Decoder residual/norm wiring diverged from the vLLM reference behavior | Model loads fine but generates **garbage/repeated tokens** (`> > > ...`), possible crashes during graph reservation | Rewrite residual flow in `k2-horizon.cpp` forward pass following vLLM semantics (see note below) |
+
+Both fixes are provided as ready-to-apply patches below.
+
+### Step-by-step
+
+```bash
+# 1. Clone the official K2-Horizon fork and pin the exact commit these models were validated against
+git clone https://github.com/MBZUAI-IFM/llama.cpp llama-cpp-k2horizon
+cd llama-cpp-k2horizon
+git checkout 35999d101cf2233fc54f09c3c8d599da7303ce02
+
+# 2. Apply the two patches (from this repo's patches/ directory)
+git apply /path/to/patches/0001-fix-k2-horizon-msvc-regex.patch
+git apply /path/to/patches/0002-fix-k2-horizon-parallel-norm.patch
+```
+
+#### Patch 1 — MSVC tokenizer regex fix (`src/llama-vocab.cpp`)
+
+MSVC's `std::regex` rejects `\uXXXX` escape sequences, so loading any K2-Horizon model crashes on Windows before inference even starts. The zero-width joiner/non-joiner alternatives are irrelevant for BPE pre-tokenization quality and are removed:
+
+```diff
+             case LLAMA_VOCAB_PRE_TYPE_K2_HORIZON:
+                 regex_exprs = {
+-                    "...(?:\\p{L}|\\p{M}|\\u200C|\\u200D)+...",
++                    "...(?:\\p{L}|\\p{M})+...",
+                 };
+```
+
+*(Full, exact diff: [`patches/0001-fix-k2-horizon-msvc-regex.patch`](patches/0001-fix-k2-horizon-msvc-regex.patch))*
+
+#### Patch 2 — Parallel norm residual fix (`src/models/k2-horizon.cpp`)
+
+This patch rewires the decoder residual path to follow the vLLM reference semantics (`K2HorizonRMSNorm.forward(x, residual)`), where the running sum of sublayer outputs is tracked across layers and folded in *before* normalization.
+
+> **Status note:** With this patch applied, the repeated-token (`> > >`) failure mode and the graph-reservation crash disappeared and generation became coherent. The exact causal mechanism has **not** been independently verified against the upstream weights — treat this patch as a validated fix rather than a proven diagnosis.
+
+The new residual flow tracks an accumulated `residual` tensor across layers:
+
+```cpp
+ggml_tensor * residual = nullptr;          // outside the layer loop
+
+for (int il = 0; il < n_layer; ++il) {
+    // --- parallel norm BEFORE attention ---
+    if (il == 0) {
+        residual = inpL;                   // layer 0: residual = embedding
+        cur = group_rms_norm(inpL, attn_norm);
+    } else {
+        cur      = ggml_add(ctx0, inpL, residual);  // prev MLP out + accumulated sum
+        residual = cur;
+        cur      = group_rms_norm(cur, attn_norm);
+    }
+    /* ...attention... */
+
+    // --- parallel norm BEFORE FFN ---
+    cur      = ggml_add(ctx0, cur, residual);       // attn out + accumulated sum
+    residual = cur;
+    cur      = group_rms_norm(cur, ffn_norm);
+    /* ...MoE FFN... */
+
+    cur  = build_cvec(cur, il);
+    inpL = cur;                                 // NO residual add here (deferred)
+}
+
+// final layer: fold last MLP output into the accumulated sum, then normalize
+cur = ggml_add(ctx0, inpL, residual);
+cur = group_rms_norm(cur, output_norm);
+```
+
+*(Full, exact diff: [`patches/0002-fix-k2-horizon-parallel-norm.patch`](patches/0002-fix-k2-horizon-parallel-norm.patch))*
+
+### Compile
+
+**Windows (Visual Studio 2022 + CUDA):**
+```bat
+cmake -B build -G "Visual Studio 17 2022" -A x64 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
+cmake --build build --config Release
+```
+
+**Linux (GCC/Clang + CUDA):**
+```bash
+cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON
+cmake --build build --config Release -j$(nproc)
+```
+
+CPU-only builds work too (drop `-DLLAMA_CUDA=ON`).
+
+### Verify your build
+
+```bash
+./build/bin/Release/llama-cli -m K2-Horizon-36B-i-compact.gguf -n 30 --temp 0.7 --top-p 0.9 -p "Merhaba dünya"
+```
+
+✅ **Working:** coherent natural-language answer (~27–45 tok/s decode with full CUDA offload).
+❌ **Broken:** repeated `>` or command-line echo → one of the patches was not applied.
+
+## 📋 Technical Details
+
+### Architecture Information (from GGUF metadata)
+- **Architecture:** `k2-horizon`
+- **Block Count:** 48 layers (3 leading dense blocks, 45 MoE blocks)
+- **Expert Count:** 100 routed experts/layer
+- **Expert Used Count:** 8 (top-8 routing)
+- **Shared Experts:** 1/layer
+- **Expert Gating:** softmax with weight normalization (scale 2.5)
+- **Hidden Size:** 2,560
+- **Feed Forward Size:** 6,144 (dense) / 768 (per expert)
+- **Attention Heads:** 32 (KV heads: 8, grouped query attention)
+- **Head Dimension:** 128 (full RoPE rotation)
+- **RoPE Frequency Base:** 10,000,000
+- **Layer Norm:** Grouped RMSNorm (2 groups), ε = 1e-6
+- **MoVA:** Mixture-of-Value Attention — 64 value experts/layer, top-4 routed
+- **Residual Scheme:** Parallel (pre-norm), per vLLM reference
+
+### Quantize Profile Details
+
+Layer bands (48 layers): **EDGE** = L0–4 & L43–47 · **NEAR** = L5–9 & L38–42 · **MID** = L10–37
+
+#### i-quality (Q6_K/Q5_K/IQ4_XS)
+- **Routed Expert FFN:** EDGE Q6_K / NEAR Q5_K / MID IQ4_XS
+- **Shared FFN:** Q8_0
+- **Attention:** Q6_K
+- **BPW:** 5.47
+- **File Size:** 23.8 GB
+
+#### i-balanced (Q6_K/Q5_K)
+- **Routed Expert FFN:** EDGE Q6_K / NEAR Q5_K / MID Q5_K
+- **Shared FFN:** Q8_0
+- **Attention:** Q6_K
+- **BPW:** 6.02
+- **File Size:** 26.2 GB
+
+#### i-compact (Q4_K/Q3_K)
+- **Routed Expert FFN:** EDGE Q4_K / NEAR Q3_K / MID Q3_K
+- **Shared FFN:** Q6_K
+- **Attention:** Q4_K
+- **BPW:** 4.04
+- **File Size:** 17.6 GB
+
+## 💻 Usage
+
+### With llama.cpp (patched fork)
+
+```bash
+# Interactive chat
+./build/bin/Release/llama-cli -m models/K2-Horizon-36B-i-quality.gguf \
+    -n 256 --temp 0.7 --top-p 0.9 -p "Hello, how are you?"
+
+# Single-shot completion
+./build/bin/Release/llama-completion -m models/K2-Horizon-36B-i-compact.gguf \
+    -n 128 -p "Explain mixture-of-experts models:"
+```
+
+### With Python (llama-cpp-python)
+
+Build `llama-cpp-python` **against the same patched source tree** (see above), e.g.:
+
+```bash
+CMAKE_BUILD_ARGS="-DLLAMA_CURL=OFF" pip install llama-cpp-python \
+    --global-option=build_ext \
+    --global-option="--library_dir=$(pwd)/build/lib" \
+    --global-option="--include_dir=$(pwd)/include"
+```
+
+```python
+from llama_cpp import Llama
+
+llm = Llama(
+    model_path="models/K2-Horizon-36B-i-quality.gguf",
+    n_ctx=8192,
+    n_threads=8,
+)
+
+output = llm("Hello, how are you?", max_tokens=256)
+print(output["choices"][0]["text"])
+```
+
+## 📊 Model Comparison
+
+| Criterion | i-quality | i-balanced | i-compact |
+|-----------|-----------|------------|-----------|
+| **Quality** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ |
+| **Speed** | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐�� |
+| **RAM/VRAM** | High | Highest | Low |
+| **Size** | 23.8 GB | 26.2 GB | 17.6 GB |
+| **BPW** | 5.47 | 6.02 | 4.04 |
+
+## 📝 Notes
+
+- All models are in **GGUF** format and were generated from the BF16 master checkpoint.
+- The **K2Horizon** architecture combines Mixture-of-Experts (100×8) with **MoVA** (64×4 value experts) and grouped RMSNorm. Residual wiring follows the vLLM reference (see [runtime fixes](#-building-the-runtime-patched-llamacpp)).
+- Measured performance (full CUDA offload, 49/49 layers): prompt eval ≈ 28–139 tok/s, generation ≈ 27–45 tok/s (i-compact, consumer GPU).
+- Only the 3 APEX profiles are published here; the 70 GB BF16 master stays upstream.
+
+## 📄 License
+
+⚠️ Please review the **original model's license/terms** before commercial use. The GGUF files are repackaged quantizations of the upstream weights.
+
+## 🙏 Acknowledgments
+
+- **[MBZUAI-IFM](https://github.com/MBZUAI-IFM)** — K2-Horizon model and official llama.cpp fork
+- **[localai-org/apex-quant](https://github.com/localai-org/apex-quant)** — Apex-Quant MoE-aware mixed-precision quantization framework
+- **[ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)** — GGUF format and quantization engine
+
+## 🔗 Related Links
+
+- **Source GGUF:** https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B-GGUF
+- **llama.cpp fork:** https://github.com/MBZUAI-IFM/llama.cpp
+- **Apex-Quant:** https://github.com/localai-org/apex-quant
+- **GGUF Format:** https://github.com/ggerganov/ggml/blob/master/docs/gguf.md

patches/0001-fix-k2-horizon-msvc-regex.patch

@@ -0,0 +1,13 @@
+diff --git a/src/llama-vocab.cpp b/src/llama-vocab.cpp
+index 4d054db..00c7c68 100644
+--- a/src/llama-vocab.cpp
++++ b/src/llama-vocab.cpp
+@@ -537,7 +537,7 @@ struct llm_tokenizer_bpe : llm_tokenizer {
+                 break;
+             case LLAMA_VOCAB_PRE_TYPE_K2_HORIZON:
+                 regex_exprs = {
+-                    "(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\\r\\n\\p{L}\\p{N}]?(?:\\p{L}|\\p{M}|\\u200C|\\u200D)+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
++                    "(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\\r\\n\\p{L}\\p{N}]?(?:\\p{L}|\\p{M})+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+                 };
+                 break;
+             case LLAMA_VOCAB_PRE_TYPE_WHITESPACE:

patches/0002-fix-k2-horizon-parallel-norm.patch

@@ -0,0 +1,98 @@
+diff --git a/src/models/k2-horizon.cpp b/src/models/k2-horizon.cpp
+index ac901da..f8ee389 100644
+--- a/src/models/k2-horizon.cpp
++++ b/src/models/k2-horizon.cpp
+@@ -393,22 +393,37 @@ llama_model_k2_horizon::graph::graph(
+     auto * inp_attn = build_attn_inp_kv();
+     ggml_tensor * inp_out_ids = build_inp_out_ids();
+     
++    // ============ PARALLEL NORM (GPT-NeoX/PaLM style) - vLLM reference
++    // residual tracks accumulated sum across layers
++    ggml_tensor * residual = nullptr;
+ 
+     for (int il = 0; il < n_layer; ++il) {
+         res->t_layer_inp[il] = inpL;
+-        ggml_tensor * inpSA = inpL; // for residuals
+ 
+         const bool is_moe_layer = n_expert > 0 && static_cast<uint32_t>(il) >= hparams.n_layer_dense_lead;
+         const bool is_mova_layer = is_moe_layer && hparams.n_value_expert > 0;
+ 
+-        // ============ grouped rms norm
+-        cur = k2_horizon_group_rms_norm(
+-            ctx0,
+-            inpL,
+-            model.layers[il].attn_norm,
+-            hparams.n_norm_groups,
+-            hparams.f_norm_rms_eps
+-        );
++        // ============ parallel norm before attention
++        if (il == 0) {
++            residual = inpL;
++            cur = k2_horizon_group_rms_norm(
++                ctx0,
++                inpL,
++                model.layers[il].attn_norm,
++                hparams.n_norm_groups,
++                hparams.f_norm_rms_eps
++            );
++        } else {
++            cur = ggml_add(ctx0, inpL, residual);
++            residual = cur;
++            cur = k2_horizon_group_rms_norm(
++                ctx0,
++                cur,
++                model.layers[il].attn_norm,
++                hparams.n_norm_groups,
++                hparams.f_norm_rms_eps
++            );
++        }
+         cb(cur, "attn_norm", il);
+ 
+         // ============ setup attention tensors
+@@ -547,17 +562,15 @@ llama_model_k2_horizon::graph::graph(
+         // ============ output layer, and take (usually) last token for generation
+         if (il == n_layer - 1 && inp_out_ids != nullptr) {
+             cur = ggml_get_rows(ctx0, cur, inp_out_ids);
+-            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids); // pull the same positions for inpSA
++            residual = ggml_get_rows(ctx0, residual, inp_out_ids);
+         }
+ 
+-        // ============ add residuals
+-        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
+-        cb(ffn_inp, "ffn_inp", il);
+-
+-        // ============ group RMSNorm before FFN
++        // ============ parallel norm before FFN
++        cur = ggml_add(ctx0, cur, residual);
++        residual = cur;
+         cur = k2_horizon_group_rms_norm(
+             ctx0,
+-            ffn_inp,
++            cur,
+             model.layers[il].ffn_norm,
+             hparams.n_norm_groups,
+             hparams.f_norm_rms_eps
+@@ -626,8 +639,7 @@ llama_model_k2_horizon::graph::graph(
+         }
+         cb(cur, "ffn_out", il);
+         
+-        // ============ FFN residual
+-        cur = ggml_add(ctx0, cur, ffn_inp);
++        // ============ cvec adapter (residual add happens next layer / final norm)
+         cur = build_cvec(cur, il);
+         cb(cur, "l_out", il);
+         
+@@ -635,10 +647,11 @@ llama_model_k2_horizon::graph::graph(
+         inpL = cur;
+     }
+ 
+-    // final group rms norm. also becomes last layer embedding
++    // final: add last mlp_out to accumulated residual, then group rms norm
++    cur = ggml_add(ctx0, inpL, residual);
+     cur = k2_horizon_group_rms_norm(
+         ctx0,
+-        inpL,
++        cur,
+         model.output_norm,
+         hparams.n_norm_groups,
+         hparams.f_norm_rms_eps