Instructions to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
./llama-cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Use Docker

docker model run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

LM Studio
Jan

vLLM

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Ollama
How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Ollama:
```
ollama run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
```
Unsloth Desktop

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Docker Model Runner:
```
docker model run hf.co/vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF
```

Lemonade

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Run and chat with the model

lemonade run user.K2-Horizon-MoVA-36B-A4B-APEX-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vincespeed/K2-Horizon-MoVA-36B-A4B-APEX-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

K2-Horizon-MoVA-36B-A4B-APEX-GGUF / patches /0002-fix-k2-horizon-parallel-norm.patch

vincespeed

Upload 3 files

9653662 verified 4 days ago

Raw

History Blame Contribute Delete

3.61 kB

	diff --git a/src/models/k2-horizon.cpp b/src/models/k2-horizon.cpp
	index ac901da..f8ee389 100644
	--- a/src/models/k2-horizon.cpp
	+++ b/src/models/k2-horizon.cpp
	@@ -393,22 +393,37 @@ llama_model_k2_horizon::graph::graph(
	auto * inp_attn = build_attn_inp_kv();
	ggml_tensor * inp_out_ids = build_inp_out_ids();

	+ // ============ PARALLEL NORM (GPT-NeoX/PaLM style) - vLLM reference
	+ // residual tracks accumulated sum across layers
	+ ggml_tensor * residual = nullptr;

	for (int il = 0; il < n_layer; ++il) {
	res->t_layer_inp[il] = inpL;
	- ggml_tensor * inpSA = inpL; // for residuals

	const bool is_moe_layer = n_expert > 0 && static_cast<uint32_t>(il) >= hparams.n_layer_dense_lead;
	const bool is_mova_layer = is_moe_layer && hparams.n_value_expert > 0;

	- // ============ grouped rms norm
	- cur = k2_horizon_group_rms_norm(
	- ctx0,
	- inpL,
	- model.layers[il].attn_norm,
	- hparams.n_norm_groups,
	- hparams.f_norm_rms_eps
	- );
	+ // ============ parallel norm before attention
	+ if (il == 0) {
	+ residual = inpL;
	+ cur = k2_horizon_group_rms_norm(
	+ ctx0,
	+ inpL,
	+ model.layers[il].attn_norm,
	+ hparams.n_norm_groups,
	+ hparams.f_norm_rms_eps
	+ );
	+ } else {
	+ cur = ggml_add(ctx0, inpL, residual);
	+ residual = cur;
	+ cur = k2_horizon_group_rms_norm(
	+ ctx0,
	+ cur,
	+ model.layers[il].attn_norm,
	+ hparams.n_norm_groups,
	+ hparams.f_norm_rms_eps
	+ );
	+ }
	cb(cur, "attn_norm", il);

	// ============ setup attention tensors
	@@ -547,17 +562,15 @@ llama_model_k2_horizon::graph::graph(
	// ============ output layer, and take (usually) last token for generation
	if (il == n_layer - 1 && inp_out_ids != nullptr) {
	cur = ggml_get_rows(ctx0, cur, inp_out_ids);
	- inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids); // pull the same positions for inpSA
	+ residual = ggml_get_rows(ctx0, residual, inp_out_ids);
	}

	- // ============ add residuals
	- ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
	- cb(ffn_inp, "ffn_inp", il);
	-
	- // ============ group RMSNorm before FFN
	+ // ============ parallel norm before FFN
	+ cur = ggml_add(ctx0, cur, residual);
	+ residual = cur;
	cur = k2_horizon_group_rms_norm(
	ctx0,
	- ffn_inp,
	+ cur,
	model.layers[il].ffn_norm,
	hparams.n_norm_groups,
	hparams.f_norm_rms_eps
	@@ -626,8 +639,7 @@ llama_model_k2_horizon::graph::graph(
	}
	cb(cur, "ffn_out", il);

	- // ============ FFN residual
	- cur = ggml_add(ctx0, cur, ffn_inp);
	+ // ============ cvec adapter (residual add happens next layer / final norm)
	cur = build_cvec(cur, il);
	cb(cur, "l_out", il);

	@@ -635,10 +647,11 @@ llama_model_k2_horizon::graph::graph(
	inpL = cur;
	}

	- // final group rms norm. also becomes last layer embedding
	+ // final: add last mlp_out to accumulated residual, then group rms norm
	+ cur = ggml_add(ctx0, inpL, residual);
	cur = k2_horizon_group_rms_norm(
	ctx0,
	- inpL,
	+ cur,
	model.output_norm,
	hparams.n_norm_groups,
	hparams.f_norm_rms_eps