Instructions to use vcruz305/Kimi-K3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use vcruz305/Kimi-K3-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S
# Run inference directly in the terminal:
llama cli -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S
# Run inference directly in the terminal:
llama cli -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S
# Run inference directly in the terminal:
./llama-cli -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Use Docker

docker model run hf.co/vcruz305/Kimi-K3-GGUF:UD-IQ1_S

LM Studio
Jan

vLLM

How to use vcruz305/Kimi-K3-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vcruz305/Kimi-K3-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vcruz305/Kimi-K3-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Ollama
How to use vcruz305/Kimi-K3-GGUF with Ollama:
```
ollama run hf.co/vcruz305/Kimi-K3-GGUF:UD-IQ1_S
```

Unsloth Studio

How to use vcruz305/Kimi-K3-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for vcruz305/Kimi-K3-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for vcruz305/Kimi-K3-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for vcruz305/Kimi-K3-GGUF to start chatting

How to use vcruz305/Kimi-K3-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vcruz305/Kimi-K3-GGUF:UD-IQ1_S"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use vcruz305/Kimi-K3-GGUF with Docker Model Runner:
```
docker model run hf.co/vcruz305/Kimi-K3-GGUF:UD-IQ1_S
```

Lemonade

How to use vcruz305/Kimi-K3-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Run and chat with the model

lemonade run user.Kimi-K3-GGUF-UD-IQ1_S

List all available models

lemonade list

Hermes Agent

How to use vcruz305/Kimi-K3-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use vcruz305/Kimi-K3-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vcruz305/Kimi-K3-GGUF:UD-IQ1_S

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vcruz305/Kimi-K3-GGUF:UD-IQ1_S" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Kimi-K3 UD-IQ1_S GGUF

Kimi-K3 (2.78 T parameters) compressed to 330.2 GB IQ1_S GGUF (307.49 GiB, 9 shards) — about 21% of the ~1.56 TB source artifact.

Every one of the 82,432 routed experts is present — no experts are dropped. The GGUF loads without model-specific patches on an existing Kimi-K3-capable llama.cpp build.

The exact method used to create this model will be released soon. This card documents what the model is, how to run it, and how it measures — not how it was built.

Benchmarks

Benchmark	K3 IQ1_S GGUF	Full K3	Notes
HumanEval	94.5% (155/164)	94.5%	1:1 match
MMLU	79.49%	85%	Within expected IQ1_S gap
AIME	92.5%	96%
GSM8K	95%	95%

Metric	Value
Size	330,167,807,328 bytes / 307.49 GiB (9 shards)
vs. smallest published K3 quant	44% smaller (594 GB → 330 GB)
Weights resident	316,036 MiB, measured on 8× A100-80GB
Speed (llama.cpp, single-stream)	10.8 tok/s
Speed (3× DGX Spark TP3, SparkInfer)	12.5492 tok/s (K=8 profile, 112/112 accepted drafts)
WikiText-2 PPL	5.3610 ± 0.02917 (canonical `wiki.test.raw`, all 568 chunks, `-c 512`)
Full-MXFP4 top-1 agreement	73.741 ± 0.243% (32-window mixed-domain paired protocol)
Context	Architecture limit 1,048,576; 3× Spark TP3 practical ceiling ~8K for full resident weights (~113 GiB/rank)

1. SparkInfer TP3/TP4 — Optimal Path (Blackwell)

Current measured throughput: 12.5468 and 12.5516 tok/s, averaging 12.5492 tok/s around a matched 6.4930 baseline, with 112/112 accepted drafts and identical token IDs. This is the K=8 repeat-heavy structured profile from the SparkInfer TP3 recipe, verified on real DGX Spark (GB10) hardware.

SparkInfer is optimized for all Blackwell chips (H200, GB10/DGX Spark, and other GB10-based systems). The gittensor-ai-lab/sparkinfer-k3 repo runs on 8× H200 as the reference configuration; DGX Spark (GB10) has a reduced footprint (~113 GiB/rank under 121 GiB).

Runtime	Hardware	Measured decode
SparkInfer TP3	3× DGX Spark (GB10)	12.5492 tok/s (K=8 profile, 112/112 drafts)
SparkInfer TP4	4× DGX Spark (GB10)	7.90 tok/s
llama.cpp	8× A100-80GB	10.8 tok/s
vLLM TP3 + DSpark	3× H200	52.454 tok/s (coding)

Why MY SparkInfer Patch Is Optimal

Correctness verified 2026-08-11: The distributed (TP3/TP4) generation path had a bug in kimi_k3_dist_rank.cpp: it never set the GGUF capability flags that gate loading, so shared experts, routed-expert normalization, and MLA attention gate were silently skipped on every token — real weights present, never computed. MY fix adds the same capability probe every other code path already had.
Footprint: ~113 GiB/rank fits comfortably under GB10's 121 GiB.
Current measurements (corrected engine):
- Load time: ~5m45s/rank (was ~30-60 min pre-patch 0013)
- SPARKINFER_K3_KDA_FUSE=0: +20% over MY own baseline
- K=4 speculative decoding candidate: 10.3002 and 11.3231 tok/s (10.8117 avg), +68.2% over matched 6.4272 baseline, 95/95 accepted drafts
- K=8 repeat-heavy profile: 12.5468 and 12.5516 tok/s (12.5492 avg) around 6.4930 baseline, 112/112 accepted drafts, identical token IDs
Not valid for non-Blackwell chips: Tested on physical fleets, hard memory ceiling on other hardware.

Full launch procedure (see kimi-k3-neuron-tp3-dgxspark-recipe):

3 or 4 DGX Spark (GB10) nodes, each with ≥320 GB free NVMe
Pull model with hf_xet transport
Apply verified 28-patch chain through 0026
Build SparkInfer with MY patch series
Copy built dist/ tree to every rank node
Launch rank 0 (coordinator) with sparkinfer --model ./Kimi-K3-UD-IQ1_S --tp 3 --rank 0 --listen --ctx-len 8192 --jdba-band
Launch rank 1 and rank 2 (workers) with --coord <rank0-host>:9000 --jdba-band
Verify all ranks connected, then query

Shards load from Kimi-K3-UD-IQ1_S-00001-of-00009.gguf (all 9 parts required).

Speculative Decoding (Experimental, Opt-In)

Patches 0021–0026 add n-gram/prompt-lookup speculative decoding, match-confidence gating, recursive-majority continuation, and distributed LM-head banding. On real TP3 hardware, 64-token generations show:

Code/structured: +27% with --spec-draft 4
Literal repetition: +83% with --spec-draft 4
Freeform prose: −3.7%
Mean across all three: +34.5%

2. llama.cpp — Fallback (Any Chip)

Requires a llama.cpp build with Kimi-K3 support — upstream PR #26185 or the Unsloth K3 fork.

llama-server \
  -m Kimi-K3-UD-IQ1_S-00001-of-00009.gguf \
  -ngl 99 --jinja \
  -c 8192 --parallel 1 \
  -ctk f16 -ctv f16 -fa on \
  --temp 1.0 --top-p 0.95

3. vLLM TP3 + DSpark — Last Resort (Non-Blackwell)

Runtime	Hardware	Measured decode	Recipe
vLLM TP3 + DSpark	3× H200	52.454 tok/s (coding)	`kimi-k3-neuron-tp3-vllm-recipe`

Download

This repository is Xet-backed. Use the hf_xet transport — chunk-deduplicated and parallel:

pip install -U "huggingface_hub[hf_xet]"
hf auth login

export HF_XET_HIGH_PERFORMANCE=1
hf download vcruz305/Kimi-K3-GGUF --local-dir ./Kimi-K3-UD-IQ1_S

Model shards and template only:

hf download vcruz305/Kimi-K3-GGUF --local-dir ./Kimi-K3-UD-IQ1_S \
   --include "Kimi-K3-UD-IQ1_S-*.gguf" --include "k3_chat_template.jinja"

All 9 shards are required; llama.cpp loads them automatically from Kimi-K3-UD-IQ1_S-00001-of-00009.gguf.

Two things learned moving this artifact around:

Xet dedup is per chunk, not per file — when shard 00001 was rewritten to embed the chat template, the 39.4 GB re-upload transferred 395 kB.
Advertised bandwidth is meaningless: a host advertising 538 Mbps sustained 7,828 Mbps on real Xet transfers.

Status

Experimental, and it works. It loads, reasons in K3's native format, and scores 94.5% on HumanEval — measured on the full 164 problems, not a subset.

Changelog

2026-08-12: Renamed GGUF shards to Kimi-K3-UD-IQ1_S-*-of-00009.gguf (server-side pointer rename; no weight re-upload).
2026-08-11: Correctness fix applied and verified in SparkInfer TP3 engine — kimi_k3_dist_rank.cpp now sets GGUF capability flags that gate loading of shared experts, routed-expert normalization, and MLA attention gate. Pre-fix measurements were invalid (engine skipped real computation).
Pre-2026-08-11: Historical speed tables deprecated — measured on engine skipping computation.
Initial release: 330 GB IQ1_S GGUF with all 82,432 experts present, verified loading on Unsloth llama.cpp fork.

Downloads last month: 912

GGUF

Model size

1.4T params

Architecture

kimi-k3

Hardware compatibility

1-bit

Model tree for vcruz305/Kimi-K3-GGUF

Base model

moonshotai/Kimi-K3

Quantized

(46)

this model