gaber/kimi-k3-dspark-gguf

Kimi-K3 DSpark Draft Model — GGUF (BF16)

⚠️ Experimental — This GGUF was converted from a model that's only 2 days old (as of 2026-07-30). The file format is correct but upstream llama.cpp cannot run it yet — see Runtime status below.

A GGUF conversion of Inferact/Kimi-K3-DSpark, an MLA-native DSpark draft model designed to accelerate moonshotai/Kimi-K3 inference through speculative decoding.

What is this?

DSpark is a small draft model that predicts multiple tokens ahead of the main Kimi-K3 model. The main model then verifies those predictions in a single forward pass, accepting correct ones and rejecting mistakes. This gives you 3–5× faster generation without changing the output quality — the distribution is mathematically identical to running the target model alone.

This particular draft model is "MLA-native" — it uses the same Multi-head Latent Attention architecture as Kimi-K3 itself, which means the draft and target share one KV cache layout. No separate memory management needed.

How it works

• 5 transformer layers with MLA attention (q_lora_rank=1536, kv_lora_rank=512)
• Markov head — a semi-autoregressive block predictor that drafts 7 tokens per forward pass
• Confidence head — decides when to draft and when to fall back to the target model
• Trained on Kimi-K3's own hidden states from layers [2, 23, 47, 71, 89] of 93

GGUF vs original model

The original Inferact/Kimi-K3-DSpark is 4B params (7.12 GB) because it includes embed_tokens.weight — a 1.17B parameter embedding matrix shared with Kimi-K3's vocabulary. This GGUF is 2.4B params (4.45 GB) because we stripped that tensor.

Why? Because llama.cpp's DFlash runtime never loads it. The draft model only processes hidden states from the target — it never sees raw token IDs at inference time. The embed_tokens was used during training (TorchSpec needs it for the forward pass) but is dead weight at runtime.

Verified: We compared the embed_tokens tensors from both models and confirmed they are not identical — DSpark was trained with its own embeddings, optimized for its 5-layer architecture. This is expected: a shallow model benefits from different embedding geometry than a 93-layer one.

vLLM vs llama.cpp

This same model runs on two engines, with different assumptions:

vLLM (works today):

vllm serve moonshotai/Kimi-K3 \
  --speculative-config '{"method": "dspark", "model": "Inferact/Kimi-K3-DSpark", ...}'

• Loads the full 4B-param safetensors (with embed_tokens)
• Draft runs as a standalone model with its own embedding layer
• Both share MLA KV layout — draft's pages unify with target's cache
• Native FLASHINFER_MLA backend, dedicated dspark spec decode worker
• Achieves 464 tok/s on 4× GB300 at bs=1

llama.cpp (needs runtime patch):

llama-server -m kimi-k3.gguf -md dspark-draft.gguf --spec-type draft-dspark

• Loads stripped 2.4B-param GGUF (no embed_tokens)
• Draft is a lightweight extension of the target
• Target handles tokenization/embedding, passes hidden states directly
• Needs MLA support ported from kimi_linear.cpp to dflash.cpp

Both approaches are lossless — the output distribution is identical to running the target model alone.

GGUF metadata

Key	Value
Architecture	dflash
Block size	7
Target layers	[3, 4, 5, 6, 7, 8, 9]
Embedding dim	7168
Vocab size	163,840
KV lora rank	512
Q lora rank	1536
Quantization	BF16
Tensors	72
File size	4.45 GB

Runtime status

The GGUF file is structurally correct — architecture, tensor names, MLA parameters, DSpark heads, and tokenizer are all properly encoded.

However, upstream llama.cpp cannot run it yet. The C++ runtime's DFLASH model implementation (src/models/dflash.cpp) currently only loads standard multi-head attention tensors (attn_q, attn_k, attn_v). This GGUF uses MLA tensors (attn_q_a, attn_kv_a_mqa, attn_k_b, attn_v_b) which require a patch to the runtime's tensor loading and attention graph.

In other words: the file format is right, but the engine doesn't know how to read the MLA tensors inside the DFLASH architecture yet. The code even acknowledges this with a TODO comment:

"only Qwen3-style backbones are supported for now; other backbones need their own conversion path and graph tweaks"

What's needed from llama.cpp

1. src/models/dflash.cpp — Add MLA tensor loading (attn_q_a, attn_kv_a_mqa, attn_k_b, attn_v_b) to the DFLASH architecture, similar to how src/models/kimi_linear.cpp handles them
2. Attention graph — Build the MLA decompression graph (compressed KV → full K/V via kv_b_proj) inside the DFLASH forward pass
3. Rope handling — MLA uses split rope dimensions (qk_nope_head_dim + qk_rope_head_dim) which the standard DFLASH path doesn't support

These changes exist in kimi_linear.cpp already — they need to be ported to dflash.cpp. Once upstream adds MLA support to DFLASH, this GGUF should work with:

llama-server \
  -m kimi-k3-target.gguf \
  -md Kimi-K3-DSpark-draft-bf16.gguf \
  --spec-type draft-dspark \
  -ngld 99

Conversion details

Converted from Inferact/Kimi-K3-DSpark (safetensors) using a patched convert_hf_to_gguf.py. The converter needed three patches:

1. Register K3DSparkModel architecture in conversion/__init__.py
2. Add K3DSparkModel converter class in conversion/kimi_linear.py with MLA parameter handling and tensor name remapping
3. Add MLA tensor definitions to the DFLASH arch in gguf-py/gguf/constants.py

The patches and conversion script are saved in the modal_agent project under converters/.

Links

• Source model: Inferact/Kimi-K3-DSpark
• Target model: moonshotai/Kimi-K3
• DSpark paper: arxiv.org/abs/2607.05147
• Speculative decoding in llama.cpp: docs/speculative.md