---
base_model: novita/kimi-k2.7-code-dspark
tags:
- speculators
- safetensors
- speculative-decoding
- dspark
- text-generation
- custom_code
- base_model:moonshotai/Kimi-K2.7-Code
- base_model:finetune:moonshotai/Kimi-K2.7-Code
- region:us
---

# Kimi-K2.7-Code DSpark GGUF

GGUF quantizations of [**novita DSpark draft model**](https://huggingface.co/novita/kimi-k2.7-code-dspark) for [**Kimi-K2.7-Code**](https://huggingface.co/moonshotai/Kimi-K2.7-Code).

Use with [BeeLlama.cpp](https://github.com/Anbeeld/beellama.cpp), a llama.cpp fork with advanced quantization features.

---

# Kimi-K2.7-Code DSpark speculator

## Overview

A DSpark speculator model for the `Kimi-K2.7-Code` base model, enabling faster
inference through speculative decoding. DSpark extends the DFlash parallel draft
backbone with two lightweight heads: a **Markov logit-bias head** (low-rank
intra-block token dependency) and a **per-position confidence head** (accept-rate
prediction). This checkpoint was trained in the Camelot-Ray online pipeline,
where the draft consumes hidden states streamed from a live Kimi-K2.7-Code vLLM
server.

This export is from Camelot exp38 checkpoint 3.

## Model Specifications

- **Base Model**: Kimi-K2.7-Code
- **Format**: Safetensors (single-file bf16, 6.3 GB, 44 tensors)
- **Draft**: 3 layers (Qwen3-style GQA), hidden 7168, 56 heads / 8 KV heads,
  head_dim 128, FFN 18432, rope_theta 50000, `block_size=8`
- **Vocabulary**: pruned draft vocab 32,000 (d2t/t2d remap tables shipped in the
  weights), target vocab 163,840; mappings reused from the Kimi-K2.6/K2.7
  compatible tokenizer setup
- **DSpark heads**: Markov rank 256 (vanilla), confidence head (with-markov),
  `mask_token_id=163608`
- **Aux hidden-state layers**: [1, 29, 57]
- **Trained context**: seq 20000

## Evaluation Results

Online vLLM nightly spec-decode, greedy decoding, TP=8, Kimi-K2.7-Code verifier,
`max_model_len=20000`, cudagraphs enabled, and
`fuse_allreduce_rms=false`.

The table also includes Novita's public Eagle3-MLA draft
`novita/kimi-k2.7-code-eagle3-mla` under the same Kimi-K2.7-Code verifier,
TP=8, cudagraph, and fusion-off serving setup. Cells show
`tok/s / speedup / accept_len`. The standard rows use 6 prompts per benchmark;
code-extra rows use the full LiveCodeBench and SPEED-Bench coding manifests
with `max_tokens=512`.

| benchmark | rows | baseline tok/s | DSpark n=3 | DSpark n=7 | Novita Eagle3 n=3 | Novita Eagle3 n=7 | best |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| gsm8k | 6 | 132.0 | 282.2 / 2.14x / 2.937 | 309.1 / 2.34x / 3.659 | 281.7 / 2.13x / 2.941 | 277.3 / 2.10x / 3.595 | DSpark n=7 |
| math500 | 6 | 132.0 | 317.1 / 2.40x / 3.249 | 367.4 / 2.78x / 4.303 | 288.6 / 2.19x / 3.026 | 294.5 / 2.23x / 3.851 | DSpark n=7 |
| aime | 6 | 131.5 | 276.8 / 2.10x / 2.778 | 318.4 / 2.42x / 3.716 | 263.2 / 2.00x / 2.766 | 275.3 / 2.09x / 3.626 | DSpark n=7 |
| humaneval | 6 | 132.1 | 285.1 / 2.16x / 2.875 | 336.6 / 2.55x / 3.953 | 285.9 / 2.17x / 3.029 | 291.8 / 2.21x / 3.850 | DSpark n=7 |
| livecodebench | 121 | 129.8 | 227.5 / 1.75x / 2.306 | 231.0 / 1.78x / 2.696 | 219.5 / 1.69x / 2.342 | 198.5 / 1.52x / 2.593 | DSpark n=7 |
| speedbench_coding | 80 | 131.2 | 282.0 / 2.15x / 2.837 | 303.7 / 2.31x / 3.530 | 272.1 / 2.06x / 2.886 | 281.6 / 2.13x / 3.693 | DSpark n=7 |

Use DSpark with `num_speculative_tokens=7` as the default for code, math, and
most reasoning traffic.

## Serving with vLLM

Requires a vLLM nightly with DSpark support:

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly

vllm serve moonshotai/Kimi-K2.7-Code \
    --tensor-parallel-size 8 \
    --max-model-len 20000 \
    --trust-remote-code \
    --compilation-config='{"pass_config": {"fuse_allreduce_rms": false}}' \
    --speculative-config '{
        "model": "novita/kimi-k2.7-code-dspark",
        "num_speculative_tokens": 7,
        "method": "dspark"
    }'
```

Known vLLM-nightly caveats, with workarounds:

1. Draft-side FA3 AOT scheduling can crash with `scheduler_metadata must have
   shape (metadata_size)` because the GPU-worker spec-decode path misses
   `fast_build=True` when building draft attention metadata. Patch
   `vllm/v1/worker/gpu/spec_decode/speculator.py` and
   `vllm/v1/worker/gpu/attn_utils.py` to pass `fast_build=True`.
2. CUDA-graph capture can fail with a flashinfer allreduce workspace-size error
   under spec-decode token expansion; disable the fusion:
   `--compilation-config='{"pass_config": {"fuse_allreduce_rms": false}}'`.

## Training Details

- **Initialization**: continued from the Kimi-K2.7-Code DSpark exp37 checkpoint
- **Data**: Kimi-K2.7-Code training mix with public Kimi-MTP data and hidden
  states streamed from the live Kimi-K2.7-Code verifier; seq 20000
- **Steps**: 20000 optimizer steps
- **Schedule**: lr 3e-4 cosine, warmup 300, global batch 8, accumulation 2
- **Loss**: 0.1 CE + 0.9 TV over block-diffusion anchors, decay_gamma 4.0,
  max_anchors 3072
- **Semantics**: `apply_verifier_norm=False`, hidden_states = concat of aux
  layers [1, 29, 57]