# MTP draft heads for Qwen3.8-Flash-Next

An MTP head guesses the next few tokens; the main model verifies them in one pass. Verification is
exact, so the output is unchanged - only the speed. Worth about **1.3x to 1.7x** at low concurrency.

## Which file

**Use `mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` (2.60 GB).** Fastest of the set.

| file | size | |
|---|---|---|
| `shared-Q8_0` | 2.60 GB | **recommended** |
| `shared-Q4_K_M` | 1.78 GB | smaller, ~2 points less acceptance |
| `shared-BF16` | 4.87 GB | bigger *and* slower than Q8_0 |
| `Q8_0` / `Q4_K_M` / `BF16` | 3.85 / 2.60 / 7.24 GB | self-contained variants |

`shared-` heads borrow the token embedding and output projection from the model you are already
running, saving about 1.3 GB. They draft identically to the self-contained files, which carry their
own copies and are only needed on builds without borrowing support.

BF16 is bigger and slower: a draft step is dominated by the output projection, which is cheaper to
execute at 8 bits.

## Requirements

**A stock `ggml-org/llama.cpp` build cannot use these.** Mainline has no MTP graph for the
`qwen4exp` architecture, no cross-model tensor borrowing, and no `--spec-type draft-mtp` option, so
passing a head to it does nothing. Pick one of the three below.

### Option 1: prebuilt binaries (easiest)

From [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases), tag
`b10715-mix-86bd2d3` or newer. Assets are `app-<tag>-<os>-<arch>-<backend>.tar.gz` (`.zip` on
Windows) for CPU, CUDA 12/13, ROCm and Vulkan.

### Option 2: build the upstream pull request

[ggml-org/llama.cpp#28243](https://github.com/ggml-org/llama.cpp/pull/28243) is the MTP support
going to mainline. Building it:

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch qwen4exp/mtp https://github.com/danielhanchen/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

Omit `-DGGML_CUDA=ON` for a CPU build. To track the pull request rather than the branch, which keeps
working if the branch is renamed or deleted, replace the clone with:

```bash
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin refs/pull/28243/head
git -C llama.cpp checkout FETCH_HEAD
```

Then fetch a head:

```bash
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-Flash-Next-GGUF \
    --local-dir unsloth/Qwen3.8-Flash-Next-GGUF \
    --include "*mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf*"
```

It lands in an `MTP/` subfolder, so the path to pass is
`unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`:

```bash
llama.cpp/llama-server \
    -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
    -md unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
    --spec-type draft-mtp --spec-draft-n-max 2
```

### Option 3: build the unsloth fork

[unslothai/llama.cpp#144](https://github.com/unslothai/llama.cpp/pull/144):

```bash
git clone https://github.com/unslothai/llama.cpp && cd llama.cpp
git fetch origin pull/144/head:mtp && git checkout mtp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON   # omit -DGGML_CUDA for CPU
cmake --build build -j
```

## Usage

```bash
llama-cli \
  -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 -ngl 999
```

Same flags for `llama-server`. `--spec-draft-n-max 2` is a good default; higher drafts more but
each guess is accepted less often.

**Always pass `-md` explicitly.** The heads live in an `MTP/` subfolder, which sidecar
auto-discovery does not search, so `--spec-type draft-mtp` on its own finds nothing and you get the
main model's speed with no error saying why.

**A `shared-` head logs one error line at startup and then works.** The automatic memory fit sizes
the draft by loading it on its own, before the main model exists, so there is nothing for it to
borrow from and the measurement fails:

```
E llama_model_load: error loading model: borrow_shared_tensor: this model is a draft head ...
W operator(): failed to measure the memory of the extra model, fitting without it
```

Speculation still runs. The only real consequence is that the fit does not count the draft's
memory, so on a card with little headroom it may choose a context size that does not fit. Pass
`-c` and `-ngl` yourself, or use a self-contained head, if that matters to you. Self-contained
heads measure cleanly and produce no such line.

To confirm it is running, look for this in the log. If it never appears, speculation is off and you
are probably on a build without MTP support:

```
draft acceptance = 0.66139 (325 accepted / 491 generated), mean len = 2.76
```

## Measured

Single stream, greedy, one B200, `shared-Q8_0`:

| main model | off | on | |
|---|---|---|---|
| `UD-Q4_K_XL` | 83.2 tok/s | **138.8 tok/s** | **1.67x** |
| `UD-IQ1_S` | 90.1 tok/s | **120.9 tok/s** | **1.34x** |

Acceptance: `shared-BF16` 66.5%, `shared-Q8_0` 66.1%, `shared-Q4_K_M` 64.4%.

**These are greedy numbers.** Higher temperature makes the target less predictable, so fewer
guesses are accepted and the speedup shrinks. A speculative decoding figure means nothing without
the sampler it was measured with.

## When not to use it

**Skip MTP for concurrent serving.** A win at concurrency 1, but measured as a net loss (~0.81x to
0.87x) at concurrency 8: a busy model has little idle capacity for a draft to exploit.

These heads are for Qwen3.8-Flash-Next only; a mismatched pairing is rejected with an error rather
than producing bad output. Loading a head on its own, without a main model, is also rejected.