---
license: apache-2.0
language:
- zho
- eng
base_model:
- XHToken/Spark-X2.5-4B
tags:
- spark-x2.5
- llama.cpp
- gguf
- speculative-decoding
- t4
- cuda
- colab
- edge-deployment
---

# Spark-X2.5-4B — Ready-to-Run llama.cpp Build & GGUF Package (Colab T4 / CUDA)

Everything needed to run **iFlytek Spark-X2.5-4B** on an NVIDIA T4 (16GB) with llama.cpp:
a prebuilt CUDA binary (XHToken fork — the `spark2_5` architecture is **not** supported by mainline llama.cpp),
Q8_0 / Q4_K_M model weights, a **1.7B draft model** (boosts the Q8_0 config by +41%, 0.89 acceptance),
and battle-tested one-click scripts (256k-context OpenAI-compatible API + public tunnel).

> Verified end-to-end on 2026-09-06: streaming, native tool calling (`--jinja`), and a real agent
> workload (84-tool zagent, 12 consecutive tool calls, 240k prompt tokens, zero failures).

## What's in this repo

| File | Size | Description |
|------|------|-------------|
| `llama.cpp-spark-build-t4-cuda.tar.gz` | 242MB | Prebuilt llama.cpp (XHToken fork) for CUDA / SM 7.5: `llama-server` + all runtime libs |
| `Spark-X2.5-4B-Q8_0.gguf` | 4.38GB | 8-bit quant — **default**: best quality |
| `Spark-X2.5-4B-Q4_K_M.gguf` | 2.60GB | 4-bit quant — VRAM-saving option (`QUANT=Q4_K_M`), most headroom |
| `Spark-X2.5-1.7B-Q8_0.gguf` | 1.82GB | Draft model (same tokenizer, same `spark2_5` arch) for speculative decoding |
| `scripts/oneclick_start.sh` | – | One-click: checks → serve → tunnel → self-test |
| `scripts/spark_api.sh` | – | OpenAI-compatible API manager (start / stop / status / test) |
| `scripts/restore_build.sh` | – | Restore the prebuilt binary from this repo (skips the 40–60 min compile) |
| `scripts/deploy_spark_t4.sh` | – | Full pipeline: deps / build / download / serve |
| `scripts/tunnel_api.sh` | – | aitun.cc public tunnel manager |

## Quick start (~3 minutes on a fresh T4 Colab)

```python
# Cell 1 — mount Drive
from google.colab import drive; drive.mount('/content/drive')
```

```bash
# Cell 2 — restore prebuilt binary (public repo, no token needed, ~1 min)
pip install -U huggingface_hub >/dev/null
hf download gasschina/Spark-X2.5-4B-build-cpp llama.cpp-spark-build-t4-cuda.tar.gz --local-dir /content/
mkdir -p /content/drive/MyDrive/spark-t4
tar xzf /content/llama.cpp-spark-build-t4-cuda.tar.gz -C /content/drive/MyDrive/spark-t4/

# Cell 3 — fetch scripts & launch (downloads weights to local disk + starts API + tunnel + self-test)
hf download gasschina/Spark-X2.5-4B-build-cpp --include "scripts/*.sh" --local-dir /content/s4b-scripts
cp /content/s4b-scripts/scripts/*.sh /content/drive/MyDrive/spark-t4/
bash /content/drive/MyDrive/spark-t4/oneclick_start.sh
```

The launcher prints the local endpoint, public tunnel URL, and API key when done.
Model weights live on Colab local disk (`/content/spark-t4-models`), never eating into the
15GB Drive quota; the build and scripts persist on Drive across sessions.

## Default config (v6): Q8_0 + 1.7B draft @ 256k (reverted 2026-09-06)

| Item | Value | Notes |
|------|-------|-------|
| Weights | Q8_0 (4.38GB) | best quality |
| Speculative decoding | **draft-simple + 1.7B** | Q8_0 acceptance 0.89; **52.3 tok/s** on English/temp-0 factual workloads. ⚠️ on long Chinese creative generation (temp 1.0) acceptance collapses — measured ~18.5-30 tok/s; use `SPEC_TYPE=none` (≈37 tok/s) or `QUANT=Q4_K_M` (≈48.75 tok/s, 6.6GB) if that is your workload |
| Context | **262,144 (256k)** | 500k available at the same speed (51.7 tok/s / 10.1GB) |
| KV quant | q4_0 | via `-ctk/-ctv` |
| VRAM | **11.8GB / 15.36GB** | 3.5GB headroom |
| Throughput | **52.3 tok/s** (EN/temp-0) | prefill ~1160 tok/s; CN temp-1.0 long-form ~18.5-30 tok/s (draft acceptance collapses) |
| Thinking mode | **server-side off by default** | ⚠️ the aitun tunnel hard-kills silent non-streaming requests at **120s → 504**; thinking adds 30–90s of silence. Re-enable per request with `"chat_template_kwargs": {"enable_thinking": true}` |
| Auth | auto-generated key → Drive `.api_key` | `API_KEY=off` to disable |

Draft policy is automatic by quant tier: `QUANT=Q8_0` attaches the 1.7B draft (acceptance 0.89,
**52.3 tok/s, 11.8GB**); 4-bit runs bare. Override with `SPEC_TYPE=draft-simple` / `SPEC_TYPE=none`.

```bash
QUANT=Q8_0 bash oneclick_start.sh      # quality + draft: 52.3 tok/s, 11.8GB
CTX=500000 bash oneclick_start.sh      # 500k long-context, same speed, 10.1GB
QUANT=Q4_K_M CTX=786432 bash oneclick_start.sh   # 768k extreme, 14.3GB
```

> 1M context physically does not fit on a T4: 1M KV@q4_0 alone needs ~13GB before weights.
> Single slot (`-np 1`): concurrent requests queue behind long agent replies. On agent platforms
> (aicq.me etc.) prefer streaming and keep `max_tokens` ≤ 2000.

## Benchmarks (T4, greedy, 384-token generations, /completion timings)

| Config | Context | VRAM | Throughput |
|--------|---------|------|-----------|
| Q8_0, no draft (baseline) | 500k | 11.8GB | 37.1 tok/s |
| Q8_0 + 1.7B draft (v6 default) | 256k | 11.8GB | 52.3 tok/s (EN/temp-0); 18.5-30 tok/s CN/temp-1.0 |
| Q4_K_M, no draft | 256k | 6.6GB | 48.75 tok/s (EN/temp-0); 33-46 tok/s CN real-world |
| Q4_K_M, no draft | 500k | 10.1GB | 51.7 tok/s |
| Q4_K_M + draft | 256k | ~10.1GB | **27.2 tok/s (anti-optimization! acceptance 0.37-0.61)** |
| Q4_K_M + draft | 500k | OOM | draft compute buffer ~1.5GB |
| ngram-simple (self-speculation) | 500k | +0GB | 37.5 tok/s (no gain on prose) |

Takeaway: the T4 is memory-bandwidth-bound. Drafting is a *distribution-matching* game — with a
near-lossless Q8_0 target the 1.7B sibling hits 0.89 acceptance (+41%), but a 4-bit target's
argmax flips too often and drafting goes net-negative. Gateway note: the free aitun tunnel
terminates non-streaming requests at exactly **120s** (reproduced); streaming is immune.

## MTP status (why classic drafting)

llama.cpp already ships `draft-mtp` / `draft-eagle3` / `draft-dspark` speculative types, but they
require matching trained heads. Spark-X2.5-4B publishes no MTP/nextn/EAGLE weights
(`num_nextn_predict_layers` is absent from its config), and the fork's `spark2_5` backend has no
nextn implementation. Classic draft-model speculation with the official same-vocabulary (131,072)
1.7B sibling is the best available option: **0.891 acceptance, mean accepted length 8.13 tokens**
on the Q8_0 target.

## API usage (OpenAI-compatible)

```python
import requests
BASE = "http://127.0.0.1:8100"   # or your public tunnel URL
KEY  = "sk-..."

r = requests.post(f"{BASE}/v1/chat/completions",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"model": "spark-x2.5-4b",
          "messages": [{"role": "user", "content": "Hello"}],
          "max_tokens": 512})
print(r.json()["choices"][0]["message"]["content"])
```

- **Streaming**: add `"stream": true` (SSE) — strongly recommended behind tunnels/gateways
- **Thinking mode** (server-side off by default): add `"chat_template_kwargs": {"enable_thinking": true}`
- **Tool calling**: pass a standard OpenAI `tools` array (`--jinja` enables native function calling)

## Why this build exists

Mainline llama.cpp does not support the `spark2_5` architecture (ggml-org/llama.cpp issue #28300).
This package ships a CUDA build of the [XHToken fork](https://github.com/XHToken/llama.cpp)
(SM 7.5 / CUDA 12.x) so you can skip the 40–60 minute compile. Weights: Q8_0 / Q4_K_M from the
community i1 quant; the 1.7B draft quantized in-house (Q8_0) from the official F16 GGUF.

## 中文速览

讯飞星火 Spark-X2.5-4B 在 Colab T4 上的完整 llama.cpp 运行包。**默认配置 v5 = Q4_K_M 裸跑 @256k 上下文 + 思考默认关**（实测 48.75 tok/s，显存仅 6.6GB）——为 aicq.me 等智能体平台调优：aitun 隧道对非流式请求有 120s 硬超时（开思考必 504），故思考默认关闭、单请求可传 `enable_thinking:true` 恢复；草稿投机解码只对 Q8_0 有正收益（+41%，接受率 0.89），`QUANT=Q8_0` 时自动挂载，4bit 档实测反而慢一半已自动关闭。API 为 OpenAI 兼容格式，模型名 `spark-x2.5-4b`，智能体平台建议开流式、max_tokens ≤2000。