Upload README.md with huggingface_hub

README.md

@@ -20,19 +20,19 @@ tags:
 
 Everything needed to run **iFlytek Spark-X2.5-4B** on an NVIDIA T4 (16GB) with llama.cpp:
 a prebuilt CUDA binary (XHToken fork — the `spark2_5` architecture is **not** supported by mainline llama.cpp),
-Q8_0 / Q4_K_M model weights, a **1.7B draft model for speculative decoding** (+41% throughput, 0.89 acceptance rate),
+Q8_0 / Q4_K_M model weights, a **1.7B draft model** (boosts the Q8_0 config by +41%, 0.89 acceptance),
 and battle-tested one-click scripts (256k-context OpenAI-compatible API + public tunnel).
 
-> Verified end-to-end on 2026-09-06: streaming, native tool calling (`--jinja`), thinking mode,
-> and a real agent workload (84-tool zagent, 12 consecutive tool calls, 240k prompt tokens, zero failures).
+> Verified end-to-end on 2026-09-06: streaming, native tool calling (`--jinja`), and a real agent
+> workload (84-tool zagent, 12 consecutive tool calls, 240k prompt tokens, zero failures).
 
 ## What's in this repo
 
 | File | Size | Description |
 |------|------|-------------|
 | `llama.cpp-spark-build-t4-cuda.tar.gz` | 242MB | Prebuilt llama.cpp (XHToken fork) for CUDA / SM 7.5: `llama-server` + all runtime libs |
-| `Spark-X2.5-4B-Q8_0.gguf` | 4.37GB | 8-bit quant (near-F16 quality) — **default** |
-| `Spark-X2.5-4B-Q4_K_M.gguf` | 2.60GB | 4-bit quant for extreme long context |
+| `Spark-X2.5-4B-Q8_0.gguf` | 4.37GB | 8-bit quant (near-F16 quality) — pairs best with the draft model |
+| `Spark-X2.5-4B-Q4_K_M.gguf` | 2.60GB | 4-bit quant — **default**: fastest, most VRAM headroom |
 | `Spark-X2.5-1.7B-Q8_0.gguf` | 1.82GB | Draft model (same tokenizer, same `spark2_5` arch) for speculative decoding |
 | `scripts/oneclick_start.sh` | – | One-click: checks → serve → tunnel → self-test |
 | `scripts/spark_api.sh` | – | OpenAI-compatible API manager (start / stop / status / test) |
@@ -64,29 +64,31 @@ The launcher prints the local endpoint, public tunnel URL, and API key when done
 Model weights live on Colab local disk (`/content/spark-t4-models`), never eating into the
 15GB Drive quota; the build and scripts persist on Drive across sessions.
 
-## Default config (v3): Q8_0 + 1.7B speculative decoding @ 256k
+## Default config (v5): Q4_K_M bare @ 256k, tuned for agent platforms
 
 | Item | Value | Notes |
 |------|-------|-------|
-| Weights | Q8_0 (4.37GB) | near-F16 quality |
-| Speculative decoding | 1.7B-Q8_0 draft, `draft-simple` | **acceptance 0.891, 52.3 tok/s (+41%)** |
-| Context | **262,144 (256k)** | draft KV/compute buffers inherit main ctx; >300k OOMs |
-| KV quant | q4_0 (main + draft) | via `-ctk/-ctv` |
-| VRAM | **11.8GB / 15.36GB** | ~3.5GB headroom |
-| Throughput | **52.3 tok/s** | vs 37.1 tok/s without draft (same weights) |
-| Attention | `-fa on` + `-b 512 -ub 512` | Turing requires flash attention for KV quant |
+| Weights | Q4_K_M (2.60GB) | 4-bit — fastest, most headroom |
+| Speculative decoding | **off** | ⚠️ 4-bit quantization noise drops draft acceptance 0.89 → 0.37-0.61; measured **27 tok/s (slower than bare!)**. Drafting only pays off on Q8_0 |
+| Context | **262,144 (256k)** | 500k available at the same speed (51.7 tok/s / 10.1GB) |
+| KV quant | q4_0 | via `-ctk/-ctv` |
+| VRAM | **6.6GB / 15.36GB** | 8.8GB headroom |
+| Throughput | **48.75 tok/s** | prefill 790 tok/s; Q8_0+draft does 52.3 tok/s |
+| Thinking mode | **server-side off by default** | ⚠️ the aitun tunnel hard-kills silent non-streaming requests at **120s → 504**; thinking adds 30–90s of silence. Re-enable per request with `"chat_template_kwargs": {"enable_thinking": true}` |
 | Auth | auto-generated key → Drive `.api_key` | `API_KEY=off` to disable |
-| Thinking mode | on by default | see API section to disable |
 
-Speculative decoding is **on by default** in the scripts; the draft model is auto-downloaded
-from this repo if missing. Turn it off for long-context mode:
+Draft policy is automatic by quant tier: `QUANT=Q8_0` attaches the 1.7B draft (acceptance 0.89,
+**52.3 tok/s, 11.8GB**); 4-bit runs bare. Override with `SPEC_TYPE=draft-simple` / `SPEC_TYPE=none`.
 
 ```bash
-DRAFT_MODEL= CTX=500000 bash oneclick_start.sh                  # 500k long-context: 37 tok/s, 11.8GB
-DRAFT_MODEL= QUANT=Q4_K_M CTX=786432 bash oneclick_start.sh     # 768k extreme: 14.3GB, 52 tok/s
+QUANT=Q8_0 bash oneclick_start.sh      # quality + draft: 52.3 tok/s, 11.8GB
+CTX=500000 bash oneclick_start.sh      # 500k long-context, same speed, 10.1GB
+QUANT=Q4_K_M CTX=786432 bash oneclick_start.sh   # 768k extreme, 14.3GB
 ```
 
 > 1M context physically does not fit on a T4: 1M KV@q4_0 alone needs ~13GB before weights.
+> Single slot (`-np 1`): concurrent requests queue behind long agent replies. On agent platforms
+> (aicq.me etc.) prefer streaming and keep `max_tokens` ≤ 2000.
 
 ## Benchmarks (T4, greedy, 384-token generations, /completion timings)
 
@@ -94,14 +96,16 @@ DRAFT_MODEL= QUANT=Q4_K_M CTX=786432 bash oneclick_start.sh     # 768k extreme:
 |--------|---------|------|-----------|
 | Q8_0, no draft (baseline) | 500k | 11.8GB | 37.1 tok/s |
 | **Q8_0 + 1.7B-Q8_0 draft** | **256k** | **11.8GB** | **52.3 tok/s (+41%)** |
-| Q8_0 + draft, n_max=16 | 256k | 11.8GB | 52.6 tok/s (no gain over 8) |
+| Q4_K_M, no draft (v5 default) | 256k | 6.6GB | 48.75 tok/s |
 | Q4_K_M, no draft | 500k | 10.1GB | 51.7 tok/s |
+| Q4_K_M + draft | 256k | ~10.1GB | **27.2 tok/s (anti-optimization! acceptance 0.37-0.61)** |
 | Q4_K_M + draft | 500k | OOM | draft compute buffer ~1.5GB |
 | ngram-simple (self-speculation) | 500k | +0GB | 37.5 tok/s (no gain on prose) |
 
-Takeaway: the T4 is memory-bandwidth-bound. Q8_0 + draft reaches bare-Q4_K_M speed while keeping
-the exact Q8_0 output distribution (drafting never changes the target distribution). If you need
-500k context, drop the draft; if you want raw speed without drafting, use Q4_K_M.
+Takeaway: the T4 is memory-bandwidth-bound. Drafting is a *distribution-matching* game — with a
+near-lossless Q8_0 target the 1.7B sibling hits 0.89 acceptance (+41%), but a 4-bit target's
+argmax flips too often and drafting goes net-negative. Gateway note: the free aitun tunnel
+terminates non-streaming requests at exactly **120s** (reproduced); streaming is immune.
 
 ## MTP status (why classic drafting)
 
@@ -109,8 +113,8 @@ llama.cpp already ships `draft-mtp` / `draft-eagle3` / `draft-dspark` speculativ
 require matching trained heads. Spark-X2.5-4B publishes no MTP/nextn/EAGLE weights
 (`num_nextn_predict_layers` is absent from its config), and the fork's `spark2_5` backend has no
 nextn implementation. Classic draft-model speculation with the official same-vocabulary (131,072)
-1.7B sibling is the best available option — and it works well: **0.891 acceptance, mean accepted
-length 8.13 tokens**.
+1.7B sibling is the best available option: **0.891 acceptance, mean accepted length 8.13 tokens**
+on the Q8_0 target.
 
 ## API usage (OpenAI-compatible)
 
@@ -127,8 +131,8 @@ r = requests.post(f"{BASE}/v1/chat/completions",
 print(r.json()["choices"][0]["message"]["content"])
 ```
 
-- **Streaming**: add `"stream": true` (SSE)
-- **Disable thinking** (faster, no reasoning block): add `"chat_template_kwargs": {"enable_thinking": false}`
+- **Streaming**: add `"stream": true` (SSE) — strongly recommended behind tunnels/gateways
+- **Thinking mode** (server-side off by default): add `"chat_template_kwargs": {"enable_thinking": true}`
 - **Tool calling**: pass a standard OpenAI `tools` array (`--jinja` enables native function calling)
 
 ## Why this build exists
@@ -140,4 +144,4 @@ community i1 quant; the 1.7B draft quantized in-house (Q8_0) from the official F
 
 ## 中文速览
 
-讯飞星火 Spark-X2.5-4B 在 Colab T4 上的完整 llama.cpp 运行包：CUDA 编译产物 + Q8_0/Q4_K_M 权重 + 1.7B 草稿模型 + 一键脚本。默认配置 = Q8_0 + 1.7B 草稿投机解码 @256k 上下文（实测 52.3 tok/s，+41%，接受率 0.891，显存 11.8GB）；需要 500k 长上下文时用 `DRAFT_MODEL= CTX=500000` 关闭草稿。三分钟起步见上方 Quick start；API 为 OpenAI 兼容格式（/v1/chat/completions），模型名填 `spark-x2.5-4b`。
+讯飞星火 Spark-X2.5-4B 在 Colab T4 上的完整 llama.cpp 运行包。**默认配置 v5 = Q4_K_M 裸跑 @256k 上下文 + 思考默认关**（实测 48.75 tok/s，显存仅 6.6GB）——为 aicq.me 等智能体平台调优：aitun 隧道对非流式请求有 120s 硬超时（开思考必 504），故思考默认关闭、单请求可传 `enable_thinking:true` 恢复；草稿投机解码只对 Q8_0 有正收益（+41%，接受率 0.89），`QUANT=Q8_0` 时自动挂载，4bit 档实测反而慢一半已自动关闭。API 为 OpenAI 兼容格式，模型名 `spark-x2.5-4b`，智能体平台建议开流式、max_tokens ≤2000。