Instructions to use gasschina/Spark-X2.5-4B-build-cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use gasschina/Spark-X2.5-4B-build-cpp with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0
# Run inference directly in the terminal:
llama cli -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0
# Run inference directly in the terminal:
llama cli -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Use Docker

docker model run hf.co/gasschina/Spark-X2.5-4B-build-cpp:Q8_0

LM Studio
Jan
Ollama
How to use gasschina/Spark-X2.5-4B-build-cpp with Ollama:
```
ollama run hf.co/gasschina/Spark-X2.5-4B-build-cpp:Q8_0
```
Unsloth Desktop

How to use gasschina/Spark-X2.5-4B-build-cpp with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "gasschina/Spark-X2.5-4B-build-cpp:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use gasschina/Spark-X2.5-4B-build-cpp with Docker Model Runner:
```
docker model run hf.co/gasschina/Spark-X2.5-4B-build-cpp:Q8_0
```

Lemonade

How to use gasschina/Spark-X2.5-4B-build-cpp with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Run and chat with the model

lemonade run user.Spark-X2.5-4B-build-cpp-Q8_0

List all available models

lemonade list

Hermes Agent

How to use gasschina/Spark-X2.5-4B-build-cpp with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use gasschina/Spark-X2.5-4B-build-cpp with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf gasschina/Spark-X2.5-4B-build-cpp:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "gasschina/Spark-X2.5-4B-build-cpp:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

gasschina commited on 5 days ago

Commit

e804310

verified ·

1 Parent(s): 2a71748

Upload scripts/oneclick_start.sh with huggingface_hub

Browse files

Files changed (1) hide show

scripts/oneclick_start.sh +23 -11

scripts/oneclick_start.sh CHANGED Viewed

@@ -6,24 +6,27 @@
 # 用法（Colab 任意 cell）:
 #   !bash /content/drive/MyDrive/spark-t4/oneclick_start.sh
 #
-# 默认配置 v3（2026-09-06 投机解码实测后固化，用户指定默认）:
-#   Q8_0 权重 + 1.7B-Q8_0 草稿投机解码 + CTX=262144(256k) + KV q4_0 + fa on + 别名 spark-x2.5-4b
-#   实测: 显存 11.8GB/15GB, 生成 52.3 tok/s（比无草稿 +41%）, 接受率 0.891
-#   上一版(500k 无草稿)仍可用，见下方换配置
 #
 # 换配置（示例）:
-#   长上下文无草稿: DRAFT_MODEL= CTX=500000 !bash .../oneclick_start.sh (实测 37 tok/s, 11.8GB)
-#   极限长上下文:   DRAFT_MODEL= QUANT=Q4_K_M CTX=786432 !bash .../oneclick_start.sh (14.3GB, 52 tok/s)
-#   短上下文高质量KV: DRAFT_MODEL= QUANT=Q8_0 CTX=131072 KVQ=q8_0 !bash .../oneclick_start.sh
 #   （T4 上 1M 上下文物理放不下: 1M KV@q4_0 也需 13GB+，加权重必爆）
 #   （草稿模式下 CTX>300k 会 OOM: 草稿 KV/计算缓冲继承主上下文）
 # ============================================================================
 set -euo pipefail
 BASE_DIR="/content/drive/MyDrive/spark-t4"
 MODEL_DIR="/content/spark-t4-models"     # 模型永不进 Drive（会话重置后自动重下）
 export BASE_DIR MODEL_DIR
-export QUANT="${QUANT:-Q8_0}"
 export CTX="${CTX:-262144}"                # 256k: 草稿投机解码最优档; 无草稿可设 500000
 export PORT="${PORT:-8100}"              # 8080 被 Colab 环境占用
 export NP="${NP:-1}"
@@ -31,9 +34,16 @@ export KVQ="${KVQ:-q4_0}"
 export BATCH="${BATCH:-512}"
 export UBATCH="${UBATCH:-512}"
 export EXTRA_ARGS="${EXTRA_ARGS:--np 1 -b 512 -ub 512 -fa on --alias spark-x2.5-4b}"
-export SPEC_TYPE="${SPEC_TYPE:-draft-simple}"
 export DRAFT_MODEL="${DRAFT_MODEL:-/content/spark-t4-models/Spark-X2.5-1.7B-Q8_0.gguf}"
 export DRAFT_NMAX="${DRAFT_NMAX:-8}"
 export BUILD_JOBS="${BUILD_JOBS:-2}"
 TUNNEL="${TUNNEL:-1}"                    # 1=自动开公网隧道
@@ -98,10 +108,12 @@ if curl -s --max-time 60 "http://127.0.0.1:$PORT/v1/chat/completions" \
     echo -e "  本机:  http://127.0.0.1:$PORT/v1/chat/completions"
     [[ -n "$KEY" ]] && echo -e "  Key:   $KEY"
     echo -e "  上下文: $CTX tokens | 权重: $QUANT | KV: ${KVQ:-f16}"
-    [[ -n "$DRAFT_MODEL" ]] && echo -e "  投机解码: 1.7B 草稿 draft-simple (实测 +41% ≈52 tok/s, 接受率 0.89)"
     [[ "$TUNNEL" == "1" ]] && echo -e "  公网:  https://aitun.cc/$(grep -oE 'aitun\.cc/[A-Za-z0-9]+' /content/aitun-api-tunnel.log 2>/dev/null | tail -1 | cut -d/ -f2)/v1/chat/completions"
     echo -e "  流式: 请求体加 \"stream\": true"
-    echo -e "  关思考: 请求体加 \"chat_template_kwargs\":{\"enable_thinking\":false}"
     echo -e "${CYAN}════════════════════════════════════════════════${NC}"
 else
     fail "自检未通过，查看: tail -30 $BASE_DIR/api-server.log"

 # 用法（Colab 任意 cell）:
 #   !bash /content/drive/MyDrive/spark-t4/oneclick_start.sh
 #
+# 默认配置 v5（2026-09-06, 用户指定 4bit+256k; 草稿经实测从 4bit 档移除）:
+#   Q4_K_M 权重裸跑 + CTX=262144(256k) + KV q4_0 + fa on + 别名 spark-x2.5-4b + 思考默认关
+#   实测: 显存 6.6GB/15GB(!余量8.8GB), 生成 48.75 tok/s, 预填充 790 tok/s
+#   ⚠️ 4bit 不挂草稿: 4bit 量化噪声→草稿接受率 0.89→0.37-0.61, 实测反而 27 tok/s(慢一半)
+#   草稿留给 Q8_0 档: QUANT=Q8_0 自动挂 1.7B 草稿(52.3 tok/s, 11.8GB, 接受率 0.89)
 #
 # 换配置（示例）:
+#   高质量+草稿: QUANT=Q8_0 !bash .../oneclick_start.sh (草稿@256k, 52.3 tok/s, 11.8GB)
+#   4bit 长上下文: CTX=500000 !bash .../oneclick_start.sh (实测 51.7 tok/s, 10.1GB, 草稿同样自动关)
+#   极限长上下文: QUANT=Q4_K_M CTX=786432 !bash .../oneclick_start.sh (14.3GB)
+#   强制开草稿(不建议 4bit): SPEC_TYPE=draft-simple !bash .../oneclick_start.sh
 #   （T4 上 1M 上下文物理放不下: 1M KV@q4_0 也需 13GB+，加权重必爆）
 #   （草稿模式下 CTX>300k 会 OOM: 草稿 KV/计算缓冲继承主上下文）
+#   （思考模式默认关: aitun 隧道非流式请求 120s 硬超时, 开思考会 504; THINKING=on 恢复）
 # ============================================================================
 set -euo pipefail
 BASE_DIR="/content/drive/MyDrive/spark-t4"
 MODEL_DIR="/content/spark-t4-models"     # 模型永不进 Drive（会话重置后自动重下）
 export BASE_DIR MODEL_DIR
+export QUANT="${QUANT:-Q4_K_M}"           # 默认 4bit: 实际使用易超时→提速+省显存(用户指定 2026-09-06)
 export CTX="${CTX:-262144}"                # 256k: 草稿投机解码最优档; 无草稿可设 500000
 export PORT="${PORT:-8100}"              # 8080 被 Colab 环境占用
 export NP="${NP:-1}"
 export BATCH="${BATCH:-512}"
 export UBATCH="${UBATCH:-512}"
 export EXTRA_ARGS="${EXTRA_ARGS:--np 1 -b 512 -ub 512 -fa on --alias spark-x2.5-4b}"
+export SPEC_TYPE_DEFAULT=1
+# 草稿策略(按量化档自动): Q8_0 挂草稿(接受率0.89,+41%); 4bit 裸跑(接受率0.37-0.61,草稿反而慢一半)
+if [[ "$QUANT" == Q8* ]]; then
+    export SPEC_TYPE="${SPEC_TYPE:-draft-simple}"
+else
+    export SPEC_TYPE="${SPEC_TYPE:-none}"
+fi
 export DRAFT_MODEL="${DRAFT_MODEL:-/content/spark-t4-models/Spark-X2.5-1.7B-Q8_0.gguf}"
 export DRAFT_NMAX="${DRAFT_NMAX:-8}"
+export THINKING="${THINKING:-off}"       # 思考默认关: aitun 隧道非流式 120s 硬超时→504防护; THINKING=on 恢复
 export BUILD_JOBS="${BUILD_JOBS:-2}"
 TUNNEL="${TUNNEL:-1}"                    # 1=自动开公网隧道
     echo -e "  本机:  http://127.0.0.1:$PORT/v1/chat/completions"
     [[ -n "$KEY" ]] && echo -e "  Key:   $KEY"
     echo -e "  上下文: $CTX tokens | 权重: $QUANT | KV: ${KVQ:-f16}"
+    [[ -n "$DRAFT_MODEL" && "$SPEC_TYPE" != "none" ]] && echo -e "  投机解码: 1.7B 草稿 draft-simple (Q8_0 实测 +41% ≈52 tok/s, 接受率 0.89)"
+    [[ "$SPEC_TYPE" == "none" ]] && echo -e "  投机解码: 关闭(4bit 裸跑 48.75 tok/s; 草稿仅对 Q8_0 有正收益)"
+    echo -e "  思考模式: $([[ "$THINKING" == "off" ]] && echo '默认关闭(网关504防护, 单请求可传 enable_thinking:true)' || echo '默认开启')"
     [[ "$TUNNEL" == "1" ]] && echo -e "  公网:  https://aitun.cc/$(grep -oE 'aitun\.cc/[A-Za-z0-9]+' /content/aitun-api-tunnel.log 2>/dev/null | tail -1 | cut -d/ -f2)/v1/chat/completions"
     echo -e "  流式: 请求体加 \"stream\": true"
+    echo -e "  开思考(单请求): 请求体加 \"chat_template_kwargs\":{\"enable_thinking\":true}"
     echo -e "${CYAN}════════════════════════════════════════════════${NC}"
 else
     fail "自检未通过，查看: tail -30 $BASE_DIR/api-server.log"