Instructions to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
llama cli -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16
# Run inference directly in the terminal:
./build/bin/llama cli -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Use Docker

docker model run hf.co/AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Llama new
LM Studio
Jan
Ollama
How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with Ollama:
```
ollama run hf.co/AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16
```
Unsloth Desktop

How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16
```

Lemonade

How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Run and chat with the model

lemonade run user.Qwen3.5-4B-AX8860-llama.cpp-BF16

List all available models

lemonade list

Hermes Agent

How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "AXERA-TECH/Qwen3.5-4B-AX8860-llama.cpp:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.5-4B on AX8860 (llama.cpp + ggml-axera)

在爱芯 AX8860 的 NPU 上用 llama.cpp 运行 Qwen3.5-4B：文本 + 图像输入，可选 MTP 投机解码，OpenAI 兼容服务。 Qwen3.5-4B on the Axera AX8860 NPU with llama.cpp: text + image input, optional MTP speculative decoding, OpenAI-compatible server.

性能 / Performance


decode	62.3 tokens/s
decode，MTP 投机解码 / with MTP	最高 / up to 128.0 tokens/s（2.05x）
4k prompt，MTP / MTP on 4k	58.8 → 99.7 tokens/s（1.70x，accepted 42/61）
4k prompt prefill	5040 tokens/s（3908 token 0.78 s）
上下文 / context	最长 / up to 16384

MTP（MTP=1）按内容 / by content:

内容 / content	普通 / plain	MTP	加速 / speed-up	接受率 / accept
代码 / code	62.4	84.7	1.36x	0.481
中译英 / translation	60.5	84.9	1.40x	0.598
Markdown 表格 / table	62.5	128.0	2.05x	0.816
数学解题 / maths	62.6	98.5	1.57x	0.563
中文短文 / Chinese essay	61.6	62.7	1.02x	0.274
英文故事 / English story	61.2	63.3	1.03x	0.302
代码，推荐采样 / code, sampled	61.3	87.4	1.43x	0.481
中文短文，推荐采样 / essay, sampled	60.6	66.2	1.09x	0.269

单位 tokens/s；每项最多生成 256 token，未注明的为贪心解码 / tokens/s; up to 256 tokens each, greedy unless noted.

sh bench.sh 复现 / reproduce（生成 128 token / 128 generated tokens；短 prompt 仅列耗时 / short prompts report time only）：

prompt	`-c`	prompt 耗时 / time	decode tokens/s
5 token	8192	0.08 s	—
47 token	8192	0.09 s	—
3908 token	8192	0.78 s	59.8
5 token	16384	0.09 s	—
47 token	16384	0.09 s	—
3908 token	16384	0.77 s	59.9
3908 token，推荐采样 / sampled	8192	0.78 s	58.6

快速开始 / Quick start

sh run.sh "The capital of France is" 64           # 文本 / text
sh run.sh -i bus.jpg "Describe this image." 256   # 图像 / image
sh run.sh -c                                      # 命令行对话 / chat
sh run.sh -s 8080                                 # OpenAI 兼容服务 / server
MTP=1 sh run.sh -s 8080                           # MTP 投机解码 / speculative decoding
sh bench.sh                                       # 性能复现 / benchmark

需要 AX8860，SDK V0.5.1 或更新；HF 下载的文件没有执行权限，用 sh 调用即可 / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.

文件 / Files

文件 / File	说明 / Description
`qwen3.5-4b-mxfp4.gguf`	模型权重 / weights，2.66 GB，md5 `e82f054c77152821eb8edd5ae572f2eb`
`mmproj-qwen3.5-4b-bf16.gguf`	视觉编码器 / vision encoder，676 MB，md5 `d64c1da66e6bd59ecbec5d57ccc13ec7`
`bin/`	llama.cpp 板端程序（静态链接）/ board programs (static)
`kernels/`	NPU kernel + `manifest.json`
`run.sh`，`chat_template.jinja`	运行脚本与服务端对话模板 / run script and the server's chat template
`bench.sh` 及 prompt	性能复现 / benchmark

说明 / Notes

MTP 为可选：MTP=1 ./run.sh -s 显式开启，服务端不再用自适应门控把它关掉；不设置 MTP 时保持普通解码。贪心请求请显式带 presence_penalty: 0 / MTP is opt-in: MTP=1 ./run.sh -s enables it explicitly and bypasses the adaptive gate; without MTP the server stays non-speculative. Greedy requests should set presence_penalty: 0.
服务端默认使用模型卡推荐的采样参数，文本补全与图像描述为贪心解码 / the server uses the model card's recommended sampling; completion and image description are greedy.
服务端默认关闭思考（--reasoning off），并允许对话中段的 system 消息（Codex 等 agent 客户端可用）/ the server does not think by default and accepts a mid-conversation system message (works with agent clients such as Codex).
Prompt cache 默认放在 4 GB 内存；设置 CACHE_DIR=<SSD 目录> 可落盘，重启后仍能命中 / the prompt cache lives in 4 GB of host memory; set CACHE_DIR=<an SSD directory> to keep it on disk across restarts.
运行时独占 NPU 4 核；服务端 -np 1 / takes all 4 NPU cores; the server runs -np 1.
llama-bench 可用，请加 -b 4096 -ub 4096 / llama-bench works; pass -b 4096 -ub 4096.
个别环境下连续请求偶发 ret = -3，重试即可 / in rare cases a back-to-back request fails with ret = -3; retry it.
MTP 每轮猜 3 个 token，由目标模型逐个验证；贪心时输出与普通解码一致 / MTP drafts 3 tokens per round, each verified by the target model; greedy output equals plain decoding.

正确性 / Correctness

"The capital of France is" → "Paris. …"；图像描述正确（车辆、文字、人物）/ text and image descriptions correct.
MTP 默认关闭，MTP=1 开启；默认行为与上一版一致，速度没有回退 / MTP stays off by default and MTP=1 enables it; default behaviour matches the previous release and speed did not regress.
投机解码的每个草稿 token 都经目标模型验证，贪心输出与普通解码一致 / every speculative token is verified by the target model; greedy output equals plain decoding.

源码 / Sources

llama.cpp：ZHEQIUSHUI/llama.cpp axera-dev @ 2f81e2416
kernel：ZHEQIUSHUI/ax-dsl-kernels main @ b60fc30（models/qwen35_4b）
decode / prefill kernel 源自 / based on ZHEQIUSHUI/sgl-kernel-axera fix/sglang_axera_opt

更新 / Changelog

2026-11-09：统一板端 bin；MTP 验证逻辑修复；默认行为与速度不变；贪心场景显式 presence_penalty: 0；perplexity 固定走全 logits / unified board bins; MTP verification fixes; default behaviour and speed unchanged; explicit presence_penalty: 0 in greedy scenes; perplexity keeps the full-logits head.