Instructions to use AXERA-TECH/Qwen2.5-VL-3B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AXERA-TECH/Qwen2.5-VL-3B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AXERA-TECH/Qwen2.5-VL-3B-Instruct")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("AXERA-TECH/Qwen2.5-VL-3B-Instruct", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AXERA-TECH/Qwen2.5-VL-3B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AXERA-TECH/Qwen2.5-VL-3B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen2.5-VL-3B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AXERA-TECH/Qwen2.5-VL-3B-Instruct

SGLang

How to use AXERA-TECH/Qwen2.5-VL-3B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AXERA-TECH/Qwen2.5-VL-3B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen2.5-VL-3B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AXERA-TECH/Qwen2.5-VL-3B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen2.5-VL-3B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AXERA-TECH/Qwen2.5-VL-3B-Instruct with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/Qwen2.5-VL-3B-Instruct
```

Qwen2.5-VL-3B-Instruct

This version of Qwen2.5-VL-3B-Instruct has been converted to run on the Axera NPU using w8a16 quantization.

This model has been optimized with the following LoRA:

Compatible with Pulsar2 version: 3.4

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU HOST LLM Runtime

Support Platform

AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card

Image Process

Chips	input size	image num	image encoder	ttft(384 tokens)	w8a16	DDR	Flash
AX650	448*448	1	780 ms	1651 ms	5.9 tokens/sec	4.3 GiB	4.6 GiB

Video Process

Chips	input size	image num	image encoder	ttft(512 tokens)	w8a16	DDR	Flash
AX650	308*308	8	1400 ms	2455 ms	5.9 tokens/sec	4.4 GiB	4.7 GiB

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

推荐使用 ax-llm 项目提供的安装脚本在板端直接完成安装:

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Inference

root@ax650:/mnt/sdcard/projects/llm/Qwen2.5-VL-3B-Instruct# axllm run ./Qwen2.5-VL-3B-Instruct-AX650-chunk_prefill_512
14:05:04.510 INF Init:2700 | LLM init start
tokenizer_type = 1
 97% | ###############################  |  38 /  39 [30.75s<31.56s, 1.24 count/s] init post axmodel ok,remain_cmm(6381 MB)
14:05:35.265 INF Init:3023 | max_token_len : 1023
14:05:35.265 INF Init:3026 | kv_cache_size : 256, kv_cache_num: 1023
14:05:35.265 INF init_groups_from_model:1827 | prefill_token_num : 128
14:05:35.265 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 1023
14:05:35.265 INF init_groups_from_model:2079 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
14:05:35.265 INF init_groups_from_model:2079 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
14:05:35.265 INF init_groups_from_model:2079 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
14:05:35.265 INF init_groups_from_model:2079 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
14:05:35.265 INF init_groups_from_model:2079 | prefill grp: 4, gid: 5, history_cap: 512, total_cap: 640, symbolic_cap: 512
14:05:35.265 INF init_groups_from_model:2086 | prefill_max_token_num : 640
14:05:35.265 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  39 /  39 [30.75s<30.75s, 1.27 count/s] embed_selector init ok
14:05:47.886 WRN Init:1364 | Qwen-VL vision size override: cfg=384x384 bytes=884736, model_input_bytes=1204224 -> 448x448 (square).
14:05:47.886 WRN Init:1508 | vision encoder tokens_per_block=256 (expected=196). Using fp32 by nSize inference (out0.nSize=2097152).
14:05:47.886 INF Init:1647 | Qwen token ids: vision_start=151652 image_pad=151655 video_pad=151656 audio_pad=-1
14:05:47.886 INF Init:1704 | VisionModule init ok: type=Qwen3VL, tokens_per_block=256, embed_size=2048, out_dtype=fp32
14:05:47.886 WRN Init:1713 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
14:05:47.886 INF load_config:554 | load config:
14:05:47.886 INF load_config:554 | {
14:05:47.886 INF load_config:554 |     "enable_repetition_penalty": false,
14:05:47.886 INF load_config:554 |     "enable_temperature": true,
14:05:47.886 INF load_config:554 |     "enable_top_k_sampling": true,
14:05:47.886 INF load_config:554 |     "enable_top_p_sampling": false,
14:05:47.886 INF load_config:554 |     "penalty_window": 20,
14:05:47.886 INF load_config:554 |     "repetition_penalty": 1.2,
14:05:47.886 INF load_config:554 |     "temperature": 0.9,
14:05:47.886 INF load_config:554 |     "top_k": 10,
14:05:47.886 INF load_config:554 |     "top_p": 0.8
14:05:47.886 INF load_config:554 | }
14:05:47.886 INF Init:3138 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" or "video:<video_file>[:<fps>]" for video (default fps=2), "audio:<file>" for audio.
----------------------------------------
prompt >> 描述图片内容
media >> /mnt/sdcard/projects/video/frame_0000.jpg
14:07:30.353 INF EncodeForContent:2344 | Qwen-VL pixel_values[0] bytes=1204224 min=0 max=255 (w=448 h=448 tp=2 ps=16 sm=2)
14:07:31.108 INF EncodeForContent:2403 | vision cache store: /mnt/sdcard/sunyijie/projects/llm/Qwen2.5-VL-3B-Instruct/video/frame_0000.jpg
14:07:31.109 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:280 prefer_symbolic_group:0
14:07:31.109 INF SetKVCache:3743 | current prefill_max_token_num:640
14:07:31.120 INF SetKVCache:3759 | first run
14:07:31.151 INF Run:4287 | input token num : 280, prefill_split_num : 3
14:07:31.151 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
14:07:31.569 INF Run:4367 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=128
14:07:32.240 INF Run:4367 | prefill chunk p=2 history_len=256 grpid=4 kv_cache_num=384 input_tokens=24
14:07:33.068 INF Run:4680 | ttft: 1916.95 ms
这张14:07:33.068 INF Run:4722 | VLM decode positions: rope_start=98 dense_kv_start=280
图片展示了一个多云的天气场景，蓝天中飘着大片的白云。地面是一片绿色的草地，阳光照射在草地上，反射出金色的光芒。整体画面给人一种宁静、明亮的感觉。

14:07:40.817 NTC Run:5095 | hit eos,decode avg 5.68 token/s
14:07:40.817 INF GetKVCache:3698 | precompute_len:325, remaining:315 (tracked)
prompt >> /q
14:15:01.768 INF CheckCmmBalance:2492 | [cmm-sentry] Deinit: CMM balanced (baseline 9963 MB, now 9963 MB)

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for AXERA-TECH/Qwen2.5-VL-3B-Instruct

Base model

Qwen/Qwen2.5-VL-3B-Instruct

Finetuned

(900)

this model

Collections including AXERA-TECH/Qwen2.5-VL-3B-Instruct

Qwen2.5

Collection

11 items • Updated 2 days ago

Multimodal Models

Collection

25 items • Updated 2 days ago • 2