Qwen3-VL-8B-Instruct-GPTQ-Int4

Instructions to use WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4

SGLang

How to use WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4
```

Configuration Parsing Warning:Invalid JSON for config file config.json

Qwen3-VL-8B-Instruct-GPTQ-Int4

This version of Qwen3-VL-8B-Instruct has been converted to run on the WaveMatrix NPU using w4a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

Support Platform

WM9955
- WM9955N DEMO Board
- M.2 Accelerator card

Image Process

Chips	input size	image num	image encoder	ttft(168 tokens)	w4a16	CMM	Flash
WM9955	384*384	1	280 ms	1066 ms	4.5 tokens/sec	8.3 GiB	9.1 GiB

Video Process

Chips	input size	image num	image encoder	ttft(600 tokens)	w4a16	CMM	Flash
WM9955	384*384	8	1114 ms	3480 ms	4.5 tokens/sec	8.3 GiB	9.1 GiB

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

Install axllm

Approach 1: Clone the repository and run the install script

git clone -b axllm https://github.com/WaveMatrix/ax-llm.git
cd ax-llm
./install.sh

Approach 2: One line command installation (defaul branch axllm)

curl -fsSL https://raw.githubusercontent.com/WaveMatrix/ax-llm/axllm/install.sh | bash

Approach 3: download the executable directly

Go to https://github.com/WaveMatrix/ax-llm/actions?query=branch%3Aaxllm Download the (axllm) then:

chmod +x axllm
sudo mv axllm /usr/bin/axllm

Download the model (Hugging Face)

Create the directory then get into it

mkdir -p WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4
cd WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4
hf download WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4 --local-dir .

Inference with WM9955 Host, such as WM9955 DEMO Board

Run via CLI(To be updated!)

(base) root@wm9955:~# axllm run WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4/
20:13:34.015 INF Init:218 | LLM init start
tokenizer_type = 1
 97% | ###############################  |  38 /  39 [11.25s<11.54s, 3.38 count/s] init post axmodel ok,remain_cmm(6133 MB)
20:13:45.263 INF Init:368 | max_token_len : 2047
20:13:45.263 INF Init:371 | kv_cache_size : 1024, kv_cache_num: 2047
20:13:45.263 INF Init:374 | prefill_token_num : 128
20:13:45.263 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
20:13:45.263 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
20:13:45.263 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
20:13:45.263 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
20:13:45.263 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
20:13:45.263 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 640
20:13:45.263 INF Init:379 | grp: 7, prefill_max_kv_cache_num : 768
20:13:45.263 INF Init:379 | grp: 8, prefill_max_kv_cache_num : 896
20:13:45.263 INF Init:379 | grp: 9, prefill_max_kv_cache_num : 1024
20:13:45.263 INF Init:379 | grp: 10, prefill_max_kv_cache_num : 1152
20:13:45.263 INF Init:384 | prefill_max_token_num : 1152
20:13:45.263 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  39 /  39 [11.25s<11.25s, 3.47 count/s] embed_selector init ok
20:13:47.224 WRN Init:511 | Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square).
20:13:47.224 INF Init:695 | Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656
20:13:47.224 INF Init:728 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2560, out_dtype=fp32
20:13:47.224 INF Init:734 | VisionModule deepstack enabled: layers=3
20:13:47.224 INF load_config:282 | load config:
20:13:47.224 INF load_config:282 | {
20:13:47.224 INF load_config:282 |     "enable_repetition_penalty": false,
20:13:47.224 INF load_config:282 |     "enable_temperature": false,
20:13:47.224 INF load_config:282 |     "enable_top_k_sampling": false,
20:13:47.224 INF load_config:282 |     "enable_top_p_sampling": false,
20:13:47.224 INF load_config:282 |     "penalty_window": 20,
20:13:47.224 INF load_config:282 |     "repetition_penalty": 1.2,
20:13:47.224 INF load_config:282 |     "temperature": 0.9,
20:13:47.224 INF load_config:282 |     "top_k": 10,
20:13:47.224 INF load_config:282 |     "top_p": 0.8
20:13:47.224 INF load_config:282 | }
20:13:47.224 INF Init:448 | LLM init ok
Commands:
  /q, /exit  Exit
  /reset     Reset the kvcache
  /dd        Delete a conversation
  /pp        Print history conversation
Ctrl+C: Stop ongoing generation
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> describe the image
image >> ./WaveMatrix/Qwen3-VL-4B-Instruct-GPTQ-Int4/images/ssd_car.jpg
20:14:13.430 INF EncodeForContent:1121 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=255 (w=384 h=384 tp=2 ps=16 sm=2)
20:14:13.594 INF EncodeForContent:1144 | vision cache store: ./WaveMatrix/Qwen3-VL-4B-Instruct-GPTQ-Int4/images/ssd_car.jpg
20:14:13.616 INF SetKVCache:749 | prefill_grpid:3 kv_cache_num:256 precompute_len:0 input_num_token:168
20:14:13.616 INF SetKVCache:757 | current prefill_max_token_num:1152
20:14:13.616 INF SetKVCache:760 | first run
20:14:13.618 INF Run:818 | input token num : 168, prefill_split_num : 2
20:14:13.618 INF Run:858 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
20:14:13.618 INF Run:881 | prefill indices shape: p=0 idx_elems=384 idx_rows=3 pos_rows=3
20:14:13.940 INF Run:858 | prefill chunk p=1 history_len=128 grpid=2 kv_cache_num=128 input_tokens=40
20:14:13.940 INF Run:881 | prefill indices shape: p=1 idx_elems=384 idx_rows=3 pos_rows=3
20:14:14.295 INF Run:1023 | ttft: 677.29 ms
This is a vibrant street photograph taken in a city, likely London, featuring a classic red double-decker bus as the central subject.

**Key elements in the image:**

- **The Bus:** A bright red, vintage-style double-decker bus, which is a hallmark of London's public transport. The bus is parked or stopped on the street. A prominent advertisement is visible on its side: “WHEN YOU SAY ‘YES’” above the website “WIXMONEY.COM”. The bus has a classic design with large windows and ornate architectural details on its upper deck.

- **The Setting:** The background consists of tall, ornate, multi-story buildings with traditional European architecture, featuring large windows, stone facades, and decorative balconies. This strongly suggests a central or affluent district in a major European city.

- **The Person:** In the foreground, a person (likely a woman) is standing on the sidewalk, looking up at the bus. She is wearing a dark coat and a light-colored hat or head covering, and she is holding a small, light-colored handbag. Her posture and gaze suggest she is observing the bus or the scene.

- **The Atmosphere:** The photo has a bright, clear, and cheerful quality, with natural daylight illuminating the scene. The colors are vivid, especially the red of the bus, which stands out against the more muted tones of the buildings and the person’s clothing.

- **The Composition:** The image is framed to capture the bus and the surrounding architecture, with the person adding a human element and a sense of scale. The perspective is slightly elevated, looking down at the bus and the street.

Overall, the image captures a moment of urban life, blending the iconic imagery of a city bus with the everyday activity of a pedestrian, all set against a backdrop of classic architecture.

20:15:12.812 NTC Run:1145 | hit eos,avg 6.37 token/s
20:15:12.813 INF GetKVCache:721 | precompute_len:409, remaining:743
prompt >> how many people in the image?
image >>
20:15:33.058 INF EncodeForContent:1057 | vision cache hit (mem): ./WaveMatrix/Qwen3-VL-4B-Instruct-GPTQ-Int4/images/ssd_car.jpg
20:15:33.067 INF SetKVCache:749 | prefill_grpid:5 kv_cache_num:512 precompute_len:409 input_num_token:17
20:15:33.067 INF SetKVCache:757 | current prefill_max_token_num:640
20:15:33.068 INF Run:818 | input token num : 17, prefill_split_num : 1
20:15:33.068 INF Run:858 | prefill chunk p=0 history_len=409 grpid=5 kv_cache_num=512 input_tokens=17
20:15:33.068 INF Run:881 | prefill indices shape: p=0 idx_elems=384 idx_rows=3 pos_rows=3
20:15:33.502 INF Run:1023 | ttft: 433.86 ms
Based on the image provided, there is **one person** clearly visible in the foreground — the woman standing on the sidewalk, looking up at the bus. She is the only person explicitly depicted in the photograph.

There may be other people on the bus or in the background, but they are not visible or identifiable in the image. Therefore, the answer is:

> **One person.**

20:15:45.526 NTC Run:1145 | hit eos,avg 6.49 token/s
20:15:45.526 INF GetKVCache:721 | precompute_len:503, remaining:649
prompt >> /q

Downloads last month: 36

Model tree for WaveMatrix/Qwen3-VL-8B-Instruct-GPTQ-Int4

Base model

Qwen/Qwen3-VL-2B-Instruct

Finetuned

(252)

this model