Qwen3.8-27B-AX8860-SGLang

Qwen3.8-27B adapted for the Axera AX8860 NPU, served with SGLang through the sglang_axera backend. The vision tower runs as a compiled BF16 AXModel on the NPU; the language model runs with NVFP4 quantization.

Repository contents

Path	Description
model/vision/qwen3_8_27b_vision_64.axmodel	Vision encoder compiled for AX8860, fixed 64x64 input grid (Git LFS)
model/Qwen3.8-27B-NVFP4/	Language-model weights (NVFP4, ~21G) — distributed separately
compile_qwen3_5_vision_axmodel.py	Script that produces the vision AXModel from HF checkpoints
run_qwen3_8_server.sh	Online SGLang launch script for AX8860 (OpenAI-compatible server)

End-to-end: from HF checkpoint to AX8860

1. Download the official checkpoint

The unquantized Qwen3.8-27B checkpoint (multimodal, ships the vision tower) comes from the Qwen organization:

hf download Qwen/Qwen3.8-27B --local-dir Qwen3.8-27B

2. Convert the language model to NVFP4

Quantize the language model to NVFP4, producing a checkpoint directory Qwen3.8-27B-NVFP4 in HF format (safetensors, with the model.visual.* weights still present — the vision compile reads them). This step uses the team quantization toolchain and is not part of this repository; the converted weights (~21G) are distributed separately and placed under model/Qwen3.8-27B-NVFP4/.

3. Compile the vision tower

Export the vision tower to a BF16 AXModel for AX8860 (see Recompiling the vision tower for prerequisites and details):

python compile_qwen3_5_vision_axmodel.py \
    --model-dir /path/to/Qwen3.8-27B-NVFP4 \
    --image-size 64 \
    --chip-type AX8860 \
    --output model/vision/qwen3_8_27b_vision_64.axmodel

Vision encoder AXModel

The exported graph takes the HF processor's patchified pixels directly — pixel_values: [num_patches, in_channels * temporal_patch_size * patch_size^2] — with the image grid baked into the artifact:

Model	Input (64x64 image)	Grid	Output
Qwen3.8-27B	[16, 1536]	[1, 4, 4]	[4, 5120]
Qwen3.5-9B	[16, 1536]	[1, 4, 4]	[4, 4096]

The runtime wrapper (AxeraVisionEncoder) validates that each request matches the compiled square grid; other resolutions must be recompiled with a different --image-size. BF16 only — the vision encoder runtime requires bf16.

Recompiling the vision tower

Prerequisites: PyTorch + Transformers with Qwen3.5 modeling (Qwen3.8 reuses it), the torch_axera wheel (axera_air), and the axnn / sgl-kernel-axera / sglang_axera repos on PYTHONPATH (the latter is used for the post-compile verification step).

export PYTHONPATH="$PWD/axnn:$PWD/sgl-kernel-axera/python:$PWD/sglang_axera/python:$PYTHONPATH"

python compile_qwen3_5_vision_axmodel.py \
    --model-dir /path/to/Qwen3.8-27B-NVFP4 \
    --image-size 64 \
    --chip-type AX8860 \
    --output model/vision/qwen3_8_27b_vision_64.axmodel

--image-size must be divisible by the patch size (16), and the resulting grid side by the spatial merge size (2). --chip-type AX650 is also supported.

After compilation the script loads the dumped AXModel with AxeraVisionEncoder and compares it against the eager vision tower, aborting if cosine similarity drops below 0.99. Pass --skip-verify to skip this check.

Runtime usage

With the sglang_axera backend installed, the compiled artifact is loaded by AxeraVisionEncoder:

from sglang.srt.hardware_backend.axera.modules.qwen3_5_vision_encoder_axera import AxeraVisionEncoder

encoder = AxeraVisionEncoder("model/vision/qwen3_8_27b_vision_64.axmodel", vision_config)
features = encoder(pixel_values, grid_thw)  # [4, 5120] for a 64x64 image

Serving on AX8860

run_qwen3_8_server.sh launches an OpenAI-compatible HTTP server on the board. It expects the on-board sglang_axera wheels (see axera_readme.md §7.1), the NVFP4 weights under model/Qwen3.8-27B-NVFP4/, and at least 16G of swap (27B requirement). The script points SGLANG_AXERA_QWEN35_VISION_MODEL at the compiled vision AXModel, which the backend swaps in for the native vision tower (TP1/PP1, no data parallel):

bash run_qwen3_8_server.sh   # serves on 127.0.0.1:30000

The 27B runs with conservative on-board parameters (--context-length 512, --max-total-tokens 2048, --max-running-requests 1, --page-size 1, --mem-fraction-static 0.90) taken from the validated bring-up kit; tune them only with board memory headroom in mind. First BRUN launch compiles kernels into a local cache (BRUN cache miss, compiling model); later launches with the same shape parameters load from cache.

License

BSD-3-Clause.