FG-CLIP2 on AX650

FG-CLIP2 image/text encoders converted for the Axera AX650 NPU. This repository ships:

• FP32 ONNX graphs for the development host;
• U16 axmodels for the board;
• the original PyTorch weights and configs.

Two model sizes are included:

Model	Feature dim	Directory
base	768	fg-clip2-base/
large	1024	fg-clip2-large/

Repository layout

FG-CLIP/
├── bedroom.jpg                 # demo image
├── src/infer/
│   ├── fgclip2_infer_onnx.py   # host ONNX inference (base + large)
│   └── fgclip2_infer_axmodel.py# board axmodel inference (base + large)
├── fg-clip2-base/              # base weights, ONNX and axmodels
└── fg-clip2-large/             # large weights, ONNX and axmodels

Model inputs and outputs

All image encoders take patchified input, not NCHW images:

pixel_values: [1, N, 768]   # N patches, each 16x16x3 = 768 values

N is the number of tokens the graph was exported for. A grid h x w of valid patches is padded to N; if N > h*w, the model also needs a patch mask.

large (fg-clip2-large/)

File	Input	Output	Notes
fgclip2_large_image_encoder_g16x16_n256_nomask.onnx	pixel_values [1,256,768]	image_features [1,1024]	16×16 grid, no mask, raw
fgclip2_large_image_encoder_g16x16_n256_nomask_u16.axmodel	same	same	U16
fgclip2_large_image_encoder_g14x14_n196_nomask.onnx	pixel_values [1,196,768]	norm_image_features [1,1024]	14×14, no mask, normalized
fgclip2_large_image_encoder_g28x28_n784_nomask.onnx	pixel_values [1,784,768]	norm_image_features [1,1024]	28×28, no mask, normalized
fgclip2_large_text_encoder_short_t64.onnx (+ .onnx.data)	input_ids [1,64] int64	text_features [1,1024]	short branch, raw
fgclip2_large_text_encoder_short_t64_u16.axmodel	same	same	U16
fgclip2_large_text_encoder_long_t196.onnx (+ .onnx.data)	input_ids [1,196] int64	norm_text_features [1,1024]	long branch, normalized
export_info.json	—	—	grid, token count, preprocessing

base (fg-clip2-base/)

File	Input	Output	Notes
fgclip2_base_image_encoder_224x224_n196_nomask.axmodel	pixel_values [1,196,768]	norm_image_features [1,768]	14×14, no mask, normalized
fgclip2_base_image_encoder_448x448_n784_nomask.onnx	pixel_values [1,784,768]	norm_image_features [1,768]	28×28, no mask, normalized
fgclip2_base_image_encoder_448x448_n784_nomask.axmodel	same	same	U16
fgclip2_base_text_encoder_196-long_u16.axmodel	input_ids [1,196]	norm_text_features [1,768]	long branch, normalized
fgclip2_base_text_encoder_64-short_u16.axmodel	input_ids [1,64]	text_features [1,768]	short branch, raw
fgclip2_base_text_encoder_short.onnx	input_ids [1,64]	text_features [1,768]	short branch, raw

before computing cosine similarity. norm_* outputs are already normalized.

Quick start

ONNX (development host)

python src/infer/fgclip2_infer_onnx.py \
    --image-model fg-clip2-large/fgclip2_large_image_encoder_g16x16_n256_nomask.onnx \
    --text-model  fg-clip2-large/fgclip2_large_text_encoder_short_t64.onnx \
    --tokenizer   fg-clip2-large/tokenizer.json \
    --image bedroom.jpg \
    --texts "一个简约风格的卧室角落" "一只猫在窗台上晒太阳"

axmodel (AX650 board)

python src/infer/fgclip2_infer_axmodel.py \
    --image-model fg-clip2-base/fgclip2_base_image_encoder_224x224_n196_nomask.axmodel \
    --text-model  fg-clip2-base/fgclip2_base_text_encoder_196-long_u16.axmodel \
    --tokenizer   fg-clip2-base/tokenizer.json \
    --image bedroom.jpg \
    --texts "一个简约风格的卧室角落" "一只猫在窗台上晒太阳"

The axmodel script resolves the patch count from the runtime input shape, export_info.json, the filename, or explicit arguments. The ONNX script uses export_info.json (including the directory-level export_info.json) or explicit --grid-h/--grid-w/--max-num-patches. Text length is read from input_ids when available, then from export_info.json / the filename.

Preprocessing

Image (performed on the host, outside the graph):

RGB -> resize to (grid_w * 16, grid_h * 16) -> /255 -> (x - 0.5) / 0.5
    -> split into 16x16 patches, row-major -> [1, h*w, 768]
    -> pad to N tokens with zeros
    -> if the model has a mask input: mask = 1 for valid patches, 0 for padding

Text:

• tokenizer: tokenizer.json (Gemma BPE), shipped with each model;
• padding to the model's fixed length (64 for short/box, 196 for long), with truncation;
• the ONNX script lowercases captions by default (--no-lowercase disables it); the axmodel script passes text through unchanged.

Naming convention

g{H}x{W}_n{N}   grid H x W, padded to N tokens
t{L}            fixed text length
nomask          model has no pixel_attention_mask input
u16             U16 quantized axmodel

The base image axmodels use {H}x{W}_n{N} (pixel size) instead of g{H/16}x{W/16}_n{N}; they are equivalent (224×224 = 14×14 patches, 448×448 = 28×28 patches).

Reference latency

Measurements depend on the input size, quantization config and Pulsar2 version; re-measure for your build.

Measured latency: base image encoders

Measured on AX650 with:

ax_run_model -m <model> -w 3 -r 500

3 cores, 3 warmups, 500 repeats, batch={auto:0}, parallel=false. Environment: Pulsar2 8.0 e9a1aa15, engine 2.13.0sp, tool 2.5.1a.

448×448 (28×28 = 784 patches)

fgclip2_base_image_encoder_g28x28_n784_nomask.axmodel

Metric	Value
cmm size	112,003,730 Bytes (~106.8 MiB)
min	59.831 ms
avg	59.902 ms
median	59.895 ms
90%	59.952 ms
95%	59.976 ms
99%	60.010 ms
max	60.056 ms

224×224 (14×14 = 196 patches)

fgclip2_base_image_encoder_g14x14_n196_nomask.axmodel

Metric	Value
cmm size	102,118,612 Bytes (~97.4 MiB)
min	10.927 ms
avg	10.951 ms
median	10.942 ms
90%	10.992 ms
95%	11.013 ms
99%	11.073 ms
max	11.077 ms

The measured filenames use g{H}x{W}; the repository ships the same models as fgclip2_base_image_encoder_448x448_n784_nomask.axmodel and fgclip2_base_image_encoder_224x224_n196_nomask.axmodel.