sorryhyun/qwen3-asr-onnx-gqa

Qwen3-ASR text decoders, fused for onnxruntime-web WebGPU

The text decoder of Qwen/Qwen3-ASR-0.6B and Qwen/Qwen3-ASR-1.7B, re-exported with the onnxruntime-genai model builder for ORT's WebGPU execution provider. Meant as a drop-in replacement for the split decoder_init / decoder_step graphs of andrewleech/qwen3-asr-{0.6b,1.7b}-onnx, whose encoder.int4.onnx, embed_tokens.bin and tokenizer.json are still needed.

One graph serves prefill and step:

• GroupQueryAttention with RoPE folded in (Q/K RMSNorm kept as explicit nodes so the graph loads on onnxruntime-web 1.27's GQA schema); fused QKV and gate/up projections; SkipSimplifiedLayerNormalization.
• int4 MatMulNBits, RTN, block size 32, accuracy level 4 — the layout ORT's DP4A and subgroup-matrix kernels require.
• lm_head computes the last position only (prune_lm_head), so prefill returns logits [batch, 1, vocab] and never reads back a [seq, vocab] tensor.
• 519 nodes per token (the split export ran 1586).

File	I/O precision	Sidecar
decoder-0.6b-fp16.onnx	fp16	352 MB
decoder-0.6b-fp32.onnx	fp32	406 MB
decoder-1.7b-fp16.onnx	fp16	985 MB
decoder-1.7b-fp32.onnx	fp32	1.11 GB

Same weights at both precisions; the fp32 graphs are for adapters without shader-f16. Each proto names its sidecar as <name>.onnx.data, so all pairs can sit in one directory.

Interface

Inputs: inputs_embeds [B, S, H] (H = 1024 / 2048; fp16 rows from embed_tokens.bin, with the encoder's audio_features written over the <|audio_pad|> positions), attention_mask [B, total] int64 (its row sum is the real sequence length), and past_key_values.{0..27}.{key,value} [B, 8, past, 128] (empty, past = 0, at prefill). Outputs: logits [B, 1, 151936], present.{0..27}.{key,value}.

Prompt ids, mel parameters and the language header (language X<asr_text>) are as in the andrewleech export. The header can be appended to the prompt to pin the language.

Measured (Apple M1 Pro, Chrome, onnxruntime-web 1.27 WebGPU, 30 s chunk)

Build	prefill	per step	30 s clip
1.7B fp16 (this repo)	0.73 s	19.5 ms	3.4 s
1.7B split export, block 64	5.5–6.5 s	55–60 ms	~15 s
0.6B fp16 (this repo)	0.25 s	14.3 ms	2.3 s
0.6B split export, block 64	—	47.6 ms	—

Greedy output on LibriSpeech test-other matches the split int4 export up to quantization-level differences (a hyphen, a comma).

Recipe

# extract thinker.model.* + thinker.lm_head.weight into a Qwen3ForCausalLM dir, then:
python -m onnxruntime_genai.models.builder -i <dir> -o out -p int4 -e webgpu \
  --extra_options exclude_embeds=true prune_lm_head=true block_size=32 accuracy_level=4 \
                  enable_webgpu_graph=true fuse_qk_norm_gqa=false [use_webgpu_fp32=true]