JackyHoCL/qwen3-asr-yue-lora-preview

Qwen3-ASR-1.7B + LoRA — Cantonese (yue-HK) ASR SFT (PREVIEW v4)

LoRA adapter for Qwen/Qwen3-ASR-1.7B fine-tuned on LegCo Cantonese speech (Hong Kong Legislative Council webcasts) to improve Cantonese ASR. Preview model — v4: trained on the 227.9 h expanded dataset (56 sessions); selected from all 3 epoch checkpoints by full test+dev WER. Supersedes v3.

Checkpoint selection (full splits, 5,912-row test / 5,891-row dev)

Checkpoint	test WER	test CER	dev WER	dev CER
zero-shot Qwen3-ASR-1.7B	0.326	0.325	—	—
epoch 1 (ckpt-2446)	0.178	0.179	0.172	0.171
epoch 2 (ckpt-4892) — uploaded	0.174	0.173	0.163	0.163
epoch 3 (ckpt-7338)	0.176	0.176	0.165	0.164

Best epoch-2 checkpoint gives ~47% relative WER reduction vs zero-shot. (Per-epoch TB dev-subset WER — 48 seeded rows — suggested epoch 3; the full split evaluation overrode that, hence ckpt-4892 is the release.)

Data

• Source: JackyHoCL/legco-yue-asr-preview v4 snapshot (57,566 utterances / 227.90 h / 56 sessions, 16 kHz mono WAV, sliced <=20 s)
• Training label: written_yue (official-record written Cantonese)
• Filters (all splits): drop rows with any QC flag (gt_longer, llm_error, low_asr_match, ...) and match_confidence < 0.5 -> 50,937 rows (train 39,134 / dev 5,891 / test 5,912, ~207 h)

Training

• LoRA on thinker LLM linear layers only (thinker.model.layers.*.self_attn.{q,k,v,o}_proj + thinker.model.layers.*.mlp.{gate,up,down}_proj), rank 16, alpha 32, dropout 0.05 -> 17.4 M trainable params (0.85%); audio tower frozen
• 3 epochs, lr 5e-5 cosine (warmup 5%), bf16, batch 4 x grad-accum 4, gradient checkpointing, 6 h 20 m on one RTX 3080 20 GB
• Per-epoch dev eval (loss + 48-row WER subset) logged to TensorBoard; every epoch checkpoint evaluated on full test+dev before selection

Usage

Requires qwen-asr==0.0.6 + transformers==4.57.6 (the native transformers>=5 qwen3_asr classes do not match this checkpoint layout) and peft.

import torch
import soundfile as sf
from qwen_asr.core.transformers_backend import (
    Qwen3ASRForConditionalGeneration, Qwen3ASRProcessor)
from peft import PeftModel

processor = Qwen3ASRProcessor.from_pretrained("JackyHoCL/qwen3-asr-yue-lora-preview")
model = Qwen3ASRForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-ASR-1.7B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "JackyHoCL/qwen3-asr-yue-lora-preview")
model = model.to("cuda").eval()

# The outer wrapper class has no `forward`; route it to thinker.forward
# (same patch as the official Qwen3-ASR fine-tuning script):
def _patch(outer):
    outer_cls = type(outer)
    if getattr(outer_cls.__dict__.get("forward", None), "_patched", False):
        return
    def forward(self, *args, **kwargs):
        return self.thinker.forward(*args, **kwargs)
    forward._patched = True
    outer_cls.forward = forward
_patch(model)

audio = sf.read("clip.wav", dtype="float32")[0]
if audio.ndim > 1:
    audio = audio.mean(axis=1)

# prefix: empty system + audio user turn, with forced Cantonese language tag
prefix = processor.apply_chat_template(
    [{"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": "placeholder"}]}],
    add_generation_prompt=True, tokenize=False)[0] + "language Cantonese<asr_text>"
inp = processor(text=[prefix], audio=[audio], return_tensors="pt").to("cuda")
out = model.generate(**inp, max_new_tokens=256)
new = out[0][inp["input_ids"].shape[1]:]
print(processor.batch_decode([new], skip_special_tokens=True)[0])

Notes

• Repo role: dev / test / preview builds. Release: JackyHoCL/qwen3-asr-yue-cantonese-lora.
• Target format: language Cantonese<asr_text> + transcript (official Qwen3-ASR fine-tuning contract).
• The audio tower (perceiver + conv stem) is frozen; LoRA covers LLM self-attention + MLP projections only.
• Eval WER/CER: Levenshtein over character tokens, normalized by reference length, punctuation stripped on both sides.
• v3 (103.7 h snapshot, 2 epochs): test 0.170 vs its zero-shot 0.313 — different split, not directly comparable with v4 numbers.