BirdAgent GSPO adapter + model card + figures

README.md

@@ -1,3 +1,216 @@
 ---
 license: apache-2.0
+base_model: Qwen/Qwen3-VL-4B-Instruct
+library_name: peft
+pipeline_tag: image-text-to-text
+language:
+- en
+tags:
+- lora
+- peft
+- agentic
+- tool-use
+- function-calling
+- vision-language
+- bird-identification
+- fine-grained-recognition
+- gspo
+- grpo
+- rlvr
 ---
+
+<h1 align="center">BirdAgent&nbsp;·&nbsp;Qwen3-VL-4B</h1>
+
+<p align="center">
+  <b>A 4B vision–language agent that identifies birds by <i>orchestrating domain tools</i>,<br>
+  beating much larger models that are handed the very same tools.</b>
+</p>
+
+<p align="center">
+  📄 <a href="https://github.com/xinzhuwang-wxz/Bird-Agent">Paper (under review)</a> &nbsp;·&nbsp;
+  💻 <a href="https://github.com/xinzhuwang-wxz/Bird-Agent">Code</a> &nbsp;·&nbsp;
+  📊 <a href="https://huggingface.co/datasets/Chinzhu/BirdAgent-Benchmarks">Benchmarks</a> &nbsp;·&nbsp;
+  🧩 <a href="https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct">Base model</a>
+</p>
+
+---
+
+> **TL;DR** — Fine-grained bird ID is hard for *every* model class because the
+> deciding evidence often is not in the image being looked at (it is in a
+> diagnostic call, a range prior, or a magnified detail crop). Instead of scaling
+> the model, we teach a small one to **orchestrate** the tools that recover that
+> evidence. **BirdAgent** (a LoRA adapter on `Qwen3-VL-4B-Instruct`) reaches a
+> pooled **solve** rate of **0.34**, *beating every same-tool API model*
+> (Qwen3-235B + tools = 0.00, Doubao-2.1-Pro + tools = 0.17), beating Sonnet, and
+> trailing only Opus (0.46) — at ~50–60× fewer parameters.
+
+<p align="center">
+  <img src="assets/fig_main.png" width="620" alt="Pooled solve rate: our 4B agent beats every same-tool API model and Sonnet, second only to Opus."><br>
+  <em>Pooled solve rate (correct at the declared taxonomic grain). Our 4B agent
+  (orange/vermilion) beats every same-tool API model and Sonnet, and trails only
+  Opus. Dashed line = pure-recognition floor (T0).</em>
+</p>
+
+## Highlights
+
+- **Orchestration beats scale.** A trained 4B agent outperforms 30–235B API
+  models given the *identical* tool interface.
+- **Having tools ≠ using tools.** Qwen3-235B *calls* the exact same tools yet
+  scores **0.00** — tool access does not confer tool *use*.
+- **Calibrated, not reckless.** The agent names a species when the evidence
+  supports it and gracefully degrades to genus/family when it does not;
+  species-declaration precision rises to **1.0** with **overclaim = 0.00**.
+- **Three-modality.** Image, sound-only, or image+sound (audio is tool-mediated).
+- **Reproducible & honest.** All numbers are Δ-over-base under one identical tool
+  harness; no leaderboard gaming, no closed-model distillation.
+
+## Model at a glance
+
+| | |
+|---|---|
+| **Base model** | [`Qwen/Qwen3-VL-4B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) (Apache-2.0) |
+| **This repo** | LoRA adapter (the **released** checkpoint = GSPO ckpt-200) |
+| **Parameters** | 4B base + LoRA (`r=64`, `α=128`, `dropout=0.05`, bf16) |
+| **Adapter targets** | language-model `q/k/v/o/gate/up/down_proj` (`peft ≥ 0.19`) |
+| **Training** | SFT cold-start → on-policy DPO → **GSPO** (RLVR) |
+| **Task** | agentic fine-grained bird identification with calibrated abstention |
+| **Modalities** | image · sound · image+sound |
+| **License** | Apache-2.0 (adapter); base is Apache-2.0 |
+
+> ⚠️ **This repository is the policy only.** To run the full agent you also need
+> the tool servers (Grounding-DINO, BioCLIP-2, Perch-2, SINR) and the evaluation
+> harness — see the [code repository](https://github.com/xinzhuwang-wxz/Bird-Agent).
+> Loaded standalone, the model emits Hermes-format `<tool_call>` turns that
+> expect tool responses to be fed back; it is **not** a plain image→label classifier.
+
+## Results
+
+Two self-built, tier-stratified benchmarks — an **agentic** (information-gap) set
+and a **calibration** set — evaluated under an **identical tool harness** for
+every model. Metric = **solve**: correct at the *declared* grain (a genus verdict
+counts iff the genus is right; over-committed species are penalized).
+
+| # | model / condition | common | uncommon | rare | **overall** | n |
+|---|---|:--:|:--:|:--:|:--:|:--:|
+| 1 | Opus · bare | 0.75 | 0.25 | 0.38 | **0.46** | 24 |
+| 2 | Opus · web | 0.62 | 0.25 | 0.25 | 0.38 | 24 |
+| 3 | **BirdAgent-DPO (4B) · tools** | 0.61 | 0.23 | 0.22 | **0.34** | 80 |
+| 4 | Sonnet · bare | 0.62 | 0.12 | 0.25 | 0.33 | 24 |
+| 5 | **BirdAgent-SFT (4B) · tools** | 0.42 | 0.25 | 0.22 | 0.29 | 90 |
+| 6 | Sonnet · web | 0.50 | 0.25 | 0.12 | 0.29 | 24 |
+| 7 | Doubao-2.1-Pro · bare | 0.41 | 0.21 | 0.08 | 0.23 | 70 |
+| 8 | T0 tool floor | 0.46 | 0.10 | 0.03 | 0.20 | 900 |
+| 9 | Doubao-2.1-Pro · our-tools | 0.45 | 0.04 | 0.04 | 0.17 | 70 |
+| 10 | Qwen3-235B · bare | 0.14 | 0.00 | 0.08 | 0.07 | 70 |
+| 11 | base-4B · tools | 0.12 | 0.06 | 0.00 | 0.06 | 90 |
+| 12 | Qwen3-235B · web | 0.09 | 0.04 | 0.00 | 0.04 | 70 |
+| 13 | Qwen3-235B · our-tools | 0.00 | 0.00 | 0.00 | 0.00 | 70 |
+
+*Big models: C0 = bare, N = vendor web search, C1 = our exact tools. API baselines
+run on cost-bounded stratified subsets; our models on the full sets. Vendor web
+search does not help image ID (C0 ≈ N).*
+
+## Quickstart
+
+```python
+from transformers import AutoProcessor, AutoModelForImageTextToText
+from peft import PeftModel
+
+base = "Qwen/Qwen3-VL-4B-Instruct"
+model = AutoModelForImageTextToText.from_pretrained(
+    base, torch_dtype="bfloat16", device_map="auto")
+model = PeftModel.from_pretrained(model, "Chinzhu/BirdAgent-Qwen3VL-4B")
+processor = AutoProcessor.from_pretrained(base)
+
+# BirdAgent is an *agent*: give it the BirdAgent system prompt + the nine tool
+# schemas, then run a loop that answers its <tool_call> turns with real tool
+# observations (detection boxes, classifier top-k, geo prior, ...). The full
+# harness + tool servers are in the code repo:
+#   https://github.com/xinzhuwang-wxz/Bird-Agent
+```
+
+## How it works
+
+### The nine tools
+
+| group | tools |
+|---|---|
+| **Perception** (local, deterministic) | `detect_bird` (Grounding-DINO) · `quality_gate` · `zoom_in` · `crop` · `enhance` · `audio_quality` |
+| **Recognition** (served models) | `classify_image` (BioCLIP-2) · `classify_sound` (Perch-2) |
+| **External** | `geo_prior` (SINR range × month) |
+
+The agent plans a sequence of calls, accumulates the returns in an explicit
+**evidence ledger**, and commits a verdict at a **calibrated taxonomic grain**.
+All tools are Apache/MIT-licensed and deterministic given input (so their outputs
+can be pre-cached for single-GPU RL).
+
+### Training recipe (SFT → DPO → GSPO)
+
+<p align="center">
+  <img src="assets/fig_stage.png" width="300" alt="Stage chain: calibration solve and species-declaration precision across Base/SFT/DPO/GSPO.">
+  <img src="assets/fig_gspo.png" width="300" alt="GSPO-200 vs DPO initialization on both benchmark sets."><br>
+  <em>Left: DPO sharpens calibration (species-declaration precision 0.33→1.0),
+  then GSPO adds orchestration; overclaim stays 0.00 throughout. Right: GSPO lifts
+  both benchmark sets over its DPO initialization, apples-to-apples.</em>
+</p>
+
+- **SFT cold-start** on code-authored *blueprint* tool-use trajectories (loss on
+  assistant + tool-call tokens only; tool observations masked).
+- **On-policy DPO** on soft preferences only (ledger discipline, call parsimony)
+  — sharpens calibration (species-declaration precision **0.33 → 1.0**).
+- **GSPO** (sequence-level importance weighting; `β=0.04`, `lr=1e-6`,
+  `num_generations=4`) with a **grain-graded, reachability-aware** reward:
+  species `+1.0/−0.5`, genus `+0.4/−0.3`, family `+0.2/−0.1`, abstain `0`, with
+  full species credit gated on the truth being present in some tool's top-*k*.
+  GSPO improves over its DPO init apples-to-apples (pooled **0.34 → 0.35**) while
+  keeping **overclaim = 0.00** — bolder but not reckless.
+
+## Limitations & responsible use
+
+<p align="center">
+  <img src="assets/fig_tier.png" width="560" alt="Tier-stratified solve: the common/uncommon/rare cliff is universal — even Opus collapses on the tail."><br>
+  <em>The common→rare cliff is universal — even Opus collapses on the tail —
+  because the true species leaves the classifiers' top-k. This is the tool
+  ceiling, orthogonal to the agent.</em>
+</p>
+
+- **Tool ceiling.** Species accuracy is capped by the classifiers: on hard
+  rare-tier items ~76% of errors are cases where the truth is absent from every
+  classifier's top-*k*. A stronger fine-grained recognizer is an *orthogonal*
+  lever (the agent already degrades honestly in this regime).
+- **General-VLM regression (format lock).** Agentic training locks the model into
+  emitting tool calls; on generic MCQ probes it drops sharply (MMStar 0.51→0.04,
+  MMBench 0.89→0.15). Use it as a *bird agent*, not a general VLM, unless you mix
+  general trajectories back in.
+- **Evaluation n.** API baselines were run on cost-bounded stratified subsets
+  (n = 12–40); numbers are Δ-over-base under one harness, not leaderboard ranks.
+- **Responsible use.** Research / non-commercial for v0. The API models above are
+  **evaluation controls, not teachers** — no closed-model output was distilled
+  into this policy.
+
+## Related releases
+
+- [`Chinzhu/BirdAgent-Qwen3VL-4B-DPO`](https://huggingface.co/Chinzhu/BirdAgent-Qwen3VL-4B-DPO) — on-policy DPO adapter (this model's GSPO initialization).
+- [`Chinzhu/BirdAgent-Qwen3VL-4B-SFT`](https://huggingface.co/Chinzhu/BirdAgent-Qwen3VL-4B-SFT) — SFT cold-start adapter.
+- [`Chinzhu/BirdAgent-Benchmarks`](https://huggingface.co/datasets/Chinzhu/BirdAgent-Benchmarks) — the agentic (500) and calibration (400) evaluation sets.
+
+## Citation
+
+```bibtex
+@inproceedings{wang2026birdagent,
+  title     = {BirdAgent: A Small Vision--Language Model that Orchestrates
+               Domain Tools Beats Large Models that Merely Hold Them},
+  author    = {Wang, Xinzhu},
+  booktitle = {Under review},
+  year      = {2026}
+}
+```
+
+## Acknowledgements
+
+Built on [Qwen3-VL](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct),
+[BioCLIP-2](https://imageomics.github.io/bioclip-2/),
+[Perch-2](https://github.com/google-research/perch),
+[Grounding-DINO](https://github.com/IDEA-Research/GroundingDINO), and
+[SINR](https://github.com/elijahcole/sinr). Trained with
+[ms-swift](https://github.com/modelscope/ms-swift).

adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(v_proj|k_proj|up_proj|o_proj|down_proj|q_proj|gate_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e44bc99dc44f5b207f3239298cdb33f84043884c46bb0c0dcd5d217b99f6cd7
+size 264316960

additional_config.json

@@ -0,0 +1,5 @@
+{
+  "lora_dtype": null,
+  "lorap_lr_ratio": null,
+  "lorap_emb_lr": 1e-06
+}