Held-out results on the audited task set (62 tasks)

0064939 verified about 12 hours ago

5.39 kB

base_model: Qwen/Qwen3.5-4B
library_name: peft
license: apache-2.0
pipeline_tag: image-text-to-text
datasets:
  - miniweb/MiniWeb
tags:
  - lora
  - web-agents
  - gui-agents
  - browser-use
  - miniweb

MiniWebAgent executor (Qwen3.5-4B LoRA)

The executor of MiniWebAgent: a LoRA adapter on Qwen3.5-4B trained on MiniWebGen's practice data, so the model performs single web interaction skills (primitives) such as filling a form, setting a slider, drawing a signature or uploading a file. In MiniWebAgent the untrained Qwen3.5-4B is the planner: it splits a task into primitive-level instructions and hands each one to this executor, which acts in the same browser; reasoning steps that only read the page stay with the base model.

Code, benchmark and setup: github.com/miniweb-site/MiniWeb. Data: miniweb/MiniWeb.

Results

Task success (%) of Qwen3.5-4B (the paper's Table 3). MiniWeb: the 62 human tasks of the 13 held-out sites, from which no training data comes; the other three benchmarks are not trained on either.

	MiniWeb held-out	WebArena-Lite	WebVoyager	Online-Mind2Web
Untrained single agent	38.7	27.1	45.5	17.9
Untrained planner + untrained executor	45.2	30.7	59.9	21.8
MiniWebAgent (untrained planner + this executor)	58.1	36.4	65.5	33.9

On the 103 generated single-primitive tasks of the held-out sites, the trained model as a single agent goes from 55.3% to 80.3%, with gains in every skill family. On whole tasks it needs the planner: trained mostly on one-primitive episodes, the single agent often stops before reaching the skills it learned (39.2% on the held-out tasks, 38.7% untrained).

Use

Serve the base model with the adapter (vLLM), then run MiniWebAgent with the repository's evaluator:

python scripts/fetch_adapters.py          # -> data/webmix/adapters/pooled_v4
MODEL=Qwen/Qwen3.5-4B SERVED=qwen35-4b PORT=8400 MAX_LORA_RANK=64 \
    LORA_MODULES=pooled_v4=data/webmix/adapters/pooled_v4 bash src/miniwebagent/scripts/serve.sh
python -m miniwebagent.evaluate --human --arm agent_planner --agent-version v6 --planner-name qwen35-4b \
    --adapters none --fallback pooled_v4 --vllm http://127.0.0.1:8400/v1 --ports 8300,8301 --out runs/miniwebagent

With PEFT directly: PeftModel.from_pretrained(base_model, "miniweb/MiniWebAgent").

The executor expects the browser-use agent prompt it was trained on (the system prompt, the task, the agent's memory, the indexed interactive elements and the screenshot) and answers with its memory and one action. Thinking is off: the chat template renders an empty <think></think> block.

Training

Data: 27,646 steps of MiniWebGen (the dataset's training_steps table, rows with is_target and not validation): 3,128 verified single-primitive trajectories on the 52 training sites, 857 chains of two trajectories and 629 trajectories with an injected and corrected mistake, replayed through the agent harness with the base model writing each step's memory.
Recipe: rank-64 LoRA (alpha 128, dropout 0.05) on the language model's attention projections (full-attention layers) and MLP projections (every layer); no vision tower. One epoch, AdamW, learning rate 1e-4 with 3% warmup and a cosine decay, 8 steps per optimizer update, cross-entropy on the whole target (memory and action); 4 GPUs, about 5.5 hours. training_meta.json and train_log.jsonl hold the settings and the loss curve.

Other checkpoints

The paper's other trained executors, in ablations/ (same base model and LoRA shape; each folder has its own training_meta.json). python scripts/fetch_adapters.py --all downloads them under the names the paper's run commands use (scripts/reproduce.py show <arm>).

Folder	What it is	Held-out task success
(root)	MiniWebAgent's executor	58.1
`ablations/task_centric`	trained at matched size on task-centric trajectories: InSTA's pipeline reimplemented on the 52 training sites (the dataset's `task_centric_steps`)	47.8
`ablations/specialists/{drag,io,select,rest,reasoning}`	one specialist per skill family group (supplementary material): `drag`, `io`, `select` and `rest` start from the main executor and train on their families' steps with a KL penalty toward it; `reasoning` starts from the base model	52.7
`ablations/no_done`	the main recipe without each episode's final `done` step, a control for stopping early (single agent)	34.4

All numbers are with the untrained planner except no_done (single agent; 39.2 with the done steps). The specialists are served under their family names and picked by the planner's family for each subtask: LORA_MODULES='drag=.../drag io=.../io select=.../select rest=.../rest reasoning=.../reasoning pooled_v4=...' and --adapters drag,io,select,rest,reasoning --fallback pooled_v4.

Limitations

Trained only on MiniWeb's synthetic sites and English instructions, mostly on single primitives: on whole tasks it relies on the planner for their structure. Outside MiniWeb it has been evaluated only on the three benchmarks above.

License

Apache 2.0, as the base model.