inference-sh/skillos-curator-qwen3-8b-trl-fft

SkillOS curator — Qwen3-8B, TRL full fine-tune (selected arms, 3 seeds)

Curator models from an independent reproduction of SkillOS, trained with TRL's GRPOTrainer (DeepSpeed ZeRO-3 + vLLM colocate) rather than the paper's verl. Companion to inference-sh/skillos-curator-qwen3-8b-verl-gigpo, which holds a full 12-checkpoint verl/GiGPO sweep.

from transformers import AutoModelForCausalLM, AutoTokenizer
name = "inference-sh/skillos-curator-qwen3-8b-trl-fft"
tok = AutoTokenizer.from_pretrained(name, subfolder="fft-seed3-step5")
model = AutoModelForCausalLM.from_pretrained(name, subfolder="fft-seed3-step5", dtype="bfloat16")

What is here, and what is not

Three seeds × 60 GRPO steps were trained with checkpoints every 5 steps, i.e. 36 arms. This repo publishes 7 of them: each seed's peak and final checkpoint, plus seed-3 step 5. The other 29 arms exist and were evaluated — their numbers are in docs/repro_report.md Appendix A and their per-game rollouts are in the eval dataset — but the weights are not uploaded. Do not read this subset as a full sweep.

subfolder	8B Δ vs no-memory	p	32B Δ	note
fft-seed1-step20	+10.7pp	0.032	−2.1pp	seed-1 peak on 8B
fft-seed1-step60	+5.7pp	0.18	—	seed-1 final
fft-seed2-step35	+13.6pp	0.0026	−4.3pp	best 8B arm in the project
fft-seed2-step60	+4.3pp	0.33	—	seed-2 final
fft-seed3-step5	−2.1pp	0.68	+13.6pp	best 32B arm — 62.9% absolute
fft-seed3-step55	+11.4pp	0.011	+2.1pp	seed-3 peak on 8B
fft-seed3-step60	+3.6pp	0.47	—	seed-3 final

8B: 140 ALFWorld valid_seen games vs a 33.6% no-memory baseline. 32B: same games, Qwen3-32B executor, vs a 49.3% no-memory baseline.

Look at rows 3 and 5 together. fft-seed2-step35 is the strongest curator we trained as measured on the 8B executor, and it makes a 32B executor worse. fft-seed3-step5 — five GRPO steps of training — is the weakest-looking arm on 8B and the strongest on 32B, at 62.9% absolute (the paper reports 61.2% for its 32B configuration). If you only have budget to evaluate a few checkpoints, evaluate them on the executor you intend to ship.

Read this before using a checkpoint

Three findings from the sweep that produced these weights should change how you pick one:

1. Do not use the final checkpoint. In all five of our training runs, step 60 lands back at the no-memory baseline. Held-out lift peaks mid-run.
2. The peak checkpoint moves with the seed (step 20 / 30 / 35 / 55 across runs), so there is no transferable "best step". Sweep, don't guess.
3. Sweep on your target executor, not your training executor. Per-checkpoint lift on the 8B executor that generated the training data barely predicts lift on a 32B executor (pooled Pearson r = -0.20 over 24 checkpoint pairs; -0.68 within one seed). The best 32B-transfer curator we found is a step 5 checkpoint.

And the statistical caveat, stated plainly: across five sweeps we tested 50 checkpoint arms against one baseline. Family-wide Bonferroni sets the bar at p < 0.001 and our best arm anywhere is p = 0.0026, so no same-executor ALFWorld lift in this project survives multiple-comparison correction. Treat every per-arm number below as a selection statistic, not an effect size.

Absolute numbers are not comparable to the paper

Our ALFWorld no-memory baseline is 33.6%, versus the paper's 47.9% on nominally the same executor. We could not close that gap (prompt wording, retrieval, seeds, serving precision and decode parameters are all ruled out; the remaining suspect is the ReAct/atomic-verb interaction). Compare paired lifts against our baseline, not absolute success rates against the paper's.

Training setup

Framework	TRL 1.4 GRPOTrainer + accelerate + DeepSpeed ZeRO-3 + vLLM colocate
Hardware	8×H100 (curator) + remote inference for executor and judge
Environment	real ALFWorld (AlfredTWEnv)
Executor	Qwen3-8B, frozen, temp 0.6 / top_p 0.95 / top_k 20
Judge (r_cnt)	Qwen3-32B
Schedule	60 GRPO steps, `
Seeds	42 (seed-1), 123 (seed-2), 456 (seed-3)
Wall clock	~2.9 days per run

TRL is not 1:1 with verl (advantage normalisation, sampling semantics, buffer handling), and this is documented as a deviation rather than hidden — see DIVERGENCES.md #14. The reproduction ran both frameworks precisely so that difference could be tested; the checkpoint-curve shape reproduces in both.

Provenance

• Code, full report and every divergence from the paper: https://github.com/belt-sh/skillos
• Findings write-up with figures: docs/repro_report.md
• Paired eval rollouts, so every number here can be recomputed without a GPU: inference-sh/skillos-alfworld-eval-arms
• Paper: SkillOS: Learning Skill Curation for Self-Evolving Agents (Ouyang et al., 2026)

This is an independent reproduction. It is not affiliated with or endorsed by the paper's authors.

License: Apache 2.0, inherited from Qwen/Qwen3-8B.