qwen2.5-3b-grpo-finqa

A LoRA policy from GRPO + RLVR on FinQA (SEC-filing financial reasoning with checkable numeric answers). The reward is verifiable ground truth - is the final number correct, after normalizing percent / scale / sign - so there is no reward model and no value model (beta=0, group-average baseline). The online arm of finqa-rlvr.

result

FinQA validation accuracy (greedy, n=200): 0.530 -> 0.525 (-1 question). Flat, and within noise at this eval size. The honest finding from the project is that at this scale method choice is second-order: the reward design and the base-model headroom dominate, not online-vs-offline. GRPO here ran at a conservative lr=1e-6 that decayed to ~0, so the flatness is partly an optimization-budget artifact - noted, not chased. See the repo for the full teardown and the offline (DPO) counterpart trained on the same signal.

training

GRPO (TRL, use_vllm colocate), LoRA r=16 / alpha=32, group size 8, beta 0, correctness + format reward, 600 steps, 1x A100.

use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen2.5-3b-grpo-finqa")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")