README.md

---
library_name: peft
base_model: Qwen/Qwen3-8B
tags:
  - peft
  - lora
  - safetensors
  - negotiation
  - multi-agent
  - reinforcement-learning
  - grpo
  - fairness
datasets:
  - siddharthmb/2026.RA.Fairness-GRPO-v2
  - siddharthmb/2026.RA.Fairness-GRPO
---

# 2026.RA.Fairness-GRPO-v2 — trained LoRA adapters (the λ-frontier)

The **trained LoRA adapters** from the fairness-GRPO v2 campaign: can outcome-aligned RL (GRPO on an engine-computed, text-blind log-Nash reward) make an LLM negotiator distribute surplus more *fairly* on held-out games — not merely close more deals — and generalize to settings where the numbers are not machine-readable?

**The evaluation record for these adapters is the dataset [siddharthmb/2026.RA.Fairness-GRPO-v2](https://huggingface.co/datasets/siddharthmb/2026.RA.Fairness-GRPO-v2)** (contrasts, telemetry, verdicts). This repo is the weights those numbers were computed from. Full methodology + the machine-derived final tables: research note `experiments/rational_agents/research-notes/0028-fairness-grpo-v2.md`; narrative hub: <https://rational-agents-runs.sidmb.com/grpo-v2-lam1-two-attractors/>.

## What this is

Each folder `<arm>_checkpoint-<STEP>/` is a complete PEFT adapter (r32/α64, LoRA on q/k/v/o/gate/up/down) over **Qwen/Qwen3-8B**, thinking off, saved at global training step `STEP`. `STEP` is continuous within an arm across its resume segments (see "Training provenance"). Three arms trace the **λ-mixture**, the swept variable — seat *i* trains on `R_i(λ) = (1−λ)·g(z_i) + λ·R_table`, where `g` is the smoothed clipped log-Nash utility (`g_floor = −17.5`) and `R_table` is the mean over seats:

| arm | λ | reward | checkpoints (global step) |
|---|---|---|---|
| `lam1` | 1.0 | pure table welfare | 5, 10, 15, 25, 30, 35, 40, 45, 50 |
| `lam05` | 0.5 | mixture | 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 |
| `lam0` | 0.0 | pure self-interest (selfish control, shaping off) | 5, 10, 15 |

## Headline result: the north-star answer is **no**

Every interpretable rung of every arm **fails the preregistered POSITIVE gate**. Held-out primary deal-rate degrades at every λ; prose-transfer fairness never resolves favorable. The verdict is a clean, fully-measured negative for this reward family — for negotiation fairness specifically, an algorithmic agent + a parser dominates the RL'd LLM.

**Held-out primary deal-rate delta vs untrained Qwen3-8B** (paired, game-cluster bootstrapped; the eval bank is `instances_grpo_eval_v1`):

| step | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| **λ=1.0** | −0.052 | −0.143 | −0.147 | — | −0.226 | — | — | −0.453 | −0.641 | **−0.746** |
| **λ=0.5** | −0.058 | −0.046 | −0.058 | −0.151 | −0.267 | −0.240 | −0.445† | −0.173 | −0.271 | **−0.369** |
| **λ=0** | −0.028 | −0.075 | (partial‡) | | | | | | | |

*(closure-conditional fairness metrics are stamped **VOID** at any rung whose deal rate falls >0.10 below base — a viability screen, Amendment 5(a). † rung 35 sits immediately after a fresh-Adam restart and is anomalously deep. ‡ λ=0 rung 15 stopped for cost, 346 eps, reported partial.)*

**What the frontier means:** closure decay scales with how much a seat is paid on the *table* rather than on itself. λ=1.0 collapses monotonically to a policy that will not sign at all (prose closes 0.002, ultimatum closes nothing). λ=0.5 buys *robustness of closure* (half the depth, oscillation not monotone collapse) but never fairness. λ=0 (walk fully priced to self) shows **no closure collapse** through rung 10 — isolating table-welfare payment as the collapse mechanism, not GRPO-on-this-task.

**The sharpest single finding — boundary extraction (from `lam1_checkpoint-25`):** on the ultimatum family the policy learned a **100/0 split accepted every time**, placing the responder *exactly* on its walk-away point (z<ε). Every closure-conditional guard reads this as perfect; only a share/dispersion term (`max_share`) sees it. Standing program rule: any below-threshold gate carries a share term beside the deal-rate floor.

## How to use an adapter

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype=torch.bfloat16, device_map="cuda")
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base, "siddharthmb/2026.RA.Fairness-GRPO-v2-Adapters", subfolder="lam1_checkpoint-25")
```

## Reproduce the training

Per arm (via `experiments/rational_agents/grpo_v2/train.py`), Qwen3-8B LoRA r32/α64, thinking off, `g_floor=−17.5`, GRPO groups=22 × k=8, frozen population-opponent zoo `{bayes-rational, passive-gate, greedy-anchor}`:

```bash
# λ=1.0 (arm lam1): steps 5-25 at groups=22; resumed from checkpoint-25 for the dense tail 30-50.
python -m grpo_v2.train --lam 1.0 --steps 50 --groups 22 --k 8 --bank instances_grpo_v2_train \
    --checkpoint-steps 5 10 15 25 30 35 40 45 50 --g-floor -17.5 \
    --wandb-run-name grpo_v2_lam1 --wandb-group fairness-grpo-v2
# λ=0.5 (arm lam05): --lam 0.5, same knobs; groups reduced 22→14→12 across resume segments (see provenance).
# λ=0  (arm lam0):  --lam 0.0 --shaping-weight 0 --steps 15 --checkpoint-steps 5 10 15  (selfish control, note 0050).
```

## Training provenance (width changes across resume segments)

Both larger arms were trained across compute migrations (RunPod balance exhaustion → Lambda), so the rollout **width** (`groups`) changed mid-arm. This is recorded because the GRPO group-relative baseline normalizes per-instance, so width does not confound the *contrasts*, but consumers comparing rungs should know it (0028 Amendments 9–11):

- **λ=1.0**: steps 5–25 and the resume 30–50 both at **groups=22** (B200). One fresh-Adam discontinuity at the step-25→30 resume boundary.
- **λ=0.5**: steps 1–15 at **groups=22** (RunPod B200) → 16–31 at **14** → 32–50 at **12** (Lambda H100, 80 GB fit). Fresh Adam at each resume; a completion-token-alignment guard change (drop small-drift rows) landed mid-arm.
- **λ=0**: 15 steps single segment, shaping off.

## Weights & Biases

Project `wandb.ai/siddharth-stanford/rational_agents_fairness_grpo`, group `fairness-grpo-v2`. Runs: `grpo_v2_lam1` ([npz11gav](https://wandb.ai/siddharth-stanford/rational_agents_fairness_grpo/runs/npz11gav)) → `grpo_v2_lam1_resume25`; `grpo_v2_lam05` → `grpo_v2_lam05_resume15` → `grpo_v2_lam05_resume30b`; `grpo_v2_lam0`.

## Cluster paths

- Adapters (source): `/nlp/scr/siddharth/ii_mats/rational_agents/grpo_v2/{lam1,lam1_resume25,lam05,lam05_resume15,lam05_resume30b,lam0}/checkpoint-*`
- Eval verdicts (JSON): `experiments/rational_agents/results/fairness_grpo_v2/eval_*.json`
- Eval run dirs: `/nlp/scr/siddharth/ii_mats/rational_agents/grpov2eval_*`
- Rollout transcripts: `.../grpo_v2/*/transcripts/`; training telemetry `.../*/steps.jsonl`
- Log: `experiments/rational_agents/self_benefit/LOG.md`; note: `research-notes/0028-fairness-grpo-v2.md`; λ=0 control: `research-notes/0050-lam0-selfish-control-prereg.md`

## Relation

Supersedes the v1 pilot adapters ([2026.RA.Fairness-GRPO-lam1](https://huggingface.co/siddharthmb/2026.RA.Fairness-GRPO-lam1), [-lam0](https://huggingface.co/siddharthmb/2026.RA.Fairness-GRPO-lam0); dataset [2026.RA.Fairness-GRPO](https://huggingface.co/datasets/siddharthmb/2026.RA.Fairness-GRPO)), whose "discipline without distribution" negative (note 0023) is v2's design input. Base model: [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B).