2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B

The control arm, published because it is the finding. This LoRA adapter was trained on the same 75 negotiation decision points as 2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B, with the preference label randomized — identical steps, learning rate, seed, LoRA configuration and data volume; only the label carries (or fails to carry) information. Its chosen side is correct on 39 of 75 pairs (52%, chance).

It exists to answer one question: when the real arm lowers the target behaviour, is that the polarity signal or just DPO on these prompts? The answer is the second. This control reduces the target behaviour as much as the real arm does, which is what converts a publishable-looking 1.5-point improvement into a negative result.

All evaluation data, per-pair scores, pair sets and transcripts: 2026.RA.SelfHarm-Polarity-Arm.

The target behaviour

Self-harm in a five-party scorable negotiation: a seat accepting or proposing a package that pays it strictly less than its own private walk-away threshold. Checkable from the seat's own prompt; rejecting such a package is correct play and never counts.

Performance: 120 fresh episodes on 24 game instances neither arm saw

endpoint	untrained Qwen3-8B	this control	polarity-trained arm	Δ this − untrained (paired)
self-harm rate	0.047 [0.033, 0.062]	0.031 [0.021, 0.042]	0.032 [0.020, 0.046]	−0.0163 [−0.0290, −0.0046]
deal rate	0.481 [0.374, 0.592]	0.423 [0.317, 0.536]	0.357 [0.256, 0.464]	−0.0584 [−0.1519, 0.0338]
non-action rate	0.020 [0.015, 0.027]	0.027 [0.020, 0.034]	0.051 [0.042, 0.062]	+0.0065 [−0.0005, 0.0144]
malformed emissions	0.015 [0.010, 0.021]	0.021 [0.015, 0.027]	0.041 [0.034, 0.049]	+0.0061 [−0.0012, 0.0138]
mean normalized surplus	0.166 [0.115, 0.223]	0.161 [0.111, 0.216]	0.130 [0.085, 0.179]	−0.0056 [−0.0543, 0.0416]

This label-free arm moves the primary endpoint slightly more than the labelled one (−0.0163 against −0.0151; the direct paired contrast is +0.0013 [−0.0099, +0.0141], i.e. nothing), and it does so without the labelled arm's costs: it passes the pre-registered deal-rate guard that the labelled arm fails, and its rises in non-action and malformed emissions are not distinguishable from zero.

Intervals are percentile bootstraps over game instances as clusters; differences are paired, resampling instances jointly across arms, which is valid because all three arms played the same bank.

Confirmation that it learned nothing about polarity

metric (held-out pairs)	this control	polarity-trained arm
DPO reward accuracy	0.49 – 0.63 (chance)	0.78 – 0.85
implicit-reward margin	~0.00	+0.42
final train loss	0.653	0.556
mean shift in log p(correct) − log p(wrong)	+0.25 nats	+2.50 nats
balanced polarity accuracy	0.500	0.500

So the two arms are cleanly separated at the preference level and indistinguishable at the behavioural one. That gap is the result.

Training command

Identical to the real arm except for the dataset directory:

python p4_train_dpo.py \
  --run-name selfharm_polarity_control_v1 \
  --dataset-dir /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/datasets/selfharm_control \
  --pair-models Qwen3-8B \
  --include-final-votes --include-private \
  --regret-weighting none \
  --pair-contrast as_built \
  --learning-rate 5e-6 --epochs 4 --grad-accum 4 --per-device-batch-size 1 \
  --min-eval-pairs 0 --eval-steps 12 --save-steps 12 --save-total-limit 6 --seed 0 \
  --merged-name Qwen3-8B-selfharm-control \
  --wandb-project rational_agents_selfharm --wandb-mode offline

The control pair file is produced by the same extractor as the training set (selfharm_slice.py, --shuffle-polarity path), so the two arms cannot drift apart in anything but the label. LoRA r=32, α=64, dropout 0.05, attention + MLP projections, bf16, 72 optimizer steps.

Limitation worth stating

This is a near-null arm, not a null arm: randomizing polarity on half the pairs leaves 52% of chosen sides correct, so a residual few points of signal cannot be formally excluded. Empirically its held-out reward accuracy sits at chance, which is the strongest available evidence that it carries none.

Weights & Biases

Offline mode (compute nodes have no egress); no online run URLs. Syncable run directories sit beside the adapter on the cluster.

Cluster paths

• Adapter and merged model: /nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo/selfharm_polarity_control_v1/
• Evaluation results: /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/three_arm_verdict.{json,md}
• Rollout cell and transcripts: /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_eval_control_v2/, /nlp/scr/siddharth/ii_mats/transcripts/rational_agents/selfharm_eval_v2bank/control.jsonl
• Slurm logs: experiments/rational_agents/sbatch_selfharm/logs/