musicakamusic/emotion-probes

CrimsonRed — Cross-Architecture Emotion-Prime Steering Replication

Replication of the emotion-prime steering protocol from arXiv:2607.18691 (NSM semantic primes as explanans for emotion in LLMs), extended across four architectures. Generated by scripts/paper_faithful_steering.py in the CrimsonRed project.

The finding

The paper's core claim — that semantic-prime recipe directions steer emotion more strongly than Scherer appraisal directions — replicates cross-architecture when the injection protocol is matched to the reference code, not the paper's prose:

Model	Layer	prime_recipe	single_appraisal	Operating-regime ratio	Paper: 2.9×
Llama-3.2-1B	11	+0.26	+0.15	2.80×	✅
Qwen3.5-2B	18	+0.09	+0.01	2.95×	✅
Gemma-4-E2B	15	+0.04	−0.03	primes ≫ appraisals*	✅
Gemma-4-E4B	20	pending	pending	pending	TBD

⚠ Refresh note (2026-08-15, FINAL — 10k-permutation statistics). The July table above is from the pre-audit source-mode runs (probe pair-suite v1, uncalibrated layers, n_perm=1000). The 2026-08-15 runs supersede it: template-held-out probe splits (no template spans fit/eval), n_perm=10000 (p floor 0.0001), n_bootstrap=1000, fresh probe caches:

Model	Layer	Permutation p (guilt/anger/joy/sadness)	Aggregate ratio CI	Verdict
Llama-3.2-1B (paper's reference)	11	.0077 / .0001 / .0130 / .0204	[1.23, 10.09] excl. 1	full replication
Qwen3.5-2B	16	.0001 / .0001 / .0002 / .9956	[0.11, 26.7] excl. 0	replicated (3/4 targets)
Gemma-4-E2B	24	.79 / .13 / .97 / .0175	spans 0	partial (sadness only)
Gemma-4-E4B	29	.10 / .63 / .96 / .97	spans 0	not replicated

Llama-1B also replicates the paper's §7.6 fluency claim: the behavioral prime-advantage survives AND widens under PMI and contextual calibration (+0.015 → +0.058 → +0.062). §6.8 direction stability (probe-bootstrap, n=20): mean cosine .96–.97, min single prime .90.

Note the honest deflation: E2B anger was p=.012 under the old random-split probes and is p=.13 under template-held-out splits — the earlier number was flattered by near-duplicate templates spanning the split. E4B's negative stands as-is. July numbers remain valid as historical source-mode results.

*On Gemma-E2B, appraisals steer away from the target (negative on 3/4 emotions). The ratio is undefined (negative denominator), but primes ≫ appraisals is unambiguous — stronger than a 2.9× advantage.

The prose–code discrepancy (methodological contribution)

The paper's §4 describes the injection as "a unit-normalized direction, scaled by the layer's mean residual norm, at all token positions." The reference code (emo-llm/main.py:469-492) does something different:

Aspect	Paper prose (§4)	Reference code
Direction	unit-normalized	raw probe weight (unnormalized)
Scale	β × mean residual norm	β × 1.0 (fixed)
Token positions	all	last token only
Layers	single (L11)	span of 3 (L10, L11, L12)
Contrastive removal	not mentioned	disabled (Beta2 = 0.0)

Following the prose produces a false negative (Qwen 0.82×, Gemma 0.91× — apparent "Llama-specificity"). Following the code reproduces the paper's ~2.9× on every model.

Protocol (source mode)

• Injection: HF register_forward_hook, post-MLP residual, adds β × raw_probe_weight at the last token over a 3-layer span.
• Directions: 32 primes (L2 logistic, C=1.0, 210 contrastive pairs/prime, held-out acc 0.987) + 20 appraisals (Ridge α=5.0 on enVent 6,800 events) + 13 emotions (L2 logistic on enVent labels).
• Metric: target-emotion logit shift at the answer position of Tak et al.'s 2-shot classification prompt. Selectivity clamped to [−1, 1].
• Operating-regime ratio: prime/appraisal shift ratio over (target, β) cells where the emotion ceiling shift is positive.
• Targets: guilt, anger, joy, sadness.

Files

Path	Description
summary_2026-07-27_source_mode.json	Cross-architecture source-mode summary
docs/METHODOLOGY_AUDIT_PAPER_FAITHFUL_2026-07-27.md	Full methodology audit vs. the paper
docs/PAPER_DRAFT_CROSS_ARCH_PRIMES_2026-07-27.md	Publication draft
results/source/paper_faithful_llama1b_L11_source_cal.json	Llama L11 source, β=0.01–0.2
results/source/paper_faithful_qwen35_L18_source.json	Qwen3.5-2B L18 source
results/source/paper_faithful_gemma_e2b_L15_source.json	Gemma-4-E2B L15 source
results/source/paper_faithful_gemma_e4b_L20_source.json	Gemma-4-E4B L20 source (appended when run completes)
results/papermode/paper_faithful_llama1b_L11_sweep_v2.json	Llama paper-mode v2 (corrected metrics)
results/papermode/paper_faithful_llama1b_L11_paperverse.json	Llama paper's β=0.5–2.0 (collapse)
legacy/…	Earlier paper-mode results, retained for provenance

Reproduce

python scripts/paper_faithful_steering.py \
  --model <path> --layer <L> --inject-tokens last --layer-span 3 \
  --beta 0.01 0.02 0.05 0.1 0.2 --mode source --pairs-per-prime 210 \
  --output data/<name>.json

Generated by CrimsonRed, scripts/paper_faithful_steering.py. AGPL-3.0 license.