Constitutional Midtraining: Content Presence Drives Alignment Gains
Abstract
Post-training alignment is often shallow, eroding under fine-tuning. Whether midtraining interventions, cleanly isolated from post-training, can produce durable alignment remains untested. We test this via constitutional midtraining: inserting principled, values-based content into midtraining against a replay-only control at 120B scale. Our 394M-token constitutional corpus, built from Anthropic's Constitution, uses a 2x2 factorial design (curriculum ordering x deliberative reasoning) to produce four constitutionally midtrained conditions plus a control, evaluated on self-generated and established benchmarks including alignment under pressure, value conflict resolution, blackmail, and emergent misalignment across three stages: post-midtraining, post-SFT, and post-benign fine-tuning. Constitutionally midtrained models outperform the control on alignment generalization and durability, notably on blackmail: SFT instills a blackmail propensity in all models, but constitutional midtraining blunts it, with the advantage surviving benign fine-tuning (-17.5pp). This durability does not extend to settings requiring active resistance to in-context pressure or conflict, where the advantage attenuates after SFT. The presence of constitutional content at midtraining also matters more than its structure, and constitutional midtraining incurs no cost, on average, on the capabilities we test (MMLU, ARC-Easy, piqa, GSM8K) at any stage. A modest amount of constitutional content at midtraining could therefore yield broad, persistent alignment gains, offering a cheap, complementary addition to SFT-centered pipelines. Code, data, and models are available.
Community
We study Constitutional Midtraining, an earlier-stage approach to LLM alignment that inserts principled, values-based content before post-training. Using a 394M-token constitutional corpus based on Anthropic’s Constitution, we midtrain 120B-parameter models and evaluate whether the resulting alignment generalizes and persists through SFT and benign fine-tuning.
🌟 Constitutionally midtrained models show stronger alignment generalization and durability than the control, including substantially less blackmail across all evaluation stages. The advantage persists after benign fine-tuning, with a 17.5 percentage-point reduction in blackmail at the final stage.
🌟 We release a 394M-token constitutional corpus, the complete data-generation and evaluation pipeline, and a suite of matched 120B model checkpoints spanning five training conditions and three evaluation stages.
🌟 Constitutional midtraining preserves performance on the capabilities we test, including MMLU, ARC-Easy, PIQA, and GSM8K—suggesting that these alignment gains do not require a capability tax.
🌟 Curriculum ordering and deliberative-reasoning structure generally provide null or transient benefits. Our results suggest that the presence of constitutional content matters more than how that content is structured.
🌟 The gains are not universal: improvements weaken after SFT in settings requiring active resistance to in-context pressure or value conflict. Constitutional midtraining therefore appears most effective at shaping durable, aligned default behaviours and may serve as a complementary addition to safety post-training.
Accessible summary: https://www.lesswrong.com/posts/n5htoDGvKKJFAjji2
arXiv: https://arxiv.org/abs/2607.26654
GitHub and benchmarks: https://github.com/desBugger/constitutional-mt
Data and models: https://huggingface.co/collections/cho-ai/constitutional-midtraining
Get this paper in your agent:
hf papers read 2607.26654 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper