code/report.md

Reasoning decision model: results

Hypotheses

#	Result	Numbers	Confirmed
H1	joint readout vs gold-only, in-distribution accuracy	+0.1 pts [-2.0, +2.2]	no
H2	disagreement AUROC, joint vs others (need +0.03)	vs gold_only +0.012 [-0.002, +0.023]; vs base +0.011 [-0.003, +0.024]	no
H3	joint reasoner vs base reasoner, all eval	+0.0 pts [-0.8, +0.7]	yes
H4	gap recovered at f = 0.3 (need 80%)	52.7% of 69.5 → 84.7; learned router 51.3%	no

Readout

Accuracy (%) after per-type temperatures fitted on the calibration set; ECE uses 15 bins.

Readout	Procedural	tasksource	OpenDecisions	LocalLLaMA	LocalLLaMA (panel agrees)	OD Capability	OD Capability (raw)	In-dist ECE	In-dist NLL
base	46.8	60.4	66.5	63.0	74.1	61.8	55.2	0.042	1.027
gold_only	72.4	65.8	72.1	67.5	76.7	66.0	65.1	0.032	0.774
votes_only	71.7	66.3	71.9	66.3	75.1	66.4	62.3	0.041	0.780
joint	71.9	66.4	72.2	66.5	76.3	66.7	64.6	0.031	0.757

Temperatures (choice / yes-no / score): base 1.71 / 2.41 / 1.28; gold_only 1.18 / 0.87 / 1.02; votes_only 1.60 / 1.08 / 1.48; joint 1.28 / 1.18 / 1.05

Reasoning (one sample per item)

Reasoner	Procedural	tasksource	OpenDecisions	LocalLLaMA	All	Median tokens	p90 tokens	Looped	Letter = readout after chain
base	95.1	73.3	94.9	71.0	83.9	1626	5737	1.0%	98.9%
joint	97.1	74.3	94.1	71.0	83.9	1412	5497	1.7%	98.3%

Does the readout agree with its reasoning?

Agreement = the readout's answer equals the same model's thinking-mode answer (the base reasoner for the readout-only ablations). The thinking call never sees the readout's answer, so this is also the faithfulness check of explanation on demand.

Readout	Agreement	Procedural	tasksource	OpenDecisions	LocalLLaMA	Disagreement AUROC	Fixable AUROC
base	66.8	46.8	67.9	67.8	70.6	0.778	0.580
gold_only	71.6	71.6	72.8	73.3	69.0	0.777	0.489
votes_only	73.1	71.4	76.2	72.9	72.7	0.803	0.562
joint	74.4	72.2	75.8	74.1	75.0	0.789	0.526

Routing

Route the lowest-margin share f of items to thinking mode (all eval sets pooled).

Readout	Router	f=0	0.1	0.2	0.3	0.5	1.0	tokens at 0.3	random at 0.3
base	margin	62.7	66.9	70.8	74.4	79.3	84.3	1095	69.2
base	learned	62.7	67.3	71.1	74.5	79.5	84.3	1138	69.2
gold_only	margin	69.8	73.1	75.3	77.5	80.4	84.4	1134	74.2
gold_only	learned	69.8	72.7	75.1	76.8	80.4	84.4	1181	74.2
votes_only	margin	69.2	72.6	75.7	78.1	81.7	84.4	1177	73.8
votes_only	learned	69.2	72.9	75.5	78.2	81.7	84.4	1204	73.8
joint	margin	69.5	72.5	75.4	77.5	80.9	84.7	1244	74.0
joint	learned	69.5	73.0	75.0	77.3	81.1	84.7	1275	74.0

The deployable system

Readout by default; items with the lowest readout margin go to thinking mode, whose probabilities are the readout after the chain (temperatures fitted on calibration chains only). All eval sets; OpenDecisions Capability with the official scorer.

Model	Mode	Routed	Procedural	tasksource	OpenDecisions	LocalLLaMA	All	OD Capability	Mean generated tokens
base	readout only	0.0%	46.8	60.4	66.5	63.0	62.7	61.8	0
base	routed	10.0%	58.6	64.3	71.1	65.0	66.9	67.2	367
base	routed	20.0%	67.3	66.1	76.0	67.1	70.9	72.3	730
base	routed	30.0%	79.3	68.1	80.4	68.1	74.5	76.5	1095
base	routed	50.0%	90.9	71.3	87.2	69.6	79.5	82.8	1718
base	thinking always	100.0%	96.0	74.0	95.4	71.7	84.5	84.4	2823
joint	readout only	0.0%	71.9	66.4	72.2	66.5	69.5	66.7	0
joint	routed	10.0%	77.8	68.3	75.9	68.1	72.5	69.8	429
joint	routed	20.0%	82.1	70.1	79.9	69.8	75.4	73.6	829
joint	routed	30.0%	86.5	71.5	82.9	70.4	77.5	77.1	1244
joint	routed	50.0%	92.0	73.8	87.8	71.9	80.9	83.2	1946
joint	thinking always	100.0%	97.5	74.5	94.6	72.5	84.7	88.1	2831

Post-chain temperatures (choice / yes-no / score): base 4.77 / 3.49 / 4.02; joint 4.08 / 2.94 / 3.65

Latency (batch size 1)

Model	Readout median	Readout p90	Reasoning median	Reasoning p90
base	43 ms	76 ms	10.1 s	41.8 s
joint	41 ms	74 ms	11.6 s	55.6 s

Run

• Throughput probe: 1957 generated tokens/s; chain tokens median 1878, p90 6414, max 32471; parse rate 99.2%.
• Generation: 3169 items, 12676 chains, 41.0M tokens; 2818 items with a correct chain; 154 chains looped to the context limit.
• Training joint: 2673 reasoner + 2996 readout examples, 8.7M tokens, 207/207 steps, 46 min (3137 tokens/s).
• Training gold_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 11 min (2521 tokens/s).
• Training votes_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 7 min (3898 tokens/s).
• Single seed (17); see results/DEVIATIONS.md for every change to the plan.