Mercity/Desicion-Model-Custom/code - Storage Bucket

decision-reasoner

A Jev-style typed decision model on Qwen3.5-4B that can also reason, trained on one A100 80GB.

• Readout (fast mode): state, question and lettered options in one prompt with thinking off; one forward pass; softmax over the K option-letter logits. About 43 ms per decision at batch size 1.
• Thinking mode: the model writes a <think> chain; its decision is the letter softmax read right after the chain, so the decision is the one the reasoning leads into.
• Routing: low-confidence readouts go to thinking mode (margin router, plus a learned agreement router fitted on the calibration set).
• Training: one LoRA adapter (rank 32, all attention, linear-attention and MLP projections) trained jointly on the model's own shortest correct chains (reasoner) and on a readout target that mixes gold with the reasoner's vote share: q = 0.5 * gold + 0.5 * votes.

Status (Nov 1, 2026): finished

Model	Fast readout	Thinking	Routed 30% (thinking tokens / question)	S1MB Task Avg
Qwen3.5-4B, untrained	62.7%	84.0%	73.0% (1,070)	32.8
v1	69.5%	84.4%	78.0% (1,166)	43.5
V2	72.7%	81.2%	81.8% (1,749)
V2 + RL (final)	72.5%	83.1%	81.8% (729)	47.9; 55.2 routed
2B hybrid (MiniCPM5-2B)	66.0%	79.4%	75.6% (253)	37.3

Fast readout on 5,442 eval questions; thinking and routing on a 1,498-question subset. Full write-up: results/report/decision_model_report.pdf (23 pages; source results/report/REPORT.md, built by analysis/build_report_pdf.py), short version results/FINAL_REPORT.md. Deployable models in the bucket: run2_v2/checkpoints/rl60_merged (4B) and run3_2b/checkpoints/v2b_ro_merged (2B readout; thinking with the untouched openbmb/MiniCPM5-2B).

The design below is v1's; what V2, RL and the 2B changed is in the report and results/DEVIATIONS.md.

Plan: the Claude Docs page "Reasoning decision model: A100 training plan". Every change to it is in results/DEVIATIONS.md; observations as they happened are in results/NOTEBOOK.md; open work in TODO.md.

Files

File	What it does
common.py	prompt format, letter ids, answer parsing, metrics helpers
prep_data.py	train pool, eval and calibration sets from the Hub; every exclusion counted in data/summary.json
vllm_utils.py	vLLM readout (letters only, exact K-way softmax), streaming thinking sampler with deadlines, post-chain readout
gen_reasoning.py	throughput probe and time-boxed generation of 4 chains per pool item (data/gen.jsonl)
train.py	joint and readout-only LoRA training; chunked, checkpointed loss; whole examples only
merge_lora.py	adapter to bf16 checkpoint (trained models are served merged: vLLM 0.32 ignores LoRA on Qwen3.5)
evaluate.py	predictions (results/predictions/) and metrics
metrics.py	accuracy, calibration, OpenDecisions Capability, agreement, AUROCs, routing, system view, H1-H4
report.py	results/report.md
od_score.py	OpenDecisions' MIT score.py, vendored unchanged after review
run_all.sh	the whole run with stage markers, deadlines, resume and bucket sync; start inside tmux
preflight.sh	every stage on tiny sets with Qwen3.5-0.8B
setup_variant.sh	a second run (e.g. a 2B model) in runs/<name> with the same code and items
prep_data_v2.py, runs/v2/run_v2.sh	V2 data (42K pool, dev slice) and pipeline: speed benchmark, LR sweep, joint run, ablations, eval
teacher_gen.py, bench_speed.py	teacher chains via OpenRouter (key read from outside the project); training-speed benchmark
metrics_v2.py	V2-era metrics (temperatures, ECE, routing, gap recovered, disagreement AUROC)
rl_fresh.py, rl_select.py, rl_grpo.py, runs/v2/run_rl_eval.sh	RL question pool and screening, GRPO (TRL 1.17.1), merge + eval
runs/v2_2b/run_2b.sh, runs/v2_2b/twob_extra.sh	the 2B pipeline (probe, pick, sweep, train, eval) and the readout-only variant
jevals_eval.py	Jevals suite 0.1.0: rebuild (hash-checked), fast and thinking runs, scoring
s1mb_eval.py, s1mb_think.py	S1MB: items, fast readout runs, a reimplementation of its scoring (checked against published scores), routed runs
analysis/	bootstrap CIs and hypotheses (final_stats.py), error analysis (examples_*.py), yes/no offset, report figures (fig_*.py), PDF builder
dashboard/	the run dashboard (build_data.py + template.html)

Run

uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install vllm peft datasets accelerate scikit-learn pandas pyarrow hf_transfer flash-linear-attention kernels
./preflight.sh                                  # ~15 min
tmux new -s run 'TOTAL_H=8.67 ./run_all.sh >> logs/run_all.log 2>&1'

Outputs sync to the public HF Storage Bucket Mercity/Desicion-Model-Custom: run1/ (v1), run2_v2/ (V2 + RL), run3_2b/ (2B), jevals/, s1mb/, report/, analysis/, code/ (this repository). No keys are synced.