dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-pm2-pmx16-sr2-fob

PM2-CODE trained as a true first-order gradient flow (FO-b-PM2-CODE): 1.6% ahead of its second-order twin on code (Gen 3)

The question of this collection: does the conservative core of Semantic Simulation need second-order dynamics (a velocity and inertia), or does a first-order model trained the same way do as well? Each pair trains one model both ways, with everything else the same.

This model: PM2-CODE (2 Poisson-mode wells, Python code) trained as a true first-order gradient flow: the frozen spring of the wells and the stiff modes relaxed exactly, no velocity anywhere. Verdict: 1.6% ahead of its second-order twin (5.62 against 5.71; paired by step −1.55% over the last eight evaluations, spread 0.14%). Without wells first order trails by 9.2% on code, its largest gap on any corpus; two wells turn it into a lead, as they close it on OpenWebText and TinyStories. The first-vs-second-order collection holds the pairs.

Result

pair	corpus, tokens	second order	first order	first order against second
OpenWebText, no wells	532.5M	SR2 58.32	FO-b-OWT 61.16	+4.9%
OpenWebText, with 2 Poisson-mode wells	532.5M	PM2-PMX16 54.71	FO-b-PM2-PMX16 53.81	−1.6%
TinyStories, true first order (FO-b), no wells	315M	SO-TS 5.16	FO-b-TS 5.28	+2.3%
TinyStories, with 2 Poisson-mode wells	315M	PM2-TS 4.99	FO-b-PM2-TS 4.96	−0.5%
Python code, true first order (FO-b), no wells	315M	SO-CODE 6.00	FO-b-CODE 6.55	+9.2%
Python code, with 2 Poisson-mode wells	315M	PM2-CODE 5.71	FO-b-PM2-CODE 5.62	−1.6%

Without wells the gap grows with the corpus, from TinyStories through OpenWebText to code. With two Poisson-mode wells it closes on all three, and on code the first-order model is ahead.

• This model: settled 5.62 (mean of the evals at steps 18,000, 18,500 and 19,000: 5.52, 5.63, 5.71); best 5.52 at step 18,000. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split, so no file is in both).
• Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512) in one session: the ladder's WSD schedule over 16,250 steps, then 3,000 steps at its floor learning rate, as SO-CODE and FO-b-CODE. All four code models start from step 0 at seed 0 on the same batches, so their evaluations are paired at every step (this pair's step-to-step changes correlate at 0.987).
• The wells, against the same model without them: −14.2% against FO-b-CODE (6.55). They help first order about three times as much (log ratio 0.153 nats against 0.049 for second order).
• Against a transformer: the matched GPT-2 (d = 384, L = 8) on the same code tokens and schedule settles at 4.96; this model is +13.3% behind it (SO-CODE +20.9%). That model is not on the Hub yet.
• Pre-registered (protocol §5.13, the code set MS5, calls written before any code arm trained): CODE3 (FO-b-PM2-CODE within ±2% of PM2-CODE, paired by step), called 65%: HIT, −1.6% on the settled values. CODE4 (the wells help first order more: log(FO-b-CODE / FO-b-PM2-CODE) exceeds log(SO-CODE / PM2-CODE)), called 70%: HIT, 0.153 against 0.049 nats. CODE5 (PM2-CODE settles more than 1% below SO-CODE), called 55%: HIT, −4.7%. With CODE1 hit on the pair without wells, the set's MS5 call (45%) hits. Scored on the letter.
• Run health: 19,250 steps in 6.34 h in one session (1.19 s per step); no watchdog trigger or spike capture; largest rise between logged training losses 0.073, at step 13,050 as in SO-CODE (the batches are paired); gradient norm median 0.47, with peaks up to 8.4 near step 2,350 in the wells' parameter group, which is clipped to 0.3 as a group; SCAF leak monitor CLEAN at all 4 audits.

How the pair compares through training

step	phase	PM2-CODE (second order)	FO-b-PM2-CODE (first order)	first order against second
500	warmup	42.32	42.32	+0.0%
2,000	stable	15.12	13.45	−11.0%
5,000	stable	8.24	8.06	−2.2%
10,500	stable, end	6.54	6.49	−0.8%
13,000	decay	6.04	5.98	−1.0%
16,000	decay, end	5.82	5.75	−1.2%
17,000	floor	5.78	5.69	−1.6%
18,000	floor, best	5.60	5.52	−1.4%
19,000	floor, last eval	5.80	5.71	−1.6%

• First order leads throughout: paired by step, −4.21% over steps 500–3,000, −0.87% at the end of the stable phase, −0.90% in the decay and −1.58% at the floor; FO-b-PM2-CODE is lower at 37 of 38 evaluations. The early lead is the one every first-order model shows; with wells it does not turn around.

What the wells learn (Cell 6b-15)

reading	PM2-CODE (second order)	FO-b-PM2-CODE (first order)	TinyStories: second; first	OpenWebText: second; first
median depth, layer 0 / layer 1	-0.19 / +1.09	-0.12 / +1.13	-0.20 / +1.29; -0.27 / +0.97	-0.21 / +1.23; -0.40 / +0.93
half-life, p05 / median / p95 (tokens)	9.0 / 18.2 / 27.5	8.0 / 12.1 / 16.1	median 14.0; 13.1	median 14.2; 8.0
wells' force against the conservative force, layer 0 / layer 1	0.80 / 3.23	0.40 / 3.10	0.67 / 3.44; 0.83 / 2.29

The two wells learn the same landscape on all three corpora: repulsive at the first layer, attractive at the second, with median half-lives of 8 to 18 tokens; neither well is dead. At the second layer the wells' force is three times the rest of the conservative force.

Stiffness and the step (Cells 6b-13 and 6b-9)

model	median ω·Δt of V_θ's stiffest modes at the endpoint	share past the explicit-integrator wall (2.0)
SO-CODE (second order, no wells)	7.55	99.97%
PM2-CODE (second order, 2 wells)	4.85	99.98%
FO-b-CODE (first order, no wells)	0.86	22.9%
FO-b-PM2-CODE (first order, 2 wells)	1.09	11.2%

• The wells take load off the stiff modes in the second-order model, as on OpenWebText; the first-order models keep a soft potential either way.
• Cell 6b-9 (FO-b-PM2-CODE): how far the step is from the plain V_θ geodesic step, 0.88 of its length on average; the wells remove 0.19 of that, the post-step normalisation 0.22 and the pair potential V_φ 0.0045. At the second layer the step is mostly the wells'.

Off the trained schedule (Cell 6b-7)

reading	PM2-CODE (second order)	FO-b-PM2-CODE (first order)
Gate 1: perplexity cost of resetting the velocity at inference	+6,308%	0 (no velocity carried)
Gate 2: one extra layer at the trained step	+451%	+0.4%
Gate 2: eight layers	+2,135%	+7,719%
Gate 3: three steps over the trained time, nats	0.828	0.624

• With two wells the second-order model leans heavily on its velocity: resetting it costs +6,308%, against +4.0% for SO-CODE without wells (OpenWebText at two wells: +5,150%). Its first-order twin, which has no velocity, is ahead.
• Neither model refines: the penalty grows away from the trained two steps, as for every model trained at one step count; the first-order model's penalty is the smaller.

CPU readings

• The wells are positional (pm_mode_usage.py): both wells carry force at both layers (effective number 1.7–2.0 of 2), and a token's position alone explains R² ≥ 0.999 of the wells' force in both models, as on OpenWebText and TinyStories.
• A start-of-sequence clock plus a fixed landscape (pm_static_landscape.py): holding every well's occupation at its late value from token 128 on changes perplexity by +0.01% (PM2-CODE) and +0.02% (FO-b-PM2-CODE); from token 0, by +2.7% and +2.0%, almost all of it in the first 64 tokens. Removing the wells costs +251% and +207% (OpenWebText at two wells: +100% and +831%).
• What carries the context (mechanism_ablations.py, 16 sequences of 512 tokens): removing the pair potential V_φ costs +2.8% in PM2-CODE and +8.5% in FO-b-PM2-CODE, three times as much (OpenWebText at two wells: +2.8% and +3.3%). Swapping the three long context channels ξ with another sequence's costs +1,325% and +1,545% (OpenWebText: +337% and +351%); swapping the wells' occupations costs nothing (+0.04% and +0.07%). On code the first-order model leans three times as hard on the pair potential, its one direct token-to-token path.
• Where the wells and first order gain (copy_split.py, 262,144 validation targets): the wells' gain over SO-CODE, −0.047 nats per token, is in every kind of target except whitespace and grows with the distance to the target's earlier occurrence (−0.024 nats at 1–16 tokens back, −0.095 at 257–512). First order's lead over PM2-CODE, −0.016 nats, is the same in every group (ratios 0.97–0.99). Without wells first order lost on identifiers (+0.152 nats against SO-CODE); with them its loss on identifiers (2.735 nats) is below both second-order models' (PM2-CODE 2.755, SO-CODE 2.821). On these tokens FO-b-PM2-CODE (6.04) is also ahead of FOCK-CODE (6.09), the register path.
• Refinement inside a layer (FLOW-R, refinement_flow_confirmation.py, kind code, extended 2026-11-10 to freeze the wells' quadratic per layer): PM2-CODE's per-layer flow levels off about 0.33 nats from the trained step (penalties 0.372, 0.341 and 0.331 nats at 8, 12 and 16 steps) and misses the convergence criterion narrowly (successive changes 0.010 and 0.031 against 0.01 and 0.02), as PM2-PMX16 does on OpenWebText; SO-CODE converges, at 0.37. FO-b-PM2-CODE's drifts upward (0.621, 0.671, 0.692) and does not converge, as for every first-order model. Across the whole stack, 8 steps cost 2.73 and 1.72 nats: the first-order model is the better of the pair across the stack and the worse inside a layer.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
POISSON_MODES=2, POISSON_WELLS_EXACT=True,   # PM2-PMX16's two wells
FIRST_ORDER_OVERDAMPED=True,              # tag 'fob'
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7, 6b-15 and 6b-13 and 6b-9. The CPU readings are debug/pm_mode_usage.py, debug/pm_static_landscape.py, debug/mechanism_ablations.py, debug/copy_split.py and debug/refinement_flow_confirmation.py (kind code), on the downloaded run folders.

Training artefacts were written to semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_pm2_pmx16_sr2_fob_st16250_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, the first-vs-second-order switch, result and diagnostics
checkpoints/fock_cfc_*_step18000_best.pt	weights at the best eval (step 18,000), optimizer state stripped
results/	the training log, the diagnostic cells' outputs, the CPU readings and the copy split (copy_split_report*)
code/	the notebook and the model code the switches belong to (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	§5.13: the code set (MS5), its calls CODE1–CODE5, these results
Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md	what the equations say about when second order earns its keep
Competing_Mechanisms_Families_Dominance_and_Substitution.md	the mechanisms that compete for a token's step, and which can stand in for which

Caveats

• One seed per model. The pairing removes the evaluation noise, not the training-seed spread, which has not been measured.
• CODE3 is scored on the settled values. Step by step the twins are within ±2% at 31 of 38 evaluations; the exceptions are early (steps 1,500–5,000), where the first-order model is ahead by more.
• The hyperparameters were tuned for the second-order models on OpenWebText (learning rate 1.2e-3, the WSD fractions, the clip groups).
• L = 2, and perplexities on code are not comparable with OpenWebText's or TinyStories'.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-10.