dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2-fob

SR2 on TinyStories trained as a true first-order gradient flow (FO-b-TS): second order wins without wells here too (Gen 3)

The question of this collection: does the conservative core of Semantic Simulation need second-order dynamics (a velocity and inertia), or does a first-order model trained the same way do as well? Each pair trains one model both ways, with everything else the same.

This model: SO-TS, SR2's architecture on TinyStories, trained as a true first-order gradient flow (FO-b): the stiff modes relaxed exactly, no velocity anywhere. Verdict: second order wins without wells on TinyStories too (5.28 against 5.16, +2.3%; OpenWebText +4.9%). The earlier TinyStories tie was first order between layers (FO-TS, FO-a), which keeps the rotation of the stiff modes within each layer; removing that too costs the difference. With two wells the gap closes, as on OpenWebText. The first-vs-second-order collection holds the pairs.

Result

pair	corpus, tokens	second order	first order	first order against second
OpenWebText, no wells	532.5M	SR2 58.32	FO-b-OWT 61.16	+4.9%
TinyStories, first order between layers (FO-a), no wells	315M	SO-TS 5.16	FO-TS 5.13	−0.6%
TinyStories, true first order (FO-b), no wells	315M	SO-TS 5.16	FO-b-TS 5.28	+2.3%
TinyStories, with 2 Poisson-mode wells	315M	PM2-TS 4.99	FO-b-PM2-TS 4.96	−0.5%
Python code, true first order (FO-b), no wells	315M	SO-CODE 6.00	FO-b-CODE 6.55	+9.2%
Python code, with 2 Poisson-mode wells	315M	PM2-CODE 5.71	FO-b-PM2-CODE 5.62	−1.6%

• This model: settled 5.28 (mean of the evals at steps 18,000, 18,500 and 19,000: 5.35, 5.23, 5.25); best 5.10 at step 16,500. Validation perplexity at block 512, on the first 2M tokens of TinyStories' validation split; TinyStories evaluations jitter by about ±0.06, so the three-eval mean is the number to read.
• Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512) in one session: the ladder's WSD schedule over 16,250 steps, then 3,000 at the floor learning rate, the same learning-rate curve as SO-TS and FO-TS, from step 0 at seed 0.
• Paired by step against SO-TS: +1.97% through the stable phase (spread 0.27%). Past step 15,000 SO-TS's evaluations come from its extension session, resumed from step 15,000, so the pairing there is weak (+1.4%, spread 1.6%).
• The wells' gain on TinyStories is 1.8 times as large in first order (FO-b-PM2-TS 4.96: 0.061 against PM2-TS's 0.034 in log perplexity; OpenWebText at two wells: 2.0 times).
• Against a transformer: the matched GPT-2 on the same tokens and schedule settles at 4.47; this model is +18.0% behind it.
• Pre-registered (protocol §5.13, TS-K2, calls confirmed before the set trained): TS1 (FO-b-TS within ±2% of SO-TS, paired by step), called 55%: MISS, narrowly: settled +2.3% (paired +2.0% through the stable phase). TS3 (the wells help first order more on TinyStories), called 65%: HIT, 0.061 against 0.034.
• Run health: 0 watchdog triggers, 0 spike captures; SCAF leak monitor CLEAN at all 4 audits.

First order between layers against true first order

Two first-order arms exist on this base. FO-a (FO-TS) resets the velocity where each layer begins, so a token starts every layer at rest, but inside the layer the step is unchanged: the stiff modes of V_θ still rotate position into velocity and back. FO-b (this model) replaces the step by a gradient flow: the stiff modes relax exactly, without oscillating, and no velocity exists anywhere.

FO-a ties SO-TS (−0.6%); FO-b trails it (+2.3%). So what second order buys on TinyStories is the rotation within the layer, not the velocity carried between layers. That is what the OpenWebText arm SR3-crit showed by the other route: critically damping the stiff modes, which keeps the velocity between layers but stops the rotation, cost 8.5% there.

How the run compares through training

step	phase	SO-TS (second order)	FO-TS (FO-a)	FO-b-TS (this model)	this model against SO-TS
500	warmup	19.73	19.60	18.93	−4.1%
2,000	constant	8.65	8.63	8.30	−4.0%
5,000	constant	6.59	6.49	6.73	+2.1%
10,500	constant, end	6.01	5.90	6.11	+1.7%
13,000	decay	5.66	5.59	5.75	+1.6%
16,000	decay, end	5.25	5.22	5.32	+1.3%
17,000	floor	5.21	5.18	5.34	+2.5%
18,000	floor	5.13	5.10	5.35	+4.3%
19,000	floor, last eval	5.19	5.17	5.25	+1.2%

• As on OpenWebText, true first order learns faster at first and then falls behind: it leads SO-TS early in training, falls behind by step 5,000 and stays behind (FO-b-OWT led for the first third of training).

Off the trained schedule (Cell 6b-7)

reading	this model	SO-TS
Gate 1: perplexity cost of resetting the velocity at inference	0 (no velocity carried)	+2%
Gate 2: one extra layer at the trained step	+0.4%	+25.5%
Gate 3: three steps over the trained time, nats	0.122	0.211

• Extra layers are almost free and three-step refinement costs little, as for FO-b-OWT (0.18 nats against SR2's 0.41): nothing the first-order model carries between layers is tuned to the step. Cell 6b-9 reads R(geo) 0.83.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
FIRST_ORDER_OVERDAMPED=True,              # tag 'fob'
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='tinystories', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # TinyStories, then 3,000 steps at the floor

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7, 6b-9 and 6b-13 (Cell 6b-15 does not apply: no wells).

Training artefacts were written to semsimula_fock_cfc_baoab_ts_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_fob_st16250_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (steps 31,500–32,500). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, the first-vs-second-order switch, result and diagnostics
checkpoints/fock_cfc_*noattn_best.pt	weights at the best eval (step 16,500), optimizer state stripped
results/	the training log and the diagnostic cells' outputs
code/	the notebook and the model code the switches belong to (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	§5.13: the TinyStories wells set (TS-K2), its calls and these results
Competing_Mechanisms_Families_Dominance_and_Substitution.md	the mechanisms that compete for a token's step, and which can stand in for which

Caveats

• One seed per model. The pairing removes the evaluation noise, not the training-seed spread, which has not been measured; a gap of 2.3% is within the range a seed spread could reach.
• TinyStories is a narrow, low-entropy corpus: perplexities around 5 are normal on it and are not comparable with OpenWebText's.
• No CPU readings (the per-layer flow, pair-force and context ablations) on this model; the readings are the Colab cells.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-10.