dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-sr2-vs234

SO-CODE trained at 2, 3 and 4 steps (SR4a-CODE): refinement works inside the trained band and not beyond it (Gen 3)

The question: each layer of these models advances the token states by one step Δt of a damped second-order flow. If the layers were a discretised flow, splitting the same integration time T = 8 into more, smaller steps would give nearly the same answer (refinement). Every model trained at a fixed step count fails that test by 1.4 to 3 nats. Does training at several step counts fix it?

This model (SR4a-CODE): SO-CODE, two layers on Python code, trained with the number of steps N over T = 8 drawn for each optimizer step from {2, 3, 4}; step j of a walk uses trained layer ⌊jL/N⌋. No parameter is added. Verdict: refinement works inside the trained band and not beyond it. At N = 3 and 4 the model is better than at its N = 2; at N = 8 it loses 1.39 nats, a loss that grew with training. The price at N = 2 is +2.5% (settled 6.14 against 6.00). The depth-on-code collection holds SO-CODE at two, four and eight layers, and two variants: this one, and eight layers with two wells.

Result

• Settled at N = 2: 6.14 (the mean of the log perplexities of the evals at steps 18,000, 18,500 and 19,000: 6.03, 6.16, 6.24); best 6.03 at step 18,000. SO-CODE 6.00, so +2.5%. Evaluation walks N = L = 2. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split).
• Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512), the ladder's WSD schedule over 16,250 steps, then 3,000 at the floor, from step 0 at seed 0 on SO-CODE's batches. The step counts come from a separate generator, so the batches and every evaluation stay paired with SO-CODE's.
• 76,572,110 parameters, as SO-CODE: the step count is a property of the walk, not of the weights.
• Run health: 19,250 steps in 8.3 h in one session from step 0 (1.55 s per step, no layer checkpointing in training); peak memory 60.5 GB in training (an N = 4 step holds four layer steps' activations), 46.2 GB in evaluation; no non-finite gradient norm at any logged step; gradient norm median 0.45 after step 2,000, maximum 1.94 (step 750, warmup); 0 watchdog triggers; SCAF leak monitor CLEAN at 5,000, 10,000 and 15,000 (the step-19,250 audit printed nan perplexities, as in most runs of this notebook).

Paired with SO-CODE step by step: the mean difference in log perplexity over a span of evaluations, with the standard deviation of the per-evaluation differences.

steps	SR4a-CODE − SO-CODE (nats)	SD	evals
2,000–4,000	+0.052	0.019	5
8,000–10,500	+0.049	0.011	6
11,000–16,000	+0.034	0.007	11
16,000–19,000	+0.025	0.002	7

The cost of training at three step counts fell through the run, most of it in the learning-rate decay. Training loss over steps 16,000–19,250 by step count: N = 2: 1.712, N = 3: 1.677, N = 4: 1.688. The model fits N = 3 best, not the N = 2 it is evaluated at.

Through training

step	SO-CODE	SR4a-CODE	against SO-CODE
500	44.02	49.55	+12.6%
1,000	25.13	28.93	+15.1%
1,500	20.33	22.50	+10.7%
2,000	17.67	18.91	+7.0%
2,500	15.80	16.35	+3.5%
3,000	12.96	13.46	+3.9%
4,000	10.62	11.12	+4.7%
5,000	9.27	9.68	+4.4%
8,000	7.96	8.26	+3.8%
10,000	7.45	7.91	+6.2%
13,000	6.38	6.65	+4.2%
16,000	6.12	6.27	+2.5%
18,000	5.87	6.03	+2.7%
19,000	6.10	6.24	+2.3%

Refinement

Gate 3 (Cell 6b-7): the trained layers walked in N steps of Δt = 8/N over the same T = 8 (step j uses layer ⌊jL/N⌋), the loss read on 12 fixed validation batches of 4 × 512 tokens. Penalty ln(PPL_N / PPL_2) in nats.

N	Δt	trained at?	SR4a-CODE perplexity	penalty	SO-CODE perplexity	penalty
2	4.00	yes	5.21	0	5.06	0
3	2.67	yes	5.01	−0.039	9.59	+0.639
4	2.00	yes	5.07	−0.027	20.91	+1.418
6	1.33	no	8.17	+0.450	50.95	+2.311
8	1.00	no	20.98	+1.394	77.39	+2.727

• Inside the trained band the model refines, and improves. The N = 3 and N = 4 penalties are negative at every probe from step 2,000 to 18,000. At N = 3 the model's 5.01 on these batches is below SO-CODE's 5.06 at its trained N = 2 (−0.010 nats on 24,576 tokens, at 1.5 times the compute; not yet checked on the full validation set).
• Beyond the band refinement fails, and grew worse with training. The N = 8 penalty was 0.43 at step 2,000, 0.88 at step 10,000, 1.33 at step 14,000, 1.39 at step 18,000. SO-CODE's is about twice as large (2.73), but the model learned the three step counts it was trained on, not a flow that refines further.

The per-layer flow (FLOW-R). Each trained layer's share of T is integrated in k substeps, the layer's context and wells' quadratic frozen at the share's start and one LayerNorm projection at its end; the force, including the pair potential's top-k source selection, is recomputed at every substep (debug/refinement_flow_confirmation.py, 12 × 4 × 512 tokens). Penalty in nats against the trained walk:

k = N / L	1.5	2	3	4	6	8
SR4a-CODE	0.780	0.679	0.787	0.813	0.827	0.840
SO-CODE	0.970	0.526	0.300	0.384	0.376	0.369

• It settles, but at 0.84 nats, more than twice SO-CODE's 0.37. Every step this model was trained on is a whole layer step: fresh context, fresh wells and a LayerNorm projection each time. It learned to work under those walks, and in doing so moved each layer's step further from that layer's own continuous-time limit. Kick share at N = 2: 0.44 and 0.87 by layer (SO-CODE 0.41, 0.78).

Calls

Proposed 2026-11-01 and not confirmed by the author before launch; scored as proposed.

call	prediction	probability	measured
VS1	Gate 3 penalty at N = 3 and at N = 4 both ≤ 0.095 nats	70%	−0.039 and −0.027	hit
VS2	settled at N = L within +5% of SO-CODE (≤ 6.295)	65%	6.14 (+2.5%)	hit
VS3	Gate 1 ≥ +3.2% (inertia kept)	70%	+8.1%	hit
VS4	Gate 3 penalty at N = 8 ≤ 0.50 nats	40%	1.394	miss
VS5	per-layer flow limit at k = 8 ≤ 0.20 nats	45%	0.840	miss

Other diagnostics

• Gate 1, inertia: carried 5.21, reset 5.63, +8.1% (SO-CODE +4.0%). Inertia is worth twice as much as in SO-CODE.
• Gate 2, more hops at fixed Δt: as SO-CODE. Variable steps at a fixed T do not teach the model to run longer.
• Stiffness (Cell 6b-13): ω·Δt median 4.43 (layer 0 5.74, layer 1 3.43), 99.1% past 2; SO-CODE 7.55 (9.58, 4.94). Training at smaller steps made the step softer, mostly in layer 0.
• The wells' rank (Cell 6b-4): participation ratio median 3.61 of 4 (p05 2.03, p95 3.85), singular-value ratios 1.00, 0.79, 0.70, 0.62.
• Registers (Cell 7): removing all registers changes nothing (5.61 → 5.61), as in SO-CODE.

A reading

Variable-step training buys refinement invariance within the band of step sizes it is trained on, here Δt from 2 to 4, and not a flow. It costs 2.5% at N = 2 and 41% more training time. Depth is the other way to spend the compute: four layers at a fixed step settle 8.5% below SO-CODE. One seed, at L = 2, on code.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor
VARIABLE_STEP_COUNTS=(2, 3, 4)            # one N per optimizer step, drawn uniformly; step j uses trained layer min(j*L//N, L-1)

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 7, 6b-7, 6b-4 and 6b-13 (one A100 80 GB; 60.5 GB peak in training, 1.55 s per step, about 8.3 h). Cell 6 draws N from its own generator (seed + 4,000,000, saved in the checkpoint) and logs the training loss per N; every 2,000 steps it probes Gate 3 and logs an event: vs_probe record. The per-layer flow is debug/refinement_flow_confirmation.py … code seed=20261009 on the downloaded run folder (CPU).

Training artefacts were written to semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_vs234_st16250_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, result and diagnostics
checkpoints/fock_cfc_*noattn_best.pt	weights at the best eval (step 18,000), optimizer state stripped
results/	the training log (with the per-N training loss and the Gate 3 probes), the diagnostic cells' outputs (Cells 6b-4, 6b-7, 6b-13 and 7; the Cell 6 log keeps the author's file name, L4_SR4a_…, though the model has two layers), the resonance and leak records and the per-layer flow (refinement_flow_confirmation_sr4a-code_*)
code/	the notebook and the model code (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	the run records: entries p-sr4a-code and p-sr4a-code-result
Refinement_Experiments_and_Variable_Step_Training.md	the refinement gates, the per-layer flow, the SR4a design and the calls

Caveats

• One seed. The pairing with SO-CODE removes the evaluation noise, not the training-seed spread.
• The N = 3 advantage over SO-CODE is one set of 24,576 tokens, not yet repeated on the full validation set.
• The step-count walk is the hold policy (step j uses layer ⌊jL/N⌋); other ways of assigning layers to steps were not tried.
• Code only, two layers, without the register path. Perplexities on code are not comparable with other corpora's.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-08.