dimitarpg13/semsimula-ladder-live-code-d384-l8-none-norc-sr2-k2

L8-CODE with two wells (K2-L8-CODE): at eight layers the six extra wells are worth 5.8%, more than the second doubling of depth (Gen 3)

The question of this collection: on Python code, the conservative model trails the matched GPT-2 by 21% at two layers. Does depth help? Each layer advances the token states by one step Δt of a damped second-order flow in a learned potential; the collection holds the total integration time T = L · Δt = 8 fixed and splits it into more, shorter steps.

This model (K2-L8-CODE): L8-CODE, eight layers at Δt = 1, with two wells in the V_θ bank instead of eight; nothing else changes. Verdict: at eight layers the six extra wells are worth more than the second doubling of depth. It settles at 5.67: +5.8% against eight wells (5.35) and +3.3% against four layers with eight wells (5.48). It still beats two layers with eight wells by 5.5%, with 26.4M fewer parameters. The depth-on-code collection holds the same model at two, four and eight layers, and two variants: this one, and one trained at several step counts.

Result

model	layers	wells	parameters	settled perplexity	against L8-CODE
SO-CODE	2	8	76,572,109	6.00	+12.0%
L4-CODE	4	8	76,625,361	5.48	+2.4%
L8-CODE	8	8	76,731,865	5.35	—
K2-L8-CODE (this model)	8	2	50,164,435	5.67	+5.8%
matched GPT-2 (not on the Hub)	8	—		4.96	−7.4%

• This model: settled 5.67 (the mean of the log perplexities of the evals at steps 18,000, 18,500 and 19,000: 5.56, 5.68, 5.76); best 5.56 at step 18,000. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split, so no file is in both).
• Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512): the ladder's WSD schedule over 16,250 steps, then 3,000 steps at its floor learning rate, from step 0 at seed 0 on the same batches as SO-CODE, L4-CODE and L8-CODE, so the evaluations are paired step by step.
• 50,164,435 parameters, 26,567,430 fewer than L8-CODE: the six wells removed from the shared bank. Outside the embeddings, 11.6M against the matched GPT-2's 14.2M.
• Pre-registered: none registered: the arm was set up 2026-11-02 to ask whether depth changes what the six extra wells buy, and launched before calls were written.
• Run health: 19,250 steps in 19.0 h in one session from step 0 (3.5 s per step with the V_φ-force checkpoint, which recomputes only the pair force in the backward pass, and no layer checkpointing); peak memory 35.0 GB in training, 51.6 GB in evaluation; no non-finite gradient norm at any logged step (L8-CODE had 34 of 45 over steps 1,400–3,600 on the same batches); gradient norm median 0.38 after step 2,000, maximum 2.02 (step 650, warmup); largest rise between logged training losses 0.069 (step 13,050, as in L8-CODE and SO-CODE); 0 watchdog triggers; SCAF leak monitor CLEAN at all 4 audits.

Paired by step

The mean difference in log perplexity (nats, K2-L8-CODE minus the other model) over a span of evaluations, with the standard deviation of the per-evaluation differences.

steps	against L8-CODE	against L4-CODE	against SO-CODE
2,000–4,000 (5 evals)	−0.340 (SD 0.074)	+0.021 (SD 0.013)	−0.272 (SD 0.064)
8,000–13,000 (11 evals)	+0.028 (SD 0.014)	+0.029 (SD 0.004)	−0.070 (SD 0.009)
16,000–19,000 (7 evals)	+0.056 (SD 0.002)	+0.032 (SD 0.002)	−0.056 (SD 0.003)

• The gap to eight wells opened in the learning-rate decay and then held: +0.027 nats at step 10,000, +0.045 at 13,000, +0.054 at 16,000, and +0.056 with an SD of 0.0015 over the last 3,000 steps. Early on this model is ahead; over steps 1,400–3,600 L8-CODE skipped many updates and this model none.
• Against four layers with eight wells the gap stayed at +0.02 to +0.03 nats from step 3,500 on. So the six wells are worth 0.056 nats at eight layers, and the doubling from four to eight layers at eight wells 0.024.

Through training

step	SO-CODE	L8-CODE	K2-L8-CODE	against L8-CODE
500	44.02	50.93	52.38	+2.8%
1,000	25.13	28.10	28.29	+0.7%
1,500	20.33	19.94	18.35	−8.0%
2,000	17.67	18.05	13.41	−25.7%
2,500	15.80	16.29	10.98	−32.6%
3,000	12.96	14.72	9.67	−34.3%
4,000	10.62	11.02	8.69	−21.1%
5,000	9.27	8.86	8.08	−8.8%
8,000	7.96	7.27	7.28	+0.1%
10,000	7.45	6.73	6.91	+2.7%
13,000	6.38	5.75	6.02	+4.7%
16,000	6.12	5.48	5.79	+5.7%
18,000	5.87	5.26	5.56	+5.7%
19,000	6.10	5.45	5.76	+5.7%

Diagnostics

Refinement (Cell 6b-7). The trained layers are applied N times with the step changed, the loss read on 12 fixed validation batches. Gate 2 keeps Δt = 1 and lets T grow; Gate 3 keeps T = 8 and refines Δt = 8/N, the test of whether the layers approximate a continuous flow (the loss levels off as Δt shrinks) or are maps tied to the trained step.

N	Gate 2, perplexity	Gate 3, perplexity	Gate 3 penalty, nats	L8-CODE's penalty
1	96.30	397.13	4.418
2	86.98	203.27	3.748
3	63.99	146.84	3.423
4	39.67	92.45	2.960
6	9.44	8.29	0.549	0.206
8 (trained)	4.79	4.79	0.000
12		7.63	0.466	0.416
16		14.02	1.074	0.783
24		41.91	2.169
32		71.65	2.705	2.604

• A map, not a flow, as every second-order model in this programme: the loss is lowest at the trained step and rises on both sides. Two wells refine no better than eight and somewhat worse near the trained step (N = 6: 0.55 nats against 0.21).
• Resetting the velocity multiplies perplexity by 3.71 (Gate 1: 4.79 carried, 17.77 reset; L8-CODE × 2.72). With fewer wells the model leans more on the carried velocity.
• Stiffness (Cell 6b-13): ω·Δt median 1.50 at Δt = 1 (L8-CODE 1.75); by layer 3.22, 1.46, 1.15, 1.96, 1.96, 1.56, 0.90, 0.40; 25.5% of readings past 2 (L8-CODE 33.8%). The profile has L8-CODE's shape. Layer 0 is stiffer with two wells (every reading past 2; L8-CODE's median 2.08), the deep layers softer. The integrator treats the stiff modes exactly, so this is a reading of stiffness, not of instability.
• The wells' rank (Cell 6b-4): each well's precision is a rank-4 update; its participation ratio has median 3.03 of 4 (p05 2.30, p95 3.68), singular-value ratios 1.00, 0.75, 0.60, 0.47, with the Frobenius cap binding. Training concentrated the budget but kept all four directions in use.
• Registers (Cell 7): removing all registers changes nothing (5.17 → 5.17), as in L8-CODE; the reverse channel is off by design (norc).

The per-layer flow (FLOW-R). Each trained layer's share of T is integrated with k = N/8 substeps, the layer's context and wells' quadratic frozen at its first substep and one LayerNorm projection at its end; the force, including the pair potential's top-k source selection, is recomputed at every substep (debug/refinement_flow_confirmation.py, 12 × 4 × 512 tokens). Penalty ln(PPL_N / PPL_8) in nats:

k = N / L	1.5	2	3	4	6	8	whole stack at k = 4
K2-L8-CODE (L = 8, two wells; trained ppl 5.47 on these batches)	0.557	0.089	0.096	0.097	0.097	0.098	2.798
L8-CODE (L = 8, eight wells)	0.186	0.029	0.034	0.035	0.035	0.035	2.695
L4-CODE (L = 4)	0.429	0.520	0.282	0.320	0.309	0.306	2.960
SO-CODE (L = 2)	0.970	0.526	0.300	0.384	0.376	0.369	2.858

• The per-layer flow settles, 0.098 nats from the trained step, about three times L8-CODE's 0.035 and a third of L4-CODE's 0.306. At eight layers, six fewer wells move the limit by 0.06 nats, as much as the first doubling of depth did; the second doubling moved it 0.27. The whole stack still fails refinement, as in every model.
• Kick share (the part of a layer's position update owed to the explicit kick, median by layer): 0.14, 0.14, 0.12, 0.19, 0.19, 0.26, 0.32, 0.44 (L8-CODE 0.15, 0.14, 0.08, 0.13, 0.15, 0.20, 0.26, 0.39). Higher from layer 3 on; with one model of each kind, that does not show it causes the larger limit.

A reading

On code at eight layers, wells and depth are not interchangeable at the margin: removing six of eight wells costs +0.056 nats, more than the 0.024 that the second doubling of depth bought at eight wells, and moves each layer's step three times further from its own flow. Two wells at eight layers still beat eight wells at two layers by 0.056 nats with a third fewer parameters: going from two layers to eight buys more than the six wells cost. Minimum_Well_Count_for_Conservative_Models.md fits the number of wells each layer needs from frozen activations and puts the knee at four. One seed per model.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor
LADDER_L=8                                # eight layers; dt = T / L = 1, T = 8
V_THETA_WELLS_PER_HEAD=2                  # two wells in the joint V_theta bank instead of eight
VPHI_FORCE_CHECKPOINT=True                # memory switch, same arithmetic
SNAPSHOT_STEPS=[1_000, 1_500, 2_000, 2_500, 3_000]   # weights-only snapshots for the well probe

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 7, 6b-7, 6b-4 and 6b-13 (one A100 80 GB; 35.0 GB peak in training, 3.5 s per step, about 19 h). The per-layer flow is debug/refinement_flow_confirmation.py … code seed=20261009 micro=1 on the downloaded run folder (CPU, 51.6 GB peak, 1.2 h).

Training artefacts were written to semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x2_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, result and diagnostics
checkpoints/fock_cfc_*noattn_step18000_best.pt	weights at the best eval (step 18,000), optimizer state stripped
results/	the training log, the diagnostic cells' outputs (Cells 6b-4, 6b-7, 6b-13 and 7), the resonance and leak records and the per-layer flow (refinement_flow_confirmation_k2-l8-code_*)
code/	the notebook and the model code (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	the run records: entries p-k2-l8-code-setup and p-k2-l8-code-result
Minimum_Well_Count_for_Conservative_Models.md	how many wells each layer needs, from the trained models and from fits

Caveats

• One seed per model. The pairing removes the evaluation noise, not the training-seed spread, which has not been measured.
• The early comparison with L8-CODE is not clean: L8-CODE skipped many updates over steps 1,400–3,600 and this model none, which is why this model leads early. The settled gap formed after step 8,000.
• Not settled by the letter of the settling rule, like every code model; the paired gap to L8-CODE is flat over the last 3,000 steps.
• Code only, without the register path. Perplexities on code are not comparable with other corpora's.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-08.