dimitarpg13/semsimula-ladder-live-code-d384-l8-none-norc-sr2

SO-CODE at eight layers (L8-CODE): depth keeps paying on code, with diminishing returns (Gen 3)

The question of this collection: on Python code, the conservative model trails the matched GPT-2 by 21% at two layers. Does depth help? Each layer of this model advances the token states by one step Δt of a damped second-order flow in a learned potential; the collection holds the total integration time T = L · Δt = 8 fixed and splits it into more, shorter steps, so that only the number of layers changes.

This model (L8-CODE): SO-CODE at eight layers, Δt = 1, the depth of the matched GPT-2. Verdict: depth keeps paying, with diminishing returns. It settles at 5.35: −2.4% against four layers (5.48) and −10.7% against two (6.00), so the second doubling buys about a quarter of what the first did. It closes 60% of SO-CODE's gap to the matched GPT-2 (4.96; four layers 47%), again on every kind of token, and leaves an 8.0% gap at equal depth. The depth-on-code collection holds the same model at two, four and eight layers.

Result

model	layers	step Δt	settled perplexity	against the previous depth	against SO-CODE
SO-CODE, the baseline	2	4	6.00	—	—
L4-CODE	4	2	5.48	−8.5%	−8.5%
L8-CODE (this model)	8	1	5.35	−2.4%	−10.7%

For comparison, on the same code tokens and schedule:

model	what changes from SO-CODE	settled perplexity	against SO-CODE
matched GPT-2 (not on the Hub)	a transformer, 8 layers, d = 384	4.96	−17.3%
FOCK-CODE (not on the Hub)	the Fock register path on	5.66	−5.6%
FO-b-CODE	trained as a first-order gradient flow	6.55	+9.2%

• This model: settled 5.35 (the mean of the log perplexities of the evals at steps 18,000, 18,500 and 19,000: 5.26, 5.36, 5.45); best 5.26 at step 18,000. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split, so no file is in both).
• Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512): the ladder's WSD schedule over 16,250 steps, then 3,000 steps at its floor learning rate. SO-CODE, L4-CODE and this model start from step 0 at seed 0 on the same batches, so their evaluations are paired step by step.
• 76,731,865 parameters, 159,756 more than SO-CODE's 76,572,109. The layers share V_θ's well bank, V_φ and the context channels; the extra parameters are the six new layers' depth code for the bank (11,520), their V_φ scales (6) and their register destruction gates (148,230), which have no path to the tokens in this conservative-only model. Depth here adds steps, not capacity.
• Pre-registered: none on its own result: L8-CODE ran by the rule fixed before L4-CODE trained (it runs if L4-CODE settles more than 2% below SO-CODE; L4-CODE settled 8.5% below).
• Run health: 19,250 steps in 29.5 h over two Colab sessions (5.5 s per step with layer checkpointing; resumed at step 15,500 with the batch sampler restored, so the evals stay paired: their step-to-step changes correlate with L4-CODE's at 0.991 after the resume and 0.994 before it); 0 watchdog triggers; largest rise between logged training losses 0.069 (step 13,050); gradient norm median 0.38 after step 2,000, maximum 1.06 (step 400, warmup); SCAF leak monitor CLEAN at all 4 audits. 34 of the 45 logged steps between 1,400 and 3,600 had a non-finite gradient norm, and the training loop skips those updates (cause not diagnosed; none after step 3,600).

Paired by step

The mean difference in log perplexity (nats) between two models over a span of evaluations, with the standard deviation of the per-evaluation differences.

steps	L8-CODE − L4-CODE	L8-CODE − SO-CODE
2,000–4,000 (5 evals)	+0.362 (SD 0.073), +43.6%	+0.069 (SD 0.054), +7.1%
8,000–13,000 (11 evals)	+0.001 (SD 0.011), +0.1%	−0.098 (SD 0.006), −9.3%
16,000–19,000 (7 evals)	−0.024 (SD 0.002), −2.4%	−0.112 (SD 0.003), −10.6%

• Behind early, level by step 8,000, ahead from step 10,500. The early lag against four layers (+0.36 nats over steps 2,000–4,000) falls in the window where many of this model's updates were skipped (below). After it the gap closes and turns: over the last 3,000 steps this model is 0.024 nats ahead of L4-CODE with an SD of 0.002 per evaluation.
• Skipped updates, steps 1,400–3,600. 34 of the 45 logged steps in that window had a non-finite gradient norm, and the training loop skips the update whenever the norm is not finite; the exact count was not recorded. The training loss stayed finite, so the fault is in the backward pass; the cause is not diagnosed. None occurred after step 3,600, and none in SO-CODE's or L4-CODE's runs. So this model trained on fewer updates than L4-CODE, and its gain over four layers is, as far as the skipped steps go, a lower bound.
• Training loss over steps 16,000–19,250: 1.585 for this model, 1.603 for L4-CODE; the gain shows on the training data as on the validation data. One seed each.

Through training

step	SO-CODE	L4-CODE	L8-CODE	against L4-CODE
500	44.02	46.79	50.93	+8.8%
1,000	25.13	26.63	28.10	+5.5%
1,500	20.33	17.68	19.94	+12.8%
2,000	17.67	12.86	18.05	+40.4%
2,500	15.80	10.78	16.29	+51.1%
3,000	12.96	9.55	14.72	+54.1%
4,000	10.62	8.60	11.02	+28.1%
5,000	9.27	7.95	8.86	+11.4%
8,000	7.96	7.12	7.27	+2.1%
10,000	7.45	6.71	6.73	+0.3%
13,000	6.38	5.86	5.75	−1.9%
16,000	6.12	5.61	5.48	−2.3%
18,000	5.87	5.37	5.26	−2.0%
19,000	6.10	5.59	5.45	−2.5%

Where the gain is: the copy split

Every validation target (262,144 tokens in 512 windows of 512) is put in groups by what the context offers for it: a first occurrence does not occur earlier in the window, a repeat does, an induction target follows a token pair seen earlier; and by how far back its last earlier occurrence lies. For each group, the table gives the share of SO-CODE's per-token gap to the matched GPT-2 that a model closes (debug/copy_split.py, best checkpoints).

targets	share of all targets	L8-CODE closes	L4-CODE closes	FOCK-CODE closes
all	100%	59.8%	47.4%	30.4%
first occurrences (nothing to copy)	27.7%	59.1%	47.4%	33.4%
induction targets	50.0%	64.5%	49.4%	28.3%
indentation, newline, space	21.9%	53.2%	33.9%	21.1%
identifiers last seen 1–16 / 65–512 tokens back	10.0 / 7.7%	58.5 / 59.9%	48.5 / 48.0%	24.3 / 31.4%
repeats and induction targets last seen 1–16 / 17–64 / 65–512 back	34.8 / 25.2 / 12.3%	58.6 / 61.1 / 60.0%	45.2 / 48.6 / 48.1%	26.5 / 30.8 / 29.7%

• The gain is general again. Every group gains about the same over four layers; first occurrences, which have nothing to copy, close 59% of their gap. On these targets this model's perplexity is 5.765 (L4-CODE 5.898); it is better than SO-CODE in 98% of the windows and than L4-CODE in 74%.
• Far against near: 60.0% against 58.6%, +1.4 points; 95% interval over windows by a bootstrap −3.2 to +6.3. Not a long-range gain beyond the noise.

The wells, layer by layer

V_θ is a bank of eight anisotropic Gaussian wells whose centres, widths and depths are computed from the context, shared by all layers and told apart by a learned depth code. The nearest-well ratio is the distance from a token's state to the nearest well centre, divided by the state's norm; the occupancy of a well is exp(−½ (h − μ)ᵀ P (h − μ)), the factor its force and its gradient both carry; n_eff is the effective number of wells in the layer's weights (the exponential of the entropy of the softmax weights, which depend on the context only, not on the state); the removal cost is the perplexity change when every well of one layer is switched off on the trained model. Readings by debug/vtheta_well_probe.py on 16 validation windows (perplexity 6.74 on them).

reading (step 18,000)	layer 0	layer 1	layer 2	layer 3	layer 4	layer 5	layer 6	layer 7
nearest-well ratio, median (p99)	1.21 (1.78)	1.21 (1.69)	1.16 (1.63)	1.14 (1.63)	1.13 (1.61)	1.18 (1.64)	1.30 (1.76)	1.46 (1.93)
occupancy, strongest well	0.98	0.26	0.17	0.42	0.45	0.30	0.13	0.06
wells in use (n_eff)	7.1	1.9	1.9	2.1	2.1	2.0	1.8	1.8
force carried by wells 5 and 6	56%	87%	92%	92%	95%	97%	98%	99%
removal cost of the layer's wells	+2,780%	+2.8%	+3.4%	+2.2%	+26%	+25%	+6.2%	+0.6%

• No dead wells at any checkpoint. From step 500 to the end the median nearest-well ratio stays between 0.81 and 1.91 in every layer; the failure described in Dead_Wells_in_the_Joint_V_theta_Bank.md does not occur at L = 8.
• Layer 0 uses all eight wells; the seven deep layers run on two. In layers 1–7 wells 5 and 6 carry 87–99% of the force from step 500 on, as two wells carry the deep layers of L4-CODE. Layer 0 uses all eight, close to the states (occupancy 0.98).
• Each deep layer matters little on its own: removing layer 0's wells breaks the model (+2,780%), removing any one deep layer's costs between +0.6% and +26% (layers 4 and 5 the most). Removing all eight layers' wells costs +3,251%.
• Why the first layer spreads its weights: the scale of its input. The weights are a softmax of a linear function of the bank's input, the context channels plus the layer's depth code. Layer 0's input is about 13 times smaller than the deep layers' (median norm 1.9 against 22–28), because its states are not yet on the LayerNorm sphere. Rescaled to the deep layers' size, layer 0's weights concentrate on 2.0 effective wells; the deep layers' inputs shrunk to layer 0's size spread over 7.1–7.3. The eight against two is a temperature effect of the input's size, not a different need (debug/vtheta_weight_logits.py, which also checks a bound on how far two layers' counts can differ).

Diagnostics

Refinement (Cell 6b-7). The trained layers are applied N times with the step changed, the loss read on 12 fixed validation batches. Gate 2 keeps Δt = 1 and lets T grow; Gate 3 keeps T = 8 and refines Δt = 8/N, the test of whether the layers approximate a continuous flow (the loss levels off as Δt shrinks) or are maps tied to the trained step.

N	Gate 2, perplexity	Gate 3, perplexity	Gate 3 penalty, nats
1	67.35	434.93	4.552
2	59.42	157.93	3.539
3	42.51	163.81	3.576
4	26.92	34.33	2.013
6	7.50	5.64	0.206
8 (trained)	4.59	4.59	0.000
12		6.95	0.416
16		10.04	0.783
24		40.37	2.175
32		61.98	2.604

• A map, not a flow, as for every second-order model in this programme: the loss is lowest at the trained step and rises on both sides. The half-step penalty keeps falling with depth: 0.78 nats here, 1.13 at four layers, 1.42 at two. A slightly coarser step costs little (N = 6: 0.21 nats).
• Resetting the velocity costs +172% (Gate 1: every layer starts from rest instead of from the velocity the previous layer handed on), against +76% at four layers and +4% at two. With every doubling the model relies more on the carried velocity.
• Stiffness (Cell 6b-13): ω·Δt median 1.75 at Δt = 1; by layer 2.08, 1.91, 1.34, 2.26, 2.09, 1.70, 1.05, 0.62; 33.8% of readings past 2 (four layers 53.6%, two 99.97%). The stiff layers are 0, 3 and 4, and the last two are soft. The integrator treats the stiff modes exactly, so this is a reading of stiffness, not of instability.
• Retrieval per layer is SO-CODE's, repeated. In each of layers 1–7 the longest context channel (channel 4) finds earlier occurrences 65 or more tokens back at 10, 15, 20, 25, 27, 23, 12 times an even spread (debug/xi_reach.py); SO-CODE's layer 1 does it at 26.

The per-layer flow (FLOW-R). Each trained layer's share of T is integrated with k = N/8 substeps, the layer's context and wells' quadratic frozen at its first substep and one LayerNorm projection at its end; the force, including the pair potential's top-k source selection, is recomputed at every substep (debug/refinement_flow_confirmation.py, 12 × 4 × 512 tokens). Penalty ln(PPL_N / PPL_8) in nats:

k = N / L	1.5	2	3	4	6	8	whole stack at k = 4
L8-CODE (L = 8, Δt = 1; trained ppl 5.18 on these batches)	0.186	0.029	0.034	0.035	0.035	0.035	2.695
L4-CODE (L = 4, Δt = 2)	0.429	0.520	0.282	0.320	0.309	0.306	2.960
SO-CODE (L = 2, Δt = 4)	0.970	0.526	0.300	0.384	0.376	0.369	2.858

• At eight layers the per-layer flow settles 0.035 nats from the trained step, ten times closer than at two or four layers (0.37, 0.31) and closer than any model in this programme so far (SR2 on prose: 0.10). The whole stack still fails refinement (2.70 nats at k = 4). So depth at a fixed integration time brings each trained layer onto its own continuous flow; it does not make the stack one.

A reading

Depth at a fixed integration time keeps paying on code, with sharply diminishing returns: −0.090 nats from two layers to four, −0.024 from four to eight, at a fixed budget of 19,250 steps. The per-layer readings are the same at every depth (one long channel's lift, two wells in each deep layer), now in seven deep layers. In Dead_Wells_in_the_Joint_V_theta_Bank.md, §8, a conjecture says that on code the model is limited by what it does with what its context channels retrieve, not by the retrieval; more steps of readout on the same retrieval help everywhere, and help less with each doubling. An interpretation, not a measured mechanism.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
VPHI_ANALYTIC_FORCE=True                  # speed switch, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor
LADDER_L=8                                # eight layers; dt = T / L = 1, T = 8 as at L = 2 and 4
SNAPSHOT_STEPS=[1_000, 1_500, 2_000, 2_500, 3_000]   # weights-only snapshots for the well probe
# LAYER_CHECKPOINT stays at its default True (memory, same arithmetic)

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7 and 6b-13 (one A100 80 GB; training peaks at 28.9 GB with layer checkpointing, 5.5 s per step, about 29.5 h). The run crosses Colab's 24-hour limit: the notebook's wall-clock autosave writes a checkpoint, and Cell 2 resumes from the latest one with the batch sampler's state, so the batches continue as in an uninterrupted run. The well probe is debug/vtheta_well_probe.py, the channels' reach debug/xi_reach.py, the copy split debug/copy_split.py, the per-layer flow debug/refinement_flow_confirmation.py with micro=1 (CPU, on the downloaded run folder).

Training artefacts were written to semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, result and diagnostics
checkpoints/fock_cfc_*noattn_step18000_best.pt	weights at the best eval (step 18,000), optimizer state stripped
results/	the training logs (both sessions, and joined), the diagnostic cells' outputs, the well probe at the step-500 checkpoint, the five snapshots and the best checkpoint (well_probe_*; _L<n>_none = layer n's wells removed, _L01234567_none all), the channels' reach (xi_reach_l8code.txt), the copy split (copy_split_report*.txt, every code model against the matched GPT-2 and against SO-CODE; l8_near_far_bootstrap_output.txt) and the per-layer flow (refinement_flow_confirmation_l8-code_*)
code/	the notebook and the model code (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	the run records: entries p-l8-code-rule, p-l8-code-setup and p-l8-code-result
Dead_Wells_in_the_Joint_V_theta_Bank.md	the dead wells of the shared V_θ bank, and §8's conjectures on what limits the model on code

Caveats

• One seed per model. The pairing removes the evaluation noise, not the training-seed spread, which has not been measured. The 2.4% over four layers is a consistent difference between these two runs, just outside the ±2% band used for every code arm.
• Fewer updates than its neighbours. Updates were skipped over steps 1,400–3,600 (above); the exact count is unknown.
• Not settled by the letter of the settling rule, like every code model: the evals still move by a few hundredths of a nat over the last 3,000 steps. The paired gap to L4-CODE is flat.
• Code only, without the register path. There is no OpenWebText twin at eight layers. Perplexities on code are not comparable with other corpora's.
• About five times the compute per step of two layers (5.5 s with layer checkpointing on, which recomputes each layer in the backward pass, against SO-CODE's 1.1 s without it).

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-08.