README.md

metadata

pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - codeparrot
  - mechanism-ablation
  - ladder-study
  - pre-registered
  - live-gradients
  - generation-3
  - depth
  - wells
  - rank
  - refinement
  - variable-step
  - code
  - matched-token-budget
  - fock-space
  - parflm
  - anisotropic-gaussian-potential
  - cfc
  - baoab
  - fully-conservative
  - riemannian-geometry
  - riemannian-geodesics
  - jacobi-metric
  - non-transformer
  - attention-free
  - constant-memory-inference
datasets:
  - codeparrot/codeparrot-clean-train
  - codeparrot/codeparrot-clean-valid
metrics:
  - perplexity
model-index:
  - name: semsimula-ladder-live-code-d384-l8-none-norc-sr2-k2-r6-vs8-12-16
    results:
      - task:
          type: text-generation
        dataset:
          type: codeparrot/codeparrot-clean-valid
          name: CodeParrot clean (Python)
          split: validation
        metrics:
          - type: perplexity
            value: 5.6506
            name: >-
              Validation perplexity at block 512, settled (mean of the last
              three 500-step evals at block 512)
            verified: false

K2-L8-R6-CODE trained at 8, 12 and 16 steps (K2-L8-R6-VS-CODE): a conservative model on code, 14% behind the matched GPT-2, that refines over a band of step counts (Gen 3)

What this model shows. It is a conservative language model: every force on a token is the gradient of a learned potential at fixed context (up to one measured exception, the straight-through term of the pair potential's top-k source selection, which can be dropped at evaluation for 0.0006 nats), and there is no attention, register path or exchange field. On Python code it settles at perplexity 5.65, 13.9% behind a GPT-2 trained on the same tokens (4.96; its fixed-step parent is 11.8% behind). And it refines, in a qualified sense: run with its eight layers cut into more, smaller steps over the same integration time, it computes the same function within 0.01 nats at every step count from 8 to 16, including six it never saw, and within 0.08 nats at 32. The same model trained at eight steps only loses 1.3 and 1.7 nats at 24 and 32.

This model (K2-L8-R6-VS-CODE): K2-L8-R6-CODE, eight layers with two rank-6 wells, trained with the number of steps N over the fixed time T = 8 drawn afresh for every optimizer step from {8, 12, 16}; nothing else changes, and it is evaluated at N = 8, so the two are paired step by step. The depth-on-code collection holds the same model at two, four and eight layers, and its variants: two wells, two wells at rank 6, and two trained at several step counts (at two layers and, this one, at eight).

Result

model	layers	wells	rank	trained at N	settled perplexity	against the matched GPT-2
L8-CODE	8	8	4	8	5.35	+8.0%
K2-L8-CODE	8	2	4	8	5.67	+14.3%
K2-L8-R6-CODE	8	2	6	8	5.54	+11.8%
K2-L8-R6-VS-CODE (this model)	8	2	6	8, 12, 16	5.65	+13.9%
matched GPT-2 (not on the Hub)	8	—	—	—	4.96	—

• Settled 5.65 (the mean of the log perplexities of the evals at steps 18,000, 18,500 and 19,000: 5.56, 5.65, 5.74); best 5.56 at step 18,000. Validation perplexity at block 512, at N = 8, on the first 2M tokens of CodeParrot's validation files (a file-level split).
• Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512) on the same batches as K2-L8-R6-CODE, with N = 8, 12 and 16 drawn on 6,525, 6,416 and 6,309 steps.
• 53,115,092 parameters, the same as K2-L8-R6-CODE: training at several step counts adds none. Outside the embeddings 14.5M, against the matched GPT-2's 14.2M.
• Run health: relaunched from step 500 of the first launch, then two Colab sessions (steps 501–16,050 and, after the wall-clock autosave, 15,501–19,250; the evaluation at 16,000 repeated to 0.0002 nats), 5.5 s per step with the V_φ-force checkpoint and no layer checkpointing; peak memory 63.6 GB in training (the 16-step walks), 55.4 GB in evaluation; no non-finite gradient norm and no skipped step at any of the 385 logged steps; gradient norm median 0.41 after step 2,000, one spike of 3.97 at step 12,000 (the V_φ group, clipped; the loss fell), otherwise at most 1.08; largest rise between logged training losses 0.071 (step 13,050, as in the other eight-layer models); 0 watchdog triggers; SCAF leak monitor CLEAN at all audits.

Refinement

The test (Gate 3 of Cell 6b-7, and the in-training probe on the same batches). Hold the integration time T = 8 and walk the trained layers in N steps of size 8/N (step j uses trained layer ⌊8j/N⌋). If the layers were samples of one continuous flow, the loss would stop changing as N grows. The penalty is ln(PPL_N / PPL_8) in nats at step 18,000:

N	4	6	9	10	11	12	13	14	15	16	24	32
trained at N = 8 only (K2-L8-R6-CODE)	+3.683	+0.690				+0.251				+0.499	+1.279	+1.726
trained at N ∈ {8, 12, 16} (this model)	+2.568	+0.175	+0.005	−0.001	−0.004	−0.008	+0.006	−0.000	+0.003	+0.001	+0.038	+0.081

• Inside the trained band the whole stack refines, at step counts it never saw. From 8 to 16 steps the penalty stays within 0.01 nats; the untrained N = 9–11 and 13–15 read −0.004 to +0.006, although their layers get unequal shares of the time (at N = 9 the first layer takes two steps of 8/9 and the others one). It held at every probe from step 8,000 on.
• Beyond the band it nearly holds. At 24 and 32 steps, three and four steps per trained layer against at most two in training, the penalty is 0.038 and 0.081 nats, and both fell during training (0.079 and 0.107 at step 2,000). SR4a-CODE, the two-layer model trained the same way, paid 0.45 and 1.39 nats at the same three and four steps per layer.
• The qualifications. It is not a converged flow: the perplexity changes by 0.04 and 0.04 from 8 to 12 to 16 steps, then by 0.18 and 0.22 from 16 to 24 to 32, growing where a flow's would shrink. Fewer steps than the trained eight still fail (+0.175 nats at N = 6, +2.568 at N = 4). The operations applied once per step (the LayerNorm after each step, the top-k selection of the pair potential's sources) are still there; training at several step counts has made the model insensitive to how often they run, over this range. One seed.
• So depth is an inference-time choice for this model: 8 to 16 steps at no measurable cost, up to 32 at under 0.1 nats, with the same weights.

The per-layer flow (FLOW-R)

A second refinement test: each trained layer's share of T is integrated with k = N / 8 substeps, the layer's context and wells' quadratic frozen at its first substep and one LayerNorm projection at its end; the force, including the pair potential's top-k source selection, is recomputed at every substep (debug/refinement_flow_confirmation.py, 12 × 4 × 512 tokens, seed 20261009). Penalty ln(PPL_N / PPL_8) in nats:

model	k = 1.5	2	3	4	6	8	whole stack at k = 4
K2-L8-R6-VS-CODE (this model; trained ppl 5.44 on these batches)	0.382	0.030	0.033	0.034	0.034	0.034	0.085
K2-L8-R6-CODE (L = 8, two wells, rank 6; trained ppl 5.33 on these batches)	0.442	0.061	0.057	0.058	0.058	0.059	1.772
K2-L8-CODE (L = 8, two wells; trained ppl 5.47 on these batches)	0.557	0.089	0.096	0.097	0.097	0.098	2.798
L8-CODE (L = 8, eight wells)	0.186	0.029	0.034	0.035	0.035	0.035	2.695

• The per-layer flow settles 0.034 nats from the trained step (within 0.004 from k = 2 to k = 8), level with L8-CODE's eight wells (0.035) and closer than the fixed-step parent (0.059). RV-5 expected the layers to drift away from their flows, as SR4a-CODE's did at two layers (0.84 against 0.37); at eight layers they came closer. The whole stack at k = 4 costs 0.085 nats, against 1.77 for the parent: the per-step operations that make a stack a map cost about 0.05 nats here, against 1.71. (The k = 1.5 reading is high in every model: its uneven half-layer split is not the variable-step walk.)
• Kick share (the part of a layer's position update owed to the explicit kick, median by layer): 0.10, 0.16, 0.12, 0.16, 0.18, 0.23, 0.30, 0.32 (K2-L8-R6-CODE 0.11, 0.18, 0.12, 0.18, 0.17, 0.23, 0.29, 0.50).

Pre-registered calls

Proposed 2026-11-05 before the launch and not confirmed by the author; scored as proposed, on the relaunch. Probe penalties at step 18,000; paired against K2-L8-R6-CODE over steps 16,000–19,000 at N = 8.

call	prediction	called	result	outcome
RV-1	cost at N = 8: paired difference against K2-L8-R6-CODE at most +0.030 nats	60%	+0.020 (SD 0.0014)	hit
RV-2	refinement inside the trained set: probe penalty at N = 12 and 16 both ≤ 0.02 nats	65%	−0.008 and +0.001	hit
RV-3	interpolation: penalty at every untrained N from 9 to 15 ≤ 0.05 nats	45%	largest +0.006 (N = 13)	hit
RV-4	extrapolation fails: penalty at N = 24 above 0.20 nats	70%	+0.038	miss
RV-5	the per-layer flow moves away: FLOW-R limit at k = 8 above K2-L8-R6-CODE's 0.059	65%	0.034	miss (closer, not farther)
RV-6	stable: no non-finite gradient norm at any logged step	85%	none in 385 logged steps of the relaunch (the first launch: NaN from step 1,000)	hit

Paired by step at N = 8

The mean difference in log perplexity (nats, this model minus the other) over a span of evaluations, with the standard deviation of the per-evaluation differences.

steps	against K2-L8-R6-CODE	against K2-L8-CODE	against L8-CODE
2,000–4,000 (5 evals)	+0.142 (SD 0.048)	+0.126 (SD 0.047)	−0.214 (SD 0.072)
8,000–13,000 (11 evals)	+0.027 (SD 0.005)	+0.007 (SD 0.005)	+0.035 (SD 0.011)
16,000–19,000 (7 evals)	+0.020 (SD 0.001)	−0.003 (SD 0.001)	+0.053 (SD 0.001)

• The price of training at three step counts is +0.020 nats at N = 8 against the fixed-step parent, and it was still shrinking at the end: +0.22 at step 2,000, +0.03 by step 8,000, +0.018 at step 19,000. It gives back most of what rank 6 gained over rank 4 (level with K2-L8-CODE).

Through training

step	K2-L8-R6-CODE	this model	difference
500	51.67	55.18	+6.8%
1,000	27.77	29.68	+6.9%
1,500	17.89	20.77	+16.1%
2,000	13.15	16.36	+24.4%
2,500	10.79	12.83	+18.9%
3,000	9.63	10.90	+13.2%
4,000	8.56	9.32	+8.9%
5,000	7.91	8.33	+5.3%
8,000	7.15	7.38	+3.2%
10,000	6.80	6.96	+2.4%
13,000	5.90	6.05	+2.5%
16,000	5.66	5.77	+1.9%
18,000	5.44	5.56	+2.2%
19,000	5.64	5.74	+1.8%

Other diagnostics

• Inertia (Gate 1): resetting the velocity at every step multiplies perplexity by 2.96 (4.81 carried, 14.22 reset; K2-L8-R6-CODE × 2.28). The model leans on its velocity more, not less.
• Stiffness (Cell 6b-13): ω·Δt median 1.376 at Δt = 1 (K2-L8-R6-CODE 1.327); by layer 2.50, 1.33, 1.10, 1.88, 1.69, 1.38, 0.82, 0.54; 18.5% of readings past 2 (16.1%). Layer 0 is softer than the parent's, layers 1–4 stiffer. The integrator treats the stiff modes exactly, so this is a reading of stiffness, not of instability.
• The wells' rank (Cell 6b-4): participation ratio median 4.71 of 6 (p05 3.81, p95 5.30; K2-L8-R6-CODE 4.87); singular-value ratios 1.000, 0.790, 0.689, 0.618, 0.544, 0.457; σ_max² 0.332 (0.309). (The cell's printed verdict is written for rank 4.)
• Registers (Cell 7): removing all registers changes nothing (5.15 → 5.15); the reverse channel is off by design (norc).

The first launch, and the bug it found

The first launch (2026-11-05) trained for one 24-hour session with a NaN gradient norm on about 98% of steps from step 1,000, so the optimizer skipped them and the model hardly trained; the loss stayed finite and no alarm fired. The cause was in the wells' harmonic terms: a well's weight w = softmax(·) enters as sqrt(w), and with two wells a logit gap above about 104 underflows the float32 softmax to w = 0 exactly, where the square root's backward gives ∞ · 0 = NaN. Variable-step training had sharpened the well selection (the largest logit gap rose from 18 at step 500 to 107 at step 1,000; the fixed-step parent's final model reads 11), and why it does is open. The fix (15c236d) computes sqrt(w) only where w > 0 and gives such a well the gradient 0, its limit; the forward pass is bit-identical. The notebook now stops a run after 25 non-finite gradient norms in a row. This model is the relaunch from the clean step-500 checkpoint, with the batches and step-count draws of the first launch.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb at 15c236d or later, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor
LADDER_L=8                                # eight layers; dt = T / L = 1, T = 8
V_THETA_WELLS_PER_HEAD=2                  # two wells in the joint V_theta bank instead of eight
VPHI_FORCE_CHECKPOINT=True                # memory switch, same arithmetic
SNAPSHOT_STEPS=[1_000, 1_500, 2_000, 2_500, 3_000]   # weights-only snapshots for the well probe
ANISO_RANK=6                              # rank of each well's precision update (default 4); tag part ar6
VARIABLE_STEP_COUNTS=(8, 12, 16)          # N drawn per optimizer step over T = 8; tag part vs8-12-16
VARIABLE_STEP_PROBE_N=(0.5, 0.75, 1, 1.125, 1.25, 1.375, 1.5, 1.625, 1.75, 1.875, 2, 3, 4)   # probe N = 4, 6, 8-16, 24, 32

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 7, 6b-7, 6b-4 and 6b-13 (one A100 80 GB; 63.6 GB peak in training, 5.5 s per step, about 29 h over two Colab sessions; the autosave and resume keep the batch and step-count draws). The per-layer flow is debug/refinement_flow_confirmation.py … code seed=20261009 on the downloaded run folder (CPU).

Training artefacts were written to semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x2_ar6_vtjoint_cgqk_norc_vplive_xilive_sr2_vs8-12-16_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

| N (step count) | the number of steps the trained layers are walked in over the fixed time T = 8; N = 8 is one step per layer. Refinement means N > 8 at the same T |

Files

path	what
README.md	this card
config.json	architecture, training, result and diagnostics
checkpoints/fock_cfc_*noattn_step18000_best.pt	weights at the best eval (step 18,000), optimizer state stripped
results/	the training log (both sessions of the relaunch; steps up to 500 from the first launch, identical), the full Cell 6 output, the diagnostic cells' outputs (Cells 6b-4, 6b-7, 6b-13 and 7), the resonance and leak records and the per-layer flow (refinement_flow_confirmation_k2-l8-r6-vs-code_*)
code/	the notebook and the model code (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	the run records: entries p-k2-l8-r6-vs-setup and p-k2-l8-r6-vs-result
Refinement_Experiments_and_Variable_Step_Training.md	the refinement gates, the per-layer flow and variable-step training

Caveats

• One seed per model. The pairing removes the evaluation noise, not the training-seed spread.
• Qualified refinement, not a flow. Invariance over a band of step counts, reaching beyond the trained ones; the change still grows past 16 steps, and coarser steps than trained fail.
• The calls were never confirmed before the run; they are scored as proposed.
• Code only, without the register path. Perplexities on code are not comparable with other corpora's. The matched GPT-2 is untuned and has 8 layers at the same width.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-08.