README.md

metadata

pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - openwebtext
  - mechanism-ablation
  - ladder-study
  - pre-registered
  - live-gradients
  - generation-3
  - first-vs-second-order
  - first-order
  - poisson-mode-registers
  - matched-token-budget
  - fock-space
  - parflm
  - anisotropic-gaussian-potential
  - cfc
  - baoab
  - fully-conservative
  - riemannian-geometry
  - riemannian-geodesics
  - jacobi-metric
  - non-transformer
  - attention-free
  - constant-memory-inference
datasets:
  - Skylion007/openwebtext
metrics:
  - perplexity
model-index:
  - name: semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2-fob
    results:
      - task:
          type: text-generation
        dataset:
          type: Skylion007/openwebtext
          name: OpenWebText
          split: validation
        metrics:
          - type: perplexity
            value: 53.73
            name: >-
              Validation perplexity at block 512, settled (mean of the last
              three 500-step evals at block 512)
            verified: false

PMX-K16 trained as a true first-order gradient flow (FO-b-PMX-K16): ahead of its second-order twin (Gen 3)

The question of this collection: does the conservative core of Semantic Simulation need second-order dynamics (a velocity and inertia), or does a first-order model trained the same way do as well? Each pair trains one model both ways, with everything else the same.

This model: PMX-K16 (16 Poisson-mode wells) trained as a true first-order gradient flow: the frozen spring of the wells and the stiff modes relaxed exactly, no velocity anywhere. Verdict: ahead of its second-order twin (53.73 against 54.24, −0.9%; paired by step −0.87%, spread 0.14%). Without wells second order was worth 4.9%; with 16 or 64 wells, nothing. The first-vs-second-order collection holds the pairs; the Gen 3 collection holds the OpenWebText ladder.

Result

pair	corpus, tokens	second order	first order	first order against second
OpenWebText	532.5M	SR2 58.32	FO-b-OWT 61.16	+4.9%
TinyStories, first order between layers (FO-a)	315M	SO-TS 5.16	FO-TS 5.13	−0.6%
OpenWebText, with 64 Poisson-mode wells	532.5M	PMX 54.63	FO-b-PMX 54.02	−1.1%
OpenWebText, with 16 Poisson-mode wells	532.5M	PMX-K16 54.24	FO-b-PMX-K16 53.73	−0.9%
OpenWebText, with 2 Poisson-mode wells	532.5M	PM2-PMX16 54.71	FO-b-PM2-PMX16 53.81	−1.6%
TinyStories, true first order (FO-b), no wells	315M	SO-TS 5.16	FO-b-TS 5.28	+2.3%
TinyStories, with 2 Poisson-mode wells	315M	PM2-TS 4.99	FO-b-PM2-TS 4.96	−0.5%
Python code, true first order (FO-b), no wells	315M	SO-CODE 6.00	FO-b-CODE 6.55	+9.2%
Python code, with 2 Poisson-mode wells	315M	PM2-CODE 5.71	FO-b-PM2-CODE 5.62	−1.6%

• This model: settled 53.73 (mean of the last three evals, 53.77, 53.81, 53.61); best 51.82 at step 31,000. Validation perplexity at block 512, on OpenWebText's own validation tokens.
• Trained on exactly 532,480,000 tokens (32,500 steps × 32 × 512), the Gen 3 ladder's budget and schedule, from step 0 at seed 0 like its twin: the two runs see the same training and evaluation batches, so their difference is measured paired (their step-to-step evaluation changes correlate at 0.999).
• Pre-registered: MS2a (the second-order gap at K = 16 in +1% to +4%), called 50%: MISS, −0.9%, first order ahead. MS2b (first-order half: between FO-b-OWT's 61.16 and FO-b-PMX's 54.02), called 65%: MISS, 53.73, sixteen wells as good as sixty-four. MS2c (the wells help first order more), called 60%: HIT, 13.0% against 7.3%. MS2d (stable training), called 75%: HIT. MS2e (within ±2% of PMX-K16, paired by step), called 55%: HIT, −0.87%.
• Run health: 0 watchdog triggers, 0 spike captures; largest training loss 1.007 times its running mean; SCAF leak monitor CLEAN at all 7 audits.

How the pair compares through training

step	phase	PMX-K16 (second order)	FO-b-PMX-K16 (first order)	first order against second
500	warmup	459.08	452.39	−1.5%
5,000	constant	97.17	95.27	−2.0%
10,000	constant	74.79	74.14	−0.9%
16,000	constant	65.42	64.67	−1.1%
21,000	constant, end	64.69	63.72	−1.5%
25,000	decay	60.32	59.57	−1.2%
28,000	decay	56.10	55.44	−1.2%
30,000	decay	53.98	53.52	−0.9%
32,500	decay, end	54.17	53.61	−1.0%

• The first-order model leads from the first evaluations to the end: by 1.9% over the first 5,000 steps, 1.1% through the stable phase and the decay, 0.9% over the last eight evaluations (spread 0.14%).
• The decay-phase pattern of earlier pairs did not appear: from step 21,000 to the end the second-order model gained 0.176 nats, the first-order model 0.171.
• Sixteen wells do as well as sixty-four, in both orders (PMX-K16 54.24 against PMX 54.63; FO-b-PMX-K16 53.73 against FO-b-PMX 54.02).

Reliance is not need

Resetting this pair's second-order model's velocity at inference costs +5,788% (PMX at 64 wells: +91%), the most of any model in the series: with fewer wells, the second-order model carries more in its velocity and its stiff rotation (V_θ's stiffest-mode ω·Δt median 2.87 / 4.55 at layer 0 / 1, against PMX's 2.19 / 4.21). Yet the twin with no velocity at all is ahead. A state variable present in training is loaded with information whether or not a model without it could have put that information elsewhere.

What replaced the velocity (pre-registered readings M1–M4)

Registered before the first-order twin's result: if the twin matched, the readings below would name what took the velocity's place, each against a threshold relative to the second-order model.

reading	second order (PMX-K16)	first order (FO-b-PMX-K16)	verdict
M1: the wells reorganise (layer-0 attractive share of the wells' force; modes in use)	4%; 11.0 / 13.6 of 16	23%; 12.7 / 9.5 of 16	not fired
M2: token–token coupling (cost of removing the pair force V_φ)	+1.71%	+3.60%	fired (2.1 times)
M2: context (cost of another text's context channels)	+334%	+337%	not fired
M3: a stiffer landscape (V_θ's stiffest-mode ω·Δt, layer 0 / 1)	2.87 / 4.55	2.13 / 5.31	fired, narrowly (layer 1 +17%)

The first-order model leans more on direct token–token coupling and builds a stiffer second layer. It does not lean harder on context, and its wells do not reorganise by the registered criterion. The K = 64 pair shows the same signature (pair force 1.7 times, layer 1 21% stiffer, context unchanged). These are reliance measurements on the trained models, not what models trained without these terms would reach.

What the wells remember

reading	its twin (second order)	this model (first order)
wells with attractive depth, layer 0 / layer 1 (Cell 6b-15)	19% / 94%	31% / 75%
median depth, layer 0 / layer 1	-0.029 / +0.392	-0.051 / +0.195
half-life of a mode, p05 / median / p95 (tokens)	4.2 / 7.9 / 15.7	3.3 / 5.1 / 16.7
wells' force against the conservative force, layer 0 / layer 1	0.53 / 3.53	0.56 / 3.24
wells' force reproduced from position alone	99.96% / 99.95%	99.90% / 99.98%
cost of another text's occupations / of removing the wells	+0.09% / +156%	+0.01% / +343%
cost of fixing the occupations at their late value: first 16 tokens / from token 128	+15% / +0.3%	+29% / +0.1%

The wells act as a positional, then fixed, landscape, not a memory of content. The occupations follow the position, another text's occupations cost nothing, and beyond the first tokens a single fixed value does as well. The models depend on that landscape heavily, the first-order model about twice as much as its twin.

Off the trained schedule (Cell 6b-7) and the per-layer flow (FLOW-R)

reading	this model	its pair partner
Gate 1: perplexity cost of resetting the velocity at inference	0 (no velocity carried)	+5,788%
Gate 2: one extra layer at the trained step	+0.5%	+845.5%
Gate 2: eight layers	+6,117%	+2,357%
Gate 3: three steps over the trained time (standard refinement), nats	0.859	1.092
per-layer flow: penalty at N = 16, nats	1.414	0.333

Per-layer flow of this model (FLOW-R, 12 × 4 × 512 fresh tokens; the context, the stiff modes' and the wells' quadratic held at each layer's entry, one LayerNorm projection per layer), perplexity by the number of steps N over the trained time T = 8; N = 2 is the trained model:

steps N	2 (trained)	3	4	6	8	12	16
perplexity	50.85	164.56	204.38	201.27	177.92	203.06	209.19
penalty, nats	0	1.174	1.391	1.376	1.252	1.385	1.414

• This model's per-layer flow has not converged (it moves 0.030 nats between N = 12 and 16), 1.41 nats above the trained step; its twin's sits at 0.33. At 64 wells the pair read 0.39 (second order) and 0.58 (first order).
• So the first-order model wins on perplexity at the trained step and departs much further from its continuous law: it learned a coarse two-step map, which refinement breaks. Whether that gap grows with training is the 2B-token test TL3 (protocol §5.13).

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
POISSON_MODES=16, LOWRANK_DAMPED_FLOW=True, POISSON_WELLS_EXACT=True,   # PMX with 16 wells
FIRST_ORDER_OVERDAMPED=True,              # tag 'fob'
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False   # speed switches, same arithmetic

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7, 6b-9, 6b-13 and 6b-15. The readings M1–M3 and the wells' measurements are CPU tools in notebooks/conservative_arch/scaleup/debug/ (pm_mode_usage.py, pm_content_swap.py, pm_static_landscape.py, mechanism_ablations.py, vtheta_omega_dt.py); the per-layer flow is refinement_flow_confirmation.py.

Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_pm16_pmx16_sr2_fob_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (steps 31,500–32,500). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, the first-vs-second-order switch, result and diagnostics
checkpoints/fock_cfc_*noattn_best.pt	weights at the best eval (step 31,000), optimizer state stripped
results/	the training log, the diagnostic cells' outputs, and the CPU readings of the wells, M1–M3 and the per-layer flow
code/	the notebook and the model code the switches belong to (it also carries later, default-off switches)

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	§5.13: the series, the memory-substitution calls (MS), the K = 16 pair's result and M1–M4
Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md	what the equations say about when second order earns its keep
Implementing_Semantic_Templates_A_Path.md	why the trained wells do not match content, and the content-selective design (Stage 0′)

Caveats

• One seed per model. The pairing removes the evaluation noise (to about 0.1%), not the training-seed spread, which has not been measured.
• The hyperparameters were tuned for the second-order models.
• L = 2. One layer boundary is the weakest possible test of a velocity carried between layers.
• The mechanism readings are reliance, not need: they measure what each trained model routes through a term.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-10.