README.md

metadata

pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - openwebtext
  - mechanism-ablation
  - ladder-study
  - pre-registered
  - live-gradients
  - generation-3
  - first-vs-second-order
  - first-order
  - poisson-mode-registers
  - matched-token-budget
  - fock-space
  - parflm
  - anisotropic-gaussian-potential
  - cfc
  - baoab
  - fully-conservative
  - riemannian-geometry
  - riemannian-geodesics
  - jacobi-metric
  - non-transformer
  - attention-free
  - constant-memory-inference
datasets:
  - Skylion007/openwebtext
metrics:
  - perplexity
model-index:
  - name: semsimula-ladder-live-owt-d384-l2-none-norc-pm64-pmx16-sr2-fob
    results:
      - task:
          type: text-generation
        dataset:
          type: Skylion007/openwebtext
          name: OpenWebText
          split: validation
        metrics:
          - type: perplexity
            value: 54.02
            name: >-
              Validation perplexity at block 512, settled (mean of the last
              three 500-step evals at block 512)
            verified: false

PMX trained as a true first-order gradient flow (FO-b-PMX): level with its second-order twin — the wells are a first-order memory (Gen 3)

The question of this collection: does the conservative core of Semantic Simulation need second-order dynamics (a velocity and inertia), or does a first-order model trained the same way do as well? Each pair trains one model both ways, with everything else the same.

This model: PMX, the conservative core with 64 Poisson-mode memory wells, trained as a true first-order gradient flow (FO-b-PMX): the wells and the stiff modes relaxed exactly, no velocity anywhere. Verdict: the first-order twin is level (54.02 against 54.63, −1.1%, a tie on any plausible seed spread). Without the wells second order was worth 4.9%; with them, nothing measurable. The reading proposed for this pair of facts is the memory-substitution hypothesis below. The first-vs-second-order collection holds the pairs; the Gen 3 collection holds the OpenWebText ladder.

Result

pair	corpus, tokens	second order	first order	first order against second
OpenWebText	532.5M	SR2 58.32	FO-b-OWT 61.16	+4.9%
TinyStories, first order between layers (FO-a)	315M	SO-TS 5.16	FO-TS 5.13	−0.6%
OpenWebText, with 64 Poisson-mode wells	532.5M	PMX 54.63	FO-b-PMX 54.02	−1.1%
OpenWebText, with 16 Poisson-mode wells	532.5M	PMX-K16 54.24	FO-b-PMX-K16 53.73	−0.9%
OpenWebText, with 2 Poisson-mode wells	532.5M	PM2-PMX16 54.71	FO-b-PM2-PMX16 53.81	−1.6%
TinyStories, true first order (FO-b), no wells	315M	SO-TS 5.16	FO-b-TS 5.28	+2.3%
TinyStories, with 2 Poisson-mode wells	315M	PM2-TS 4.99	FO-b-PM2-TS 4.96	−0.5%
Python code, true first order (FO-b), no wells	315M	SO-CODE 6.00	FO-b-CODE 6.55	+9.2%
Python code, with 2 Poisson-mode wells	315M	PM2-CODE 5.71	FO-b-PM2-CODE 5.62	−1.6%

• This model: settled 54.02 (mean of the last three evals, 53.95, 54.16, 53.96); best 52.18 at step 31,000. Validation perplexity at block 512, on OpenWebText's own validation tokens.
• Trained on exactly 532,480,000 tokens (32,500 steps × 32 × 512), the Gen 3 ladder's budget and schedule.
• Pre-registered: FBX1, second order earns its keep in the wells (FO-b-PMX at least 3% behind PMX), called 60%: MISS, −1.1%. FBX2, the gap with wells exceeds the gap without by 2 points, called 55%: MISS (−1.1% against +4.9%). FBX3, the wells help a first-order model (FO-b-PMX below FO-b-OWT), called 60%: HIT, −11.7%. FBX4, the first-order wells train stably, called 70%: HIT.
• Run health: 0 watchdog triggers, 0 spike captures; largest gradient norm 1.32; largest jump of the training loss above its running median ×1.006; SCAF leak monitor CLEAN at every audit.

What first order means here

PMX's layer integrates, exactly and with damping, one frozen spring on all of the hidden space: the 16 stiffest modes of the well potential V_θ together with the memory wells' quadratic at the step's start (an isotropic curvature per token and the rank parts of its 16 strongest wells, of either sign), as damped oscillators, with the velocity handed from layer to layer. FO-b-PMX relaxes the same spring first order: each direction moves by the mobility × step × a relaxation factor times its force, with the mobility the layer step divided by twice the learned mass; a direction of negative stiffness (the outer slope of an attractive well) grows instead of relaxing, exactly as in the second-order flow. The rest of the force takes one gradient step, no velocity exists anywhere, and no parameter is added. The switch was verified on PMX's weights before the run: off, bit-identical; the relaxations match a fine solve to 1e-14; the per-layer flow converges to the gradient flow at first order.

How the pair compares through training

step	phase	PMX	FO-b-PMX (this model)	this model against PMX
500	warmup	454.67	450.37	−0.9%
5,000	constant	96.46	96.39	−0.1%
10,000	constant	75.60	74.77	−1.1%
15,000	constant	68.73	67.83	−1.3%
20,000	constant	65.47	63.19	−3.5%
22,500	decay: the largest lead	64.18	60.90	−5.1%
25,000	decay	61.20	59.69	−2.5%
30,000	decay	55.83	53.80	−3.6%
32,500	decay, end	53.21	53.96	+1.4%

• Level throughout, ahead at the end of the stable phase: this model was ahead at 47 of 65 evaluations, by up to 5.1% (step 22,500).
• The decay closed most of it: from step 21,000 to the end PMX gained 0.218 nats, this model 0.173; the settled difference, 0.011 nats, is inside PMX's own evaluation scatter.
• The wells did not destabilise the gradient flow (FBX4): no spike, largest gradient norm 1.32.

Where the two models differ: the wells (Cell 6b-15)

reading	PMX	FO-b-PMX (this model)
half-life of a mode, p05 / median / p95 (tokens)	3.1 / 9.1 / 21.2	3.2 / 8.4 / 19.8
wells with attractive (positive) depth, layer 0 / layer 1	50% / 48%	64% / 61%
median depth, layer 0 / layer 1	-0.001 / -0.009	+0.031 / +0.071
depth tails at layer 1 (p05, p95)	-0.54, +0.55	-0.22, +0.58
wells' force against the conservative force, layer 0 / layer 1	0.81 / 4.79	0.60 / 3.32
share of the wells' force carried by the longest-lived quarter of modes, layer 0 / layer 1	45% / 38%	33% / 19%
by the shortest-lived quarter	9% / 16%	14% / 38%
context channels' decay rates at the end of training	0.386, 0.521, 0.887, 0.981, 0.995	0.378, 0.467, 0.879, 0.986, 0.991

The first-order model built its memory from attractive wells. PMX's depths are balanced between attractive and repulsive, with symmetric tails; FO-b-PMX's are attractive by a majority, and its repulsive tail at layer 1 is less than half as deep. Under a gradient flow an attractive well captures every token that enters it; a token with inertia can pass through, and PMX uses repulsive wells as much as attractive ones. The half-lives are the same, so the two models remember over the same horizons and differ in how the memory acts. At layer 1 the first-order model's force is carried by its shortest-lived modes.

The memory-substitution hypothesis

Two facts need one explanation: on the conservative core second order is worth 4.9% (and critical damping of the stiff modes, which keeps the velocity but removes their oscillation, is worse than removing inertia altogether), and with the wells the first-order twin loses nothing, while the wells help the first-order model twice as much (11.7% against 6.3%). A second-order step carries an implicit memory: its layer map reads two past states, and inside a step a lightly damped stiff mode keeps the phase-space area of a cloud of tokens, the distinctions between them, where a gradient flow contracts it to nothing. The wells are an explicit memory: an exponentially weighted trace of earlier tokens' overlaps with each mode, addressed by content, acting through a force. The hypothesis: the second-order advantage is the advantage of a memory, and where an explicit memory carries at least the same information, the implicit one is redundant. A hypothesis of function, not of mechanism, supported at one scale on one corpus with one seed. Its pre-registered tests (protocol §5.13, MS1–MS7) start with the seed spread and a dose-response in the number of wells (K = 16 and 256 against the recorded 0 and 64); critical damping on the well base and the Fock register path follow. The full argument: Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md (§7) and the book, §26.9.

Qualified 2026-10-24. Measured on the trained models (protocol §5.13), the wells do not use content: a mode's overlap varies by 1–4% from token to token, the occupations are 99.8% or more a function of the position, and giving a sequence another text's occupations costs 0.02% (this model) to 0.09%. Beyond the first ~100 tokens they act as a fixed extra landscape. So the "explicit memory" that closes the gap here is a positional, then fixed, potential rather than a memory of what came earlier; the hypothesis stands as one of function, with this narrower mechanism. The K = 16 pair (PMX-K16 and its first-order twin) extends the series.

Off the trained schedule (Cell 6b-7) and the per-layer flow (FLOW-R)

reading	this model	its pair partner
Gate 1: perplexity cost of resetting the velocity at inference	0 (no velocity carried)	+91.3%
Gate 2: one extra layer at the trained step	+0.3%	+956.5%
Gate 2: eight layers	+5,470%	+23,883%
Gate 3: three steps over the trained time (standard refinement), nats	0.666	1.097
per-layer flow: penalty at N = 16, nats	0.584	0.387

Per-layer flow of this model (FLOW-R, 12 × 4 × 512 fresh tokens; the context and the stiff modes' quadratic held at each layer's entry, one LayerNorm projection per layer), perplexity by the number of steps N over the trained time T = 8; N = 2 is the trained model, and N = 4, 6, 8 split each of its two layers into 2, 3, 4 substeps:

steps N	2 (trained)	3	4	6	8	12	16
perplexity	50.89	203.82	105.28	108.11	82.50	93.58	91.24
penalty, nats	0	1.388	0.727	0.753	0.483	0.609	0.584

• Extra layers are harmless: one more layer costs +0.3% against PMX's +956.5%, the momentum lock PMX's step-8,000 probe first showed. The first-order well model is the most extensible model in the series: nothing it carries between layers is tuned to the step.
• Its standard refinement is better than PMX's (Gate 3 at three steps 0.666 nats against 1.097) and its per-layer flow has not converged (it moves 0.025 nats between N = 12 and 16), to about 0.58 nats above the trained step, against PMX's 0.39.

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
POISSON_MODES=64, LOWRANK_DAMPED_FLOW=True, POISSON_WELLS_EXACT=True,   # PMX
FIRST_ORDER_OVERDAMPED=True,              # tag 'fob'
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False   # speed switches, same arithmetic

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7 and 6b-7. Cell 5b prints the Poisson-mode, SR2 and FO-b blocks; Cell 6b-15 reads the trained wells; Cell 6b-13's FO-b recorder reads the first-order relaxation routine (added 2026-10-22; the version shipped under code/ has it). The per-layer flow is debug/refinement_flow_confirmation.py (CPU).

Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_pm64_pmx16_sr2_fob_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.

Glossary

term	meaning
settled PPL	validation perplexity at block 512, the mean of the last three evals (steps 31,500–32,500). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3	the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients	whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ	the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ	a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ	the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel	a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field	an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field)	the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential)	the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none')	the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live')	attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe	a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched	the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…)	pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only	a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator	the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ	a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share	the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS	see Terms in the geometry section
CG1–CG7, 6b-N	the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B	the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only	the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit	the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered	a prediction recorded, with its threshold, before the run

Files

path	what
README.md	this card
config.json	architecture, training, the first-vs-second-order switch, result and diagnostics
checkpoints/fock_cfc_*noattn_best.pt	weights at the best eval (step 31,000), optimizer state stripped
results/	the training log and full run outputs, Cell 6b-7, the per-layer flow and the endpoint stiffness
code/	the notebook and the model code the switches belong to

Companion notes

note	what it covers
Depth_Ladder_and_Matched_Baseline_Protocol.md	§5.13: the first-vs-second-order series, its pre-registrations, amendments and scored results
Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md	what the equations say about when second order earns its keep, revised after this series' OpenWebText result

Caveats

• One seed per model. The yardstick for small gaps, the seed spread, has not been measured.
• The hyperparameters were tuned for the second-order models (learning rate 1.2e-3, the WSD fractions, the clip groups).
• PMX ran without the speed switches (the analytic pair force and checkpointing off are the same arithmetic, verified) and resumed once; this model ran straight through. FO-b adds no parameters, so the two start from the same initial weights.
• A tie needs a yardstick. The seed spread is first in the queue; until it is measured, −1.1% is read as level, not as a first-order win.
• L = 2. One layer boundary is the weakest possible test of a velocity carried between layers.
• OpenWebText only; the two pairs differ in corpus and token budget, so perplexities compare within a pair, not across pairs.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-10.