semsimula-ladder-owt-d384-l2-gpt2-matched / README.md

Card: cite book v25 (paper v6.2, Nov 9, 2026); FLOW-R wording

06636ab verified 1 day ago

26.6 kB

pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - openwebtext
  - mechanism-ablation
  - ladder-study
  - pre-registered
  - matched-token-budget
  - transformer
  - gpt2
  - baseline
  - matched-baseline
datasets:
  - Skylion007/openwebtext
metrics:
  - perplexity
model-index:
  - name: semsimula-ladder-owt-d384-l2-gpt2-matched
    results:
      - task:
          type: text-generation
        dataset:
          type: Skylion007/openwebtext
          name: OpenWebText
          split: validation
        metrics:
          - type: perplexity
            value: 49.81
            name: >-
              Validation perplexity at block 512, settled (mean of the last
              three 500-step evals at block 512)
            verified: false

Matched GPT-2 baseline

Shared reference for Generations 2 and 3. A plain transformer: the gradient-convention change that supersedes Gen 2 does not touch it, and the same checkpoint anchors both ladders.

One arm of Generation 2 of the SemSimula mechanism ladder — OpenWebText, d=384, L=2: five models that differ by exactly one mechanism each, so that every gap in the ordering isolates one component by construction rather than by attribution.

This arm removes: the reference architecture.

The ladder

arm	mechanism removed	settled PPL	vs GPT-2
Matched GPT-2 baseline (this model)	the reference architecture	49.81	1.000×
Fock-PARFLM, exchange field on (arm 'attention')	the transformer itself	63.51	1.275×
Fock-PARFLM, exchange field as a potential (arm 'attention_potential')	the exchange field's non-conservative part — the same ξ-routed attention, entering as a scalar potential so its force is a gradient; the reverse channel is still on, and is not the gradient of anything, so the model as a whole remains non-conservative	80.90	1.624×
Fock-PARFLM, no exchange field (arm 'no-exchange')	the exchange field	66.98	1.345×
Conservative-only control — the Fock mechanism off	the Fock mechanism — the register-to-token path	87.93	1.765×

Italic entries are pre-registered predictions, recorded before the run. The ordering is itself the prediction; any inversion is a result.

Every arm above sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — on the same corpus, tokenizer and validation batches. No arm is trained longer than another, so a gap in the table is a gap between mechanisms and never a difference in budget.

What each gap compares

Every value below was measured under this generation's detached-source convention. Two of them were once read as prices of a mechanism; those readings are withdrawn and marked.

gap	compares	value
Matched GPT-2 baseline → Fock-PARFLM, exchange field on	what the transformer has that this architecture does not	49.81 → 63.51: +13.70 PPL (+27.5%), measured
Fock-PARFLM, exchange field on → Fock-PARFLM, exchange field as a potential	the same field, made a potential. Once read as the price of conservativity: withdrawn, see below	63.51 → 80.90: +17.39 PPL (+27.4%), measured
Fock-PARFLM, exchange field on → Fock-PARFLM, no exchange field	dropping the exchange field outright	63.51 → 66.98: +3.47 PPL (+5.5%), measured
Fock-PARFLM, no exchange field → Fock-PARFLM, exchange field as a potential	a conservative exchange field over none at all	66.98 → 80.90: +13.92 PPL (+20.8%), measured
Fock-PARFLM, no exchange field → Conservative-only control — the Fock mechanism off	the register-to-token path. Once read as the price of the Fock mechanism: withdrawn, see the banner	66.98 → 87.93: +20.95 PPL (+31.3%), measured

Each row compares the two arms it names: the value is the right-hand arm's settled PPL minus the left-hand arm's, both shown. It is not measured against the matched GPT-2; that ratio is the last column of the ladder table above.

Why the conservative exchange field did so badly — resolved

The rows for the arm attention_potential were this generation's least expected result: 27.4% worse than the arm attention, and 20.8% worse than no exchange field at all. The cause was the training signal, not conservativity.

To keep its force a gradient, the conservative field held its sources and routing fixed, and it did so by detaching them. Detaching changes no forward value, but the field then sent no gradient into any hidden state.
With the identical forward force written out over live tensors, the same arm settled at 61.11, below the arm attention's 63.51, with a 0.8% clip-hit rate against 29.2% (single seed).
The explanation previously given here, expressivity (the gradient ties the output projection to the query projection), is withdrawn as the cause of the gap.

The same detach applies to Vφ and ξ in every arm of this collection, which is why the collection is superseded rather than corrected in place.

What the ladder is measuring

The models below integrate a damped mechanical system in semantic space. With the register mechanism on, the equation of motion carries a non-conservative term and is Lagrangian only in the Lagrange–d'Alembert sense:

$m \ddot{h} = -\nabla V_\theta(h) - \gamma m \dot{h} + F_{\mathrm{rc}}(h, r) + F_\phi(h)$ $V_\theta$ is the pointwise scalar potential, $V_\phi$ the pairwise one, and $F_{\mathrm{rc}}$ the reverse channel through which the register bank acts on the token state. $F_{\mathrm{rc}}$ is not the gradient of anything in $h$ , so it is the term that makes the trajectory non-geodesic.

The damping $-\gamma m \dot{h}$ is not neutral either. It points along the velocity, but it does change the path, not only the speed: a trajectory bends with curvature $|F_\perp| / (m v^2)$ , so slowing down lets the potential bend it more sharply. A damped trajectory is therefore not a geodesic of a fixed Jacobi metric, which needs conserved energy. What these cards call a damped geodesic step is one integrated step of $m \ddot{h} = -\nabla V - \gamma m \dot{h}$ : the geodesic equation of the layer-dependent metric $\Omega^2_\ell = 2 T_\ell m$ with a friction term added, which is the object E1 measures.

Setting $F_{\mathrm{rc}} = 0$ leaves every force a gradient, so each layer step is a damped geodesic step in that sense, in the total potential with the context frozen for the layer, followed by the LayerNorm projection. It holds per step only: the trained models read MAPS, not FLOW, under refinement (6b-7), so the steps are not samples of one continuous curve. One arm in this collection is fully conservative in this sense: the arm conservative-only, with $V = V_\theta + V_\phi$ and the register-to-token path off. The multi-ξ SPLM, the arm splm-multixi (planned for Gen 3, not yet published), with $V = V_\theta$ alone, is the other. This generation measured the arm conservative-only at +31.3% against the arm no-exchange; that gap is withdrawn as a price, because its only inter-token channels had their sources detached. The arm attention_potential is not fully conservative: its exchange field enters as a potential, but its reverse channel is still on, so $F_{\mathrm{rc}} \neq 0$ .

This model


architecture	GPT-2 decoder, d=384, L=8, 6 heads, d_ff=1536, tied embeddings
parameters	33,691,776
training tokens	532,480,000 — the same on every arm
learning rate	0.0006
settled PPL	49.81
best PPL	49.76 (step 32,500)
final PPL	49.76
ratio to matched GPT-2	1.000×
pre-registration	below 54.59, point 49.5, band 49.0-50.0 — HIT, error +0.26
run health	reference arm

Protocol

Corpus OpenWebText (GPT-2 BPE, 50,257); 32,500 steps × 32 × 512 = 532,480,000 tokens.
Settled PPL is the mean of the last three 500-step evals at block 512 — not the best checkpoint.
Shared across all arms: tokenizer, corpus, validation batches, width d=384, block 512, effective batch 32, token budget (532,480,000 tokens, identical on every arm); the Fock arms also share the WSD schedule (warmup 5%, stable to 65%, decay to 100%, floor 6e-05).
Not shared: depth (GPT-2 L=8 vs Fock L=2), and the GPT-2 arm's learning rate and schedule (cosine, peak 6e-04, separately tuned; the schedule controls W1 and W2 on its card retrain it on the Fock ladder's WSD).
This model's schedule is cosine, not WSD: linear warmup over 2,000 steps, then a cosine decay from 6e-04 to 6e-05. The two schedule controls below retrain it on the Fock ladder's WSD.
Full protocol, pre-registered bands and run health: Depth_Ladder_and_Matched_Baseline_Protocol.md §5.4.

Reproduction

This model has its own notebook, colab_matched_gpt2_baseline_openwebtext.ipynb. Its Cell 0 defaults reproduce this checkpoint (learning rate 6e-04, cosine schedule, 32,500 steps; Cell 0 prints VARIANT_TAG = ''). Run Cell 0 → 1 → 2 → 3 → 4 → 5 → 6. Cell 4 must report batch 32 × 512 with no fallback warning, so that tokens per step equal the Fock arms'.

The schedule controls below change Cell 0 only:

# W1: the Fock ladder's WSD shape at this model's peak       (tag '_wsd')
LR_SCHEDULE = 'wsd'
# W2: the Fock ladder's schedule exactly, peak included       (tag '_wsd_lr0p0012')
LR_SCHEDULE = 'wsd'; LR = 1.2e-3

Schedule controls: W1 and W2

This model was trained on a cosine schedule, while every Fock arm was trained on a warmup-stable-decay (WSD) schedule at twice this peak learning rate. So any comparison with the Fock arms mixed architecture with schedule. Two controls retrain the same model, on the same data and token budget, on the Fock ladder's schedule. Each was pre-registered with a decision rule before it ran, and both fell inside the rule's "within 2%" band, so this checkpoint remains the reference.

Terms.

WSD schedule: a linear warmup, a long phase at the constant peak learning rate, then a decay to a floor.
End of the constant-rate phase: step 21,000, just before the decay begins.
Decay gain: how much the decay improves the model, the natural log of the perplexity at step 21,000 divided by the settled perplexity, in nats.

run	schedule	peak learning rate	settled PPL	PPL at step 21,000	decay gain (nats)
published (this checkpoint)	cosine: warmup 2,000 steps, decay to 6e-05	6e-04	49.81	—	—
W1	WSD: warmup 5%, constant to 65%, cosine decay to 6e-05	6e-04	48.82	64.01	0.271
W2	WSD, the Fock ladder's schedule exactly (peak, floor and shape)	1.2e-03	48.37	67.79	0.338

The schedule moves this model by under 3% (48.37 to 49.81).
On the Fock ladder's exact schedule, this model is behind the Fock models at the end of the constant-rate phase and overtakes them in the decay. At step 21,000 W2 reads 67.79, against 60.80 for L=4 Fock, 63.45 for G3′ and 63.61 for G2; its decay gain is 0.338 nats, against 0.18–0.22 for every Fock arm. Raising the peak from 6e-04 to 1.2e-03 widened that gap instead of closing it, so it is a property of the architecture, not of the learning rate.
Comparisons, both ways. Against W2, L=4 Fock (50.10) is +3.6%, G3′ +9.4% and PMX +12.9%; against this checkpoint they are +0.6%, +6.2% and +9.7%. A claim of parity should state both.
At 2B tokens, the same model on the Fock ladder's schedule at 1.2e-03 settles at 42.69, 11.7% below W2; it is the reference of the Gen 3 2B-token collection.
W1's and W2's weights are not published; neither is a reference. Pre-registrations, results and the stable-phase fits: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.17.

Training artefacts were written to semsimula_matched_gpt2_baseline_owt/.

Companion notes

Every number on this card is derived in one of these. They are the working notes, not a paper: pre-registrations sit next to the results that scored them, and withdrawn claims are marked withdrawn rather than deleted.

note	what it covers for this arm
`Depth_Ladder_and_Matched_Baseline_Protocol.md`	what is and is not matched between this baseline and the Fock arms -- depth and learning rate are not
`Hyperparameter_Tuning_Checklist.md`	how the learning rate was chosen, and the forecast record -- three hits and four misses, kept because the failures were systematic

Synchronization and binding (SY programme)

Do the couplings of these models synchronize tokens, and does that group related words? The question was put to every Gen 3 model and to the matched GPT-2 with criteria fixed before any measurement. Full results and figures: Spontaneous_Synchronization_and_Semantic_Binding_Test_Programme.md, Part II; pre-registrations and scoring: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.22.

Terms.

Coupling: any term through which another token's state moves this token: the pair potential $V_{\phi}$ , the Poisson-mode wells, the Fock register read, the exchange field, and the context field $ξ \xi$ , which couples tokens because it is built from the earlier ones.
Synchronization, contraction $S$ : how much closer the token states end up because of one coupling, measured against a replay in which that coupling pushes every token alike. $S \gt 0$ : drawn together; $S \lt 0$ : spread apart.
Binding contrast: how much more alike the states of related words are than those of unrelated words at the same distance in the text, in cosine units. An effect counts only if it is at least 0.02 and its 95% interval excludes zero.
Noun chunk: the words of one simple noun phrase, such as the old wooden table.
Head–dependent pair: a word and the word it grammatically depends on, such as keys and are in the keys to the cabinet are.
Reference, coreference, antecedent: what an expression stands for. Two expressions are coreferent when they pick out the same thing, as Mary and she do in Mary said she would come; the earlier one, Mary, is the antecedent. Binding reference means linking a pronoun to the expression it stands for.
Prediction control: what remains of a grouping after removing everything that the similarity of the words' predicted next words explains. A grouping that survives it is not just a side effect of predicting the next word.
Coreference probe: a small attention-shaped reader trained to pick each pronoun's antecedent from the states. It is compared with always picking the nearest earlier mention, which is right 54.1% of the time (chance: 13.6%).
Extra depth, over-synchronization: the trained model run for more steps than it was trained for, at the trained step size. Over-synchronization is the tokens drawing together ($S \geq 0.05$) or losing dimensions (effective rank down by 20% or more) as it fails.

Findings across the models (each call was pre-registered):

The couplings do draw tokens together (SY-L0: hit), the pair potential modestly and the Poisson wells strongly, though toward their fixed centres rather than toward each other. None groups related words on the routes the test was designed around (SY-L1: miss).
The context field $ξ \xi$ groups the words of noun chunks in every model, and the grouping survives the prediction control. It spreads the tokens apart overall while drawing each chunk's words together: clustered synchronization, as in oscillator networks that split into synchronized groups.
The matched GPT-2 forms the same grouping, built by attention in its middle layers. The grouping is generic to language models at this budget; how it is made is not.
No model binds reference. No probe beats the nearest-mention rule by a margin the test can resolve, in these models or in GPT-2.
Extra depth collapses the tokens at the catastrophic stage of its failure (SY-G2: hit), not at its onset.

Measured on this model (the matched GPT-2):

measurement	result
contraction by attention	attention spreads the token set at every layer (−0.21 and −0.26 at the first two)
noun-chunk grouping of the final states	+0.054, and +0.050 after the prediction control: as strong as in any Lagrangian model
who makes it	attention, gradually in layers 2–5 (+0.026 at layer 3), nothing at the last layer; the Lagrangian models make it through ξ at the last step
coreference probe, last layer / best layer	0.54 / 0.64 (layers 2–4); the best is not significantly above the nearest-mention baseline (0.541)
pre-registered SY-GPT2	miss, 0 of 3: at the last layer attention makes less binding than the best Lagrangian coupling

Theoretical foundations of the ladder

The design and theory behind every arm in this collection: the integrator they share, the potential they share, and the mechanisms the ladder compares. The same list appears on every card.

note	what it establishes
`Closed_Form_and_Hybrid_Integration_Strategies_for_Fock-PARFLM.md`	why the integrator is closed-form: which parts of the Vθ dynamics admit exact propagation, and three strategies in decreasing order of exactness, following the CfC lesson from liquid neural networks
`Blended_CfC_BAOAB_Deep_Dive.md`	the full construction of the CfC-BAOAB step every arm integrates with: the Gaussian Vθ's exact harmonic propagator placed inside the BAOAB Langevin splitting, every matrix written out
`Modified_BAOAB_with_STP_identity_Detailed_Analysis.md`	the BAOAB splitting with the STP identity as the simulator's integrator backbone: component-by-component analysis, coupling to the equations of motion, theoretical properties and trade-offs
`Analytic_Multi_Channel_Integration_in_Structured_Vtheta.md`	the joint Vθ (`vtjoint` in the run tags): the five ξ context channels fused into one closed-form well bank instead of summed as separable ones, and why CfC-BAOAB stays valid
`Conservative_Obstruction_and_Virtual_Particle_Necessity.md`	the theorem the ladder tests: no scalar potential on the tokens alone reproduces attention's structure, and the Fock registers are the minimal extension. Gen 2 read the +31.3% Fock gap as that obstruction's price; the reading is withdrawn (detached sources, see the banner)
`Fock_Mechanism_v2.1_Principle_of_Operation.md`	how the Fock mechanism of these models works, as implemented: the exchange field, the registers and their write port, and the reverse channel that is the registers' only read port; their equations, the virtual-particle reading, the relation to attention, and measurements on the trained G3′. An exploratory tool for studying attention-like routing in a conservative model
`Measuring_the_Price_of_Conservativity.md`	the λ-gated relaxation protocol first proposed to price conservativity, and how the ladder's pair the arm attention → the arm attention_potential answers the same question by construction instead
`Geodesic_Preservation_Experiment.md`	why measuring the geometry of the model's own dynamics is the capability to demonstrate, and the design and first analyses of the geodesic measurements (d=384 and d=1024)
`Composing_Single_Layer_Inferences_Flow_or_Maps.md`	whether the layer stack is a continuous flow or a sequence of maps; Gates 0–3 on the L=2 the arm no-exchange returned MAPS

Files

path	what
`README.md`	this card
`config.json`	architecture, training and result, machine-readable
`checkpoints/*_best.pt`	weights at the best eval
`checkpoints/*_step25000.pt`	an intermediate checkpoint; see the note below
`results/training_log.jsonl`	per-eval record; the settled PPL above is the mean of its last three rows
`results/*_training_output.txt`	the full run log, including per-step diagnostics
`results/gpt2_baseline_summary.json`	end-of-run summary; read `NOTE_summary_json_fock_fields.md` beside it first
`code/`	everything needed to reproduce the run, below

The code, and why this subset

code/ mirrors the layout of the companion repository, so the notebook's own sys.path lines resolve unchanged.

path	what
`code/colab_matched_gpt2_baseline_openwebtext.ipynb`	the whole run. Self-contained: the model is defined in Cell 3 and nothing is imported from the companion repository

Caveats

This is a reference, not a tuned transformer. It exists to give the ladder a scale, and it is the only arm whose learning rate was tuned separately (6e-04). Read it as "what a conventional architecture reaches on this budget", not as a claim about GPT-2.
Depth is not matched. L=8 here against L=2 on every Fock arm. The ratio column elsewhere in the collection is for orientation, not a like-for-like efficiency claim.
The second checkpoint is not the same kind as the other arms'. The Fock arms each ship step 500, essentially untrained. This arm has no step-500 checkpoint; its second file is step 25,000 at PPL 52.87, already within 6% of the endpoint. Do not read a cross-arm comparison of "first" checkpoints as like-for-like.
gpt2_baseline_summary.json carries a stale cross-reference. Its fock_* fields and ratio_fock_to_gpt2_settled are not a ladder arm: the file was written before any L=2 arm existed and those numbers come from a pre-ladder L=8 run read mid-schedule. The GPT-2 fields in it are correct. NOTE_summary_json_fock_fields.md sits beside it.
OpenWebText only.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-08.