Blame for README.md

---
pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - codeparrot
  - mechanism-ablation
  - ladder-study
  - pre-registered
  - live-gradients
  - generation-3
  - first-vs-second-order
  - first-order
  - code
  - matched-token-budget
  - fock-space
  - parflm
  - anisotropic-gaussian-potential
  - cfc
  - baoab
  - fully-conservative
  - riemannian-geometry
  - riemannian-geodesics
  - jacobi-metric
  - non-transformer
  - attention-free
  - constant-memory-inference
datasets:
  - codeparrot/codeparrot-clean-train
  - codeparrot/codeparrot-clean-valid
metrics:
  - perplexity
model-index:
  - name: semsimula-ladder-live-code-d384-l2-none-norc-sr2-fob
    results:
      - task:
          type: text-generation
        dataset:
          type: codeparrot/codeparrot-clean-valid
          name: CodeParrot clean (Python)
          split: validation
        metrics:
          - type: perplexity
            value: 6.5487
            name: Validation perplexity at block 512, settled (mean of the last three 500-step evals at block 512)
            verified: false
---

# SR2 on Python code as a true first-order gradient flow (FO-b-CODE): 9.2% behind, in identifiers recalled from far back (Gen 3)

> **The question of this collection:** does the conservative core of Semantic Simulation need second-order dynamics (a velocity and inertia), or does a first-order model trained the same way do as well? Each pair trains one model both ways, with everything else the same.
>
> **This model:** [SO-CODE](https://huggingface.co/dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-sr2), SR2's architecture on **Python code**, trained as a **true first-order gradient flow** (FO-b): the stiff modes relaxed exactly, no velocity anywhere. **Verdict: 9.2% behind its second-order twin, the largest gap on any corpus** (6.55 against 6.00; TinyStories +2.3%, OpenWebText +4.9%). The loss sits in identifiers and grows with the distance to their earlier occurrence; this model finds those occurrences as well as its twin and does less with them.
> [The first-vs-second-order collection](https://huggingface.co/collections/dimitarpg13/semantic-simulation-gen-3-first-vs-second-order-6ad7681596d9a670c268b524) holds the pairs.

## Result

| pair | corpus, tokens | second order | first order | first order against second |
| --- | --- | ---: | ---: | ---: |
| OpenWebText, no wells | 532.5M | [SR2](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2) **58.32** | [FO-b-OWT](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2-fob) **61.16** | **+4.9%** |
| TinyStories, true first order (FO-b), no wells | 315M | [SO-TS](https://huggingface.co/dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2) **5.16** | [FO-b-TS](https://huggingface.co/dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2-fob) **5.28** | **+2.3%** |
| Python code, true first order (FO-b), no wells | 315M | [SO-CODE](https://huggingface.co/dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-sr2) **6.00** | [FO-b-CODE](https://huggingface.co/dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-sr2-fob) **6.55** | **+9.2%** |
| Python code, with 2 Poisson-mode wells | 315M | [PM2-CODE](https://huggingface.co/dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-pm2-pmx16-sr2) **5.71** | [FO-b-PM2-CODE](https://huggingface.co/dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-pm2-pmx16-sr2-fob) **5.62** | **−1.6%** |

Without wells the gap grows with the corpus, from TinyStories (short children's stories) through OpenWebText to code. With two Poisson-mode wells it closes on the first two corpora; on code, with two wells, the first-order model is ahead.

- **This model: settled 6.55** (mean of the evals at steps 18,000, 18,500 and 19,000: 6.43, 6.57, 6.65); best 6.43 at step 18,000. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split, so no file is in both).
- **Trained on exactly 315,392,000 tokens** (19,250 steps × 32 × 512) in one session: the ladder's WSD schedule over 16,250 steps, then 3,000 steps at its floor learning rate, the TinyStories learning-rate curve. Python source with each four-space indentation level written as one tab. Both twins start from step 0 at seed 0 on the same batches.
- **Paired by step:** +9.1% to +9.5% over the last seven evaluations (steps 16,000–19,000; the two arms' evaluations correlate at 0.996), and FO-b-CODE is worse in 98% of 512 validation windows.
- **Against a transformer:** the matched GPT-2 (d = 384, L = 8) on the same code tokens and schedule settles at 4.96; this model is +32.0% behind it (its twin +20.9%). That model is not on the Hub yet.
- Pre-registered (protocol §5.13, MS5, calls written before either arm trained): CODE1 (FO-b-CODE settles more than 2% above SO-CODE), called 65%: HIT, +9.2%. CODE2 (the gap is larger than OpenWebText's +4.9%), called 35%: HIT. Scored without an extension: both arms fail the settling check by its letter (−0.020 and −0.023 nats over the last 3,000 steps, against a residual scatter of 0.015) but fall together, and the paired gap moves by 0.003 nats over that window. CODE3–CODE5 were scored on the two-well pair on code (all three HIT; see PM2-CODE and FO-b-PM2-CODE).
- Run health: 19,250 steps in 5.65 h in one session; 0 watchdog triggers; largest rise between logged training losses 0.073, at the same step as SO-CODE's (the batches are paired); gradient norm median 0.67, maximum 1.83 (in warmup); SCAF leak monitor CLEAN at all 4 audits; trained leak probe CLEAN at step 10,000.

## How the pair compares through training

| step | phase | SO-CODE (second order) | FO-b-CODE (first order) | first order against second |
| ---: | --- | ---: | ---: | ---: |
| 500 | warmup | 44.02 | 41.91 | −4.8% |
| 2,000 | stable | 17.67 | 16.60 | −6.1% |
| 5,000 | stable | 9.27 | 9.26 | −0.1% |
| 10,500 | stable, end | 7.02 | 7.77 | +10.7% |
| 13,000 | decay | 6.38 | 7.06 | +10.7% |
| 16,000 | decay, end | 6.12 | 6.70 | +9.5% |
| 17,000 | floor | 6.07 | 6.62 | +9.1% |
| 18,000 | floor, best | 5.87 | 6.43 | +9.5% |
| 19,000 | floor, last eval | 6.10 | 6.65 | +9.0% |

- **As on the other corpora, first order learns faster at first and then falls behind:** it leads until step 5,000, then trails by 11.1% at the end of the stable phase, 10.5% in the decay and 9.2% at the floor.

## Where the gap sits: the copy split

Both models' loss per token on the same 262,144 validation targets (512 windows of 512 tokens), grouped by what the context offers each target. *Identifier-like* tokens are words of letters, digits and underscores: names the programmer chose, and Python keywords. A target's *earlier occurrence* is its last appearance in the window before it.

| targets | share | FO-b − SO, nats | share of the gap | SO − GPT-2, nats |
| --- | ---: | ---: | ---: | ---: |
| all | 100% | +0.086 | 100% | +0.186 |
| identifier-like (names and keywords) | 49.3% | +0.152 | 87% | +0.253 |
| indentation, newlines, spaces | 21.9% | +0.008 | 2% | +0.042 |
| numbers, punctuation and other | 28.8% | +0.034 | 11% | +0.182 |
| first occurrence in the window | 27.7% | +0.085 | 27% | +0.189 |
| earlier occurrence 1–16 tokens back | 34.8% | +0.043 | 17% | +0.109 |
| earlier occurrence 17–64 tokens back | 25.2% | +0.102 | 30% | +0.208 |
| earlier occurrence 65–256 tokens back | 10.8% | +0.178 | 22% | +0.342 |
| earlier occurrence 257–512 tokens back | 1.5% | +0.197 | 4% | +0.395 |

- **The gap is in identifiers:** half the targets, 87% of the gap. Whitespace, a fifth of the targets, carries 2%.
- **It grows with the distance to the earlier occurrence,** and not because later positions are harder: among targets at positions 256–511 alone it is +0.04, +0.10, +0.18, +0.20 nats over the four distance bins.
- **It is not a failure to find the earlier occurrence.** The context channels ξ are causal weighted averages of earlier tokens' states with learned decays and a content term. When the earlier occurrence lies more than 64 tokens back, the longest channel of FO-b-CODE's second layer puts 24.5% of its weight on it (29 times its uniform share, over about 26 effective positions), SO-CODE's 22.3% (26 times, over about 36). Both look in the same places; the first-order model does less with what it finds.
- **SO-CODE's own gap to GPT-2 has the same shape at twice the size** (last column): the conservative core's weakness on code, against attention, is long-range recall of names, and first order makes it worse.

## Why second order helps here: a reading

| reading | SO-CODE (second order) | FO-b-CODE (first order) |
| --- | ---: | ---: |
| stiffness of V_θ's modes at the endpoint, median ω·Δt (Cell 6b-13) | **7.55** | **0.86** |
| the same, layer 0 / layer 1 | 9.58 / 4.94 | 1.17 / 0.002 |
| share of readings past the explicit-integrator wall (2.0) | 99.97% | 22.9% |

The first-order model builds a soft potential, as it does on OpenWebText (0.74 against 7.55). A gradient-flow step contracts a stiff mode by exp(−(ω·Δt)²/2) and erases the distinctions between tokens along it, so a first-order model cannot keep stiff modes and keep tokens apart; the second-order step turns a stiff mode through a phase and keeps its amplitude. A channel that reaches far back delivers the earlier identifier diluted, about a quarter of its weight spread over some 30 positions, and telling which of many names that mixture carries needs a sharply curved readout, which stiff modes provide. Nearby occurrences arrive almost undiluted through the short channels, and there first order is nearly level (+0.005 nats on tokens other than identifiers within 16 tokens). The softness is the same on every corpus; what changes is how many distinctions the prediction needs, which would explain why the gap orders the corpora as it does. **This is an interpretation of these readings, not a measured mechanism**, at one seed and L = 2. The argument is [`Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md) §3 and the book, §26.9 item (x).

## Off the trained schedule (Cell 6b-7)

| reading | SO-CODE (second order) | FO-b-CODE (first order) |
| --- | ---: | ---: |
| Gate 1: perplexity cost of resetting the velocity at inference | +4.0% | 0 (no velocity carried) |
| Gate 2: one extra layer at the trained step | +42% | +3.6% |
| Gate 2: eight layers | +6,186% | +3,987% |
| Gate 3: three steps over the trained time, nats | 0.639 | 0.277 |

- **One extra layer and standard refinement cost the first-order model little,** as on the other corpora: nothing it carries between layers is tuned to the step. Eight layers break both models.

## Reproduction

`colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb`, Cell 0 changed from the defaults:

```python
REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
FIRST_ORDER_OVERDAMPED=True,              # tag 'fob'
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor
```

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7 and 6b-13. The first code run builds the token cache (about 25 minutes). The copy split is `debug/copy_split.py` and the channels' reach `debug/xi_reach.py` (CPU, on the downloaded run folders).

Training artefacts were written to `semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_fob_st16250_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/`.

## Glossary

| term | meaning |
| --- | --- |
| settled PPL | validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval |
| Gen 1 / Gen 2 / Gen 3 | the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) *detached*, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients *live* |
| live / detached gradients | whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path |
| Vθ | the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token |
| Vφ | a pair potential between a token and a few selected earlier tokens (sparse top-k) |
| ξ | the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ |
| Fock mechanism, registers, reverse channel | a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part |
| exchange field | an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, `attention` and `attention_potential` |
| `attention` (non-conservative exchange field) | the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step |
| `attention_potential` (attention as a potential) | the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative |
| no-exchange (`LADDER_MECHANISM = 'none'`) | the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only |
| live exchange field (`RELAX_GRAD_PATH = 'live'`) | `attention_potential` with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state |
| Gen 2 probe | a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (`attention_potential` with `RELAX_GRAD_PATH = 'live'`, the full 32,500 steps) is the one that found the gradient starvation |
| gpt2-matched | the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3 |
| tag components (`vplive`, `xilive`, `rglive`, `norc`, `L4probe`, `idt…`) | pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt |
| conservative-only | a model in which every force is a gradient: no reverse channel and no non-conservative exchange field |
| CfC/BAOAB low-rank propagator | the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form |
| ω·Δt, θ | a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the *wall*); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π |
| geodesic step, R(geo), share | the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's *share* is how much it moves that distance (6b-9 attribution) |
| Gates 1–3, FLOW / MAPS | see **Terms** in the geometry section |
| CG1–CG7, 6b-N | the CfC-geometry experiments, and the ladder-notebook cells that run them |
| SCAF, Tier A, Tier B | the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model |
| leak tax, prefix-only | the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model |
| clip-hit | the share of logged steps whose gradient norm exceeded the clipping threshold (1.0) |
| pre-registered | a prediction recorded, with its threshold, before the run |

## Files

| path | what |
| --- | --- |
| `README.md` | this card |
| `config.json` | architecture, training, the first-vs-second-order switch, result and diagnostics |
| `checkpoints/fock_cfc_*noattn_best.pt` | weights at the best eval (step 18,000), optimizer state stripped |
| `results/` | the training log, the diagnostic cells' outputs, the copy split (`copy_split_report*`, against GPT-2 and against SO-CODE) and the channels' reach (`xi_reach_*`) |
| `code/` | the notebook and the model code the switches belong to (it also carries later, default-off switches) |

## Companion notes

| note | what it covers |
| --- | --- |
| [`Depth_Ladder_and_Matched_Baseline_Protocol.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Depth_Ladder_and_Matched_Baseline_Protocol.md) | §5.13: the code set (MS5), its calls, these results, the copy split and its reading |
| [`Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Future_of_Semantic_Simulation_When_Second_Order_Earns_Its_Keep.md) | what the equations say about when second order earns its keep |

## Caveats

- **One seed per model.** The pairing removes the evaluation noise, not the training-seed spread, which has not been measured; at 9.2% the gap is several times the differences of about 1% that the series reads as ties.
- **Not settled by the letter of the settling rule:** both arms still fall by about 0.02 nats over their last 3,000 steps, against a scatter of 0.015. They fall together, so the gap is settled; the levels may drop a little with more training.
- **The hyperparameters were tuned for the second-order models on OpenWebText** (learning rate 1.2e-3, the WSD fractions, the clip groups).
- **L = 2,** and perplexities on code are not comparable with OpenWebText's or TinyStories'.

## Citation

```bibtex
@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}
```

*Card generated from `_ladder/ladder.json` on 2026-11-10.*