Blame for README.md

---
pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - openwebtext
  - mechanism-ablation
  - ladder-study
  - pre-registered
  - live-gradients
  - generation-3
  - position-dependent-damping
  - second-order
  - matched-token-budget
  - fock-space
  - parflm
  - anisotropic-gaussian-potential
  - cfc
  - baoab
  - fully-conservative
  - riemannian-geometry
  - riemannian-geodesics
  - jacobi-metric
  - non-transformer
  - attention-free
  - constant-memory-inference
datasets:
  - Skylion007/openwebtext
metrics:
  - perplexity
model-index:
  - name: semsimula-ladder-live-owt-d384-l2-none-norc-sr2-sr3
    results:
      - task:
          type: text-generation
        dataset:
          type: Skylion007/openwebtext
          name: OpenWebText
          split: validation
        metrics:
          - type: perplexity
            value: 63.28
            name: Validation perplexity at block 512, settled (mean of the last three 500-step evals at block 512)
            verified: false
---

# SR2 with its stiff modes critically damped (SR3-crit): 8.5% behind SR2 — the oscillation is the mechanism (Gen 3)

> **The question of this collection:** the conservative core of Semantic Simulation integrates each token as a damped particle, and until now the damping was one constant, γ₀ = 0.1, in every direction. What happens when the stiffest directions of the potential, the modes that oscillate inside a layer step, are damped harder, by a fixed rule or by a rule the model learns token by token? Each arm is SR2 with one change to its damping.
>
> **This model:** [SR2](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2) with **every stiff mode critically damped** (damping rate γ₀ + 2ωᵢ, ratio 1), the velocity between layers kept, no new parameters. **Verdict: 8.5% behind SR2** (63.28 against 58.32), and behind the model with no inertia at all (FO-b, 61.16): what the second-order model gains from on OpenWebText is the **oscillation** of its stiff modes, not inertia as such.
> [the position-dependent-damping collection](https://huggingface.co/collections/dimitarpg13/semantic-simulation-gen-3-position-dependent-damping-6ada61f0754754e2f2deae3c) holds the damping axis; [the Gen 3 collection](https://huggingface.co/collections/dimitarpg13/semantic-simulation-gen-3-cfc-baoab-live-gradients-6abf371ea3c83f127057020c) holds the OpenWebText ladder; [the first-vs-second-order collection](https://huggingface.co/collections/dimitarpg13/semantic-simulation-gen-3-first-vs-second-order-6ad7681596d9a670c268b524) holds the first-order twins.

## Result: the damping axis

| arm | damping of a typical token's stiffest mode | settled perplexity | against SR2 |
| --- | --- | ---: | ---: |
| [SR2](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2) | light: γ₀ = 0.1, total ratio about 0.03 | **58.32** | — |
| [learned ζ(h)](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2-gzh) | moderate: total ratios about 0.59 (layer 0) and 0.39 (layer 1) | **59.29** | +1.7% |
| [FO-b](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2-fob) | the overdamped limit: no velocity in any direction | **61.16** | +4.9% |
| [SR3-crit](https://huggingface.co/dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2-sr3) (this model) | critical: total ratio about 1.03 | **63.28** | +8.5% |

More damping of the stiff modes, worse settled perplexity; SR2's light damping is the best point measured. All four are the same architecture, corpus, schedule and seed.

- **This model: settled 63.28** (mean of the last three evals, 63.15, 63.48, 63.23); best 61.27 at step 31,000. Validation perplexity at block 512.
- Trained on exactly 532,480,000 tokens (32,500 steps × 32 × 512), the Gen 3 ladder's budget and schedule, seed 0.
- Pre-registered (protocol §5.13): SC3′, settles 3% or more behind SR2 toward FO-b, called 25%: HIT, +8.5%, and behind FO-b. SC1′ (within +1% of SR2, 50%) and SC2′ (beats SR2 by 2%, 25%): MISS. SC4, Gate 1 at most SR2's +3.9%, called 60%: HIT (+3.6%). SC5, the per-layer flow converges within 0.15 nats, called 60%: MISS (0.30 nats, drifting).
- Run health: 0 watchdog triggers, 0 spike captures; SCAF leak monitor CLEAN at every audit; largest gradient norm under 2.

## What the damping does here

SR2 integrates, exactly, the 16 stiffest directions of each token's potential as damped oscillators, every one at the base rate γ₀ = 0.1. A mode with frequency ω is then *lightly* damped: its damping ratio, the rate measured against the mode's own frequency, is γ₀ / 2ω, about 0.03 for the stiffest mode of a typical token, so the mode turns through more than a full cycle inside one layer step and keeps about 82% of its amplitude.

**SR3-crit sets each stiff mode's rate to γ₀ + 2ωᵢ**, the critical rate on top of the base. At that rate a mode settles toward its well as fast as it can without overshooting; the oscillation is gone, the velocity between layers is kept, and every direction outside the stiff modes keeps γ₀. The total ratio is about 1.03 on the stiffest mode of a typical token (1 added, 0.026 from the base). The rate goes into the same exact damped flow SR2 uses (both A sub-steps), the complement keeps its friction in the O-step, and no parameter is added. The switch was verified before the run: off, bit-identical to SR2; on, the exact flow reproduces the closed-form critically damped response to 1e-16; the step changes only on the stiff modes.

## How the gap formed

| step | phase | SR2 | this model | against SR2 |
| ---: | --- | ---: | ---: | ---: |
| 500 | warmup | 461.81 | 461.20 | −0.1% |
| 5,000 | constant | 108.37 | 111.16 | +2.6% |
| 10,000 | constant | 85.96 | 95.46 | +11.1% |
| 15,000 | constant | 76.83 | 83.54 | +8.7% |
| 20,000 | constant | 70.82 | 77.71 | +9.7% |
| 25,000 | decay | 65.92 | 72.21 | +9.5% |
| 30,000 | decay | 58.07 | 63.38 | +9.1% |
| 32,500 | decay, end | 58.32 | 63.23 | +8.4% |

- **Level to step 2,500, behind from step 5,000,** and the gap held at about +0.09 nats through the constant phase and the decay.
- **The decay did not close it:** from step 21,000 to the end SR2 gained 0.196 nats, this model 0.216.
- **In flight the wells stiffened to ω·Δt 3.1 by step 5,000, then softened to about 2** for the rest of training, between the gradient flow's 0.74 and SR2's 7.55.

## Where the model differs from SR2

| reading | SR2 | this model |
| --- | ---: | ---: |
| ω·Δt of a typical token's stiffest mode at the endpoint (median over tokens, Cell 6b-13) | **7.55** | **2.06** |
| the same, layer 0 / layer 1 | 8.93 / — | 1.69 / 5.46 |
| share of those readings past the explicit-integrator wall (2.0) | 99.7% | 52.0% |
| context channels' decay rates at the end of training | 0.386, 0.521, 0.887, 0.981, 0.995 | 0.282, 0.469, 0.877, 0.982, 0.991 |
| share of the layer step carried by the kick, layer 0 / layer 1 | 0.37 / 0.51 | 0.58 / 0.96 |
| velocity reset at inference (Gate 1) | +3.9% | +3.6% |
| one extra layer at the trained Δt (Gate 2) | +35% | +132% |
| eight layers (Gate 2) | +567% | +3,449% |
| refinement at three steps, Gate 3 (nats) | 0.405 | 0.447 |
| distance of the step from the V_θ geodesic, R(geo) (Cell 6b-9) | 0.62 | 0.85 |
| V_φ's share of that distance | -0.002 | -0.300 |

Critical damping removes the rotation of the stiff modes and with it most of what makes SR2 extensible and refinable: one extra layer costs +132% (SR2 +35%), eight layers +3,449%, and the pair force V_φ takes over part of the step (its share of the distance from the geodesic −0.30, against SR2's −0.002). The two layers split, soft wells at layer 0 and stiff ones at layer 1.

## The per-layer flow

Refining each trained layer's share of the integration time into N/2 substeps with the quadratic frozen at the layer's input (`debug/refinement_flow_confirmation.py`, batch seed 20261009), against the trained step repeated at the finer Δt:

| N | 3 | 4 | 6 | 8 | 12 | 16 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| per-layer flow, perplexity | 120.56 | 69.43 | 78.16 | 78.09 | 80.30 | 81.11 |
| penalty over N = 2 (nats) | 0.697 | 0.145 | 0.263 | 0.262 | 0.290 | 0.300 |
| the trained step repeated | 400.73 at N = 8 (1.898 nats) | | | | | |

The flow drifts slowly upward toward a limit about 0.30 nats above the trained step, against SR2's converged 0.104; the pre-registered convergence call (within 0.15 nats) missed.

**Across the damping axis** (per-layer flow at N = 16, nats above the trained step): SR2 0.104, converged; learned ζ(h) 0.312, levelling off; SR3-crit 0.300, drifting; FO-b 0.904, not converging. Only the lightly damped model's trained step closely samples a convergent flow.

## Reproducing

`colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb`, Cell 0 changed from the defaults:

```python
REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,
STIFF_MODE_DAMPING_RATIO=1.0,             # tag 'sr3': gamma_i = gamma_0 + 2 * omega_i on the stiff modes
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False   # speed switches, same arithmetic
```

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7, 6b-13 and 6b-9. Cell 5b prints a damping block (`SR3-crit`). The switch verification is `debug/verify_sr3_switch.py`; the per-layer flow is `debug/refinement_flow_confirmation.py` (all CPU).

Training artefacts were written to `semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_sr3_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/`.

## Glossary

| term | meaning |
| --- | --- |
| settled PPL | validation perplexity at block 512, the mean of the last three evals (steps 31,500–32,500). Best is the lowest single eval |
| Gen 1 / Gen 2 / Gen 3 | the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) *detached*, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients *live* |
| live / detached gradients | whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path |
| Vθ | the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token |
| Vφ | a pair potential between a token and a few selected earlier tokens (sparse top-k) |
| ξ | the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ |
| Fock mechanism, registers, reverse channel | a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part |
| exchange field | an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, `attention` and `attention_potential` |
| `attention` (non-conservative exchange field) | the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step |
| `attention_potential` (attention as a potential) | the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative |
| no-exchange (`LADDER_MECHANISM = 'none'`) | the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only |
| live exchange field (`RELAX_GRAD_PATH = 'live'`) | `attention_potential` with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state |
| Gen 2 probe | a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (`attention_potential` with `RELAX_GRAD_PATH = 'live'`, the full 32,500 steps) is the one that found the gradient starvation |
| gpt2-matched | the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3 |
| tag components (`vplive`, `xilive`, `rglive`, `norc`, `L4probe`, `idt…`) | pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt |
| conservative-only | a model in which every force is a gradient: no reverse channel and no non-conservative exchange field |
| CfC/BAOAB low-rank propagator | the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form |
| ω·Δt, θ | a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the *wall*); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π |
| geodesic step, R(geo), share | the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's *share* is how much it moves that distance (6b-9 attribution) |
| Gates 1–3, FLOW / MAPS | see **Terms** in the geometry section |
| CG1–CG7, 6b-N | the CfC-geometry experiments, and the ladder-notebook cells that run them |
| SCAF, Tier A, Tier B | the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model |
| leak tax, prefix-only | the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model |
| clip-hit | the share of logged steps whose gradient norm exceeded the clipping threshold (1.0) |
| pre-registered | a prediction recorded, with its threshold, before the run |

## Files

| path | what |
| --- | --- |
| `README.md` | this card |
| `config.json` | architecture, training, the damping switch, result and diagnostics |
| `checkpoints/fock_cfc_*noattn_best.pt` | weights at the best eval (step 31,000), optimizer state stripped |
| `results/` | the training log and full run outputs, Cells 6b-7, 6b-13 and 6b-9, the per-layer flow |
| `code/` | the notebook and the model code the switches belong to |

## Companion notes

| note | what it covers |
| --- | --- |
| [`Tech_Report_Position_Dependent_Damping_CfC_BAOAB.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Tech_Report_Position_Dependent_Damping_CfC_BAOAB.md) | the technical report: the theory of position-dependent damping on the exact integrator, the parameterisations, the implementation with code excerpts, the results and the open programme |
| [`Depth_Ladder_and_Matched_Baseline_Protocol.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Depth_Ladder_and_Matched_Baseline_Protocol.md) | §5.13: the damping arms' pre-registrations, amendments and scored results |
| [`Position_Dependent_Damping_and_Reinforcement_Field.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Position_Dependent_Damping_and_Reinforcement_Field.md) | the earlier note (Gen 1), kept as the record the pre-registrations cite |

## Caveats

- **One seed per model.** The seed spread, the yardstick for a gap under 3%, has not been measured.
- **L = 2, d = 384, 0.53 billion tokens, OpenWebText.** Everything here is at one scale on one corpus; in three of four comparisons the second-order arm gained more in the learning-rate decay than its comparison, so a longer run could move the gaps.

## Citation

```bibtex
@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}
```

*Card generated from `_ladder/ladder.json` on 2026-11-08.*