dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2
Text Generation • Updated • 596
Semantic Simulation — Gen 3: first vs second order
The Gen 3 conservative core trained second and first order: second order wins without wells (+2.3% to +9.2% over three corpora), not with them.
Note OpenWebText, 532.5M tokens. SECOND ORDER: SR2, the conservative core with its stiff modes integrated as exact damped oscillators. Settled 58.32. Its layer step is a close sample of a convergent per-layer flow (0.104 nats). Pair partner: FO-b-OWT, the same model trained as a true first-order gradient flow, 4.9% worse.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2-fob
Text Generation • Updated • 633Note OpenWebText, 532.5M tokens. FIRST ORDER: the same model trained as a true gradient flow (FO-b): no velocity anywhere, stiff modes relaxed exactly. Settled 61.16, 4.9% behind SR2. Led until step 11,000, then fell behind; builds wells ten times softer (omega*dt 0.74 against 7.55) with the same context horizons. Extra layers cost almost nothing (+0.6%), but the per-layer flow does not converge.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm64-pmx16-sr2
Text Generation • Updated • 659Note OpenWebText, 532.5M tokens, with 64 Poisson-mode wells integrated exactly. SECOND ORDER: PMX, settled 54.63, the 64-well point of the dose series (K = 0, 2, 16, 64); 16 wells do as well (54.24). Resetting its velocity at inference costs +91%. Pair partner: FO-b-PMX, the same model trained as a first-order gradient flow, ahead of it (54.02, −1.1%): with the wells present, second order earns nothing measurable.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm64-pmx16-sr2-fob
Text Generation • Updated • 644Note OpenWebText, 532.5M tokens, with 64 Poisson-mode memory wells. FIRST ORDER: PMX trained as a true gradient flow (FO-b): the wells and the stiff modes relaxed exactly, no velocity anywhere. Settled 54.02 against PMX's 54.63 (−1.1%, a tie): the first-order twin that led without wells by nothing and trailed by 4.9% is level once the wells remember. Its wells are attractive by a majority where PMX's are balanced; one extra layer costs +0.3% (PMX +956%). The memory-substitution hypothesis follows.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2
Text Generation • Updated • 628Note OpenWebText, 532.5M tokens, 16 Poisson-mode wells integrated exactly. SECOND ORDER: PMX-K16, settled 54.24, level with PMX's 64 wells (54.63). Leans on its velocity more than any model (resetting it costs +5,788%), yet its first-order twin is ahead by 0.9%, measured paired. Reliance is not need.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2-fob
Text Generation • Updated • 637Note OpenWebText, 532.5M tokens, 16 Poisson-mode wells. FIRST ORDER: PMX-K16 trained as a true gradient flow (FO-b), no velocity anywhere. Settled 53.73, ahead of its second-order twin by 0.9% (paired by step, spread 0.14%). In its place the model leans more on token-token coupling and a stiffer second layer. Sixteen wells do as well as sixty-four in both orders.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm2-pmx16-sr2
Text Generation • Updated • 622Note OpenWebText, 532.5M tokens, 2 Poisson-mode wells integrated exactly. SECOND ORDER: PM2-PMX16, settled 54.71: two wells give it most of what sixteen or sixty-four do (none: 58.32). Resetting its velocity costs +5,150%, yet its first-order twin is ahead by 1.6%, measured paired. Its per-layer flow converges.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm2-pmx16-sr2-fob
Text Generation • Updated • 649Note OpenWebText, 532.5M tokens, 2 Poisson-mode wells. FIRST ORDER: PM2-PMX16 trained as a true gradient flow (FO-b), no velocity anywhere. Settled 53.81, ahead of its second-order twin by 1.6% (paired, spread 0.14%) and level with the best conservative model (53.73) with an eighth of its wells. It leans on the two wells eight times as hard. Its per-layer flow does not converge.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2
Text Generation • Updated • 733Note TinyStories, 315M tokens (16,250 steps plus a 3,000-step extension at the floor). SECOND ORDER: SR2's architecture and settings on TinyStories. Settled 5.16. Its per-layer flow converges 0.097 nats above the trained step, as SR2's does on OpenWebText (0.104): the convergent flow belongs to the integrator, not the corpus. FO-TS (velocity reset between layers) is level (−0.6%); FO-b-TS (true first order) trails it by 2.3%.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2-fo
Text Generation • Updated • 702Note TinyStories, 315M tokens. FIRST ORDER BETWEEN LAYERS (FO-a): the velocity is reset where each layer begins; inside a layer the step is unchanged. Settled 5.13, level with SO-TS (-0.6%): on the narrow corpus the velocity between layers is not needed. One extra layer costs +1.1% (SO-TS +25.5%). It keeps the stiff modes' rotation within each layer; the true first-order twin FO-b-TS, which removes it, trails by 2.3%.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2-fob
Text Generation • Updated • 614Note TinyStories, 315M tokens, no wells. TRUE FIRST ORDER (FO-b): SR2 on TinyStories as a gradient flow, no velocity anywhere. Settled 5.28, 2.3% behind SO-TS: second order wins without wells on TinyStories too (OpenWebText +4.9%). FO-a, which keeps the stiff modes' rotation within each layer, ties; removing the rotation costs the difference. With two wells the gap closes.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-pm2-pmx16-sr2
Text Generation • Updated • 892Note TinyStories, 315M tokens, 2 Poisson-mode wells integrated exactly. SECOND ORDER: PM2-TS, settled 4.99, 3.4% below SO-TS without wells: the wells help on TinyStories too, about half as much as on OpenWebText. They learn the same landscape as there. Resetting its velocity costs +3,907% (SO-TS +2.0%), yet its first-order twin is level.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-pm2-pmx16-sr2-fob
Text Generation • Updated • 908Note TinyStories, 315M tokens, 2 Poisson-mode wells. FIRST ORDER: PM2-TS trained as a true gradient flow (FO-b), no velocity anywhere. Settled 4.96, level with its second-order twin (−0.5%, paired, spread 0.15%), as with two wells on OpenWebText: first order leads in the stable phase, second order narrows the gap in the decay. Its wells are deeper at layer 0, shallower at layer 1.
dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-sr2
Text Generation • Updated • 1.15kNote Python code, 315M tokens, no wells. SECOND ORDER (SO-CODE): SR2 on the third corpus. Settled 6.00; its true first-order twin trails by 9.2%, the largest gap of the three corpora (TinyStories 2.3%, OpenWebText 4.9%). The matched GPT-2 on code: 4.96 (+21.0% for this model).
dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-sr2-fob
Text Generation • Updated • 345Note Python code, 315M tokens, no wells. TRUE FIRST ORDER (FO-b-CODE): SR2 on code as a gradient flow. Settled 6.55, 9.2% behind SO-CODE, the largest gap of the three corpora. The loss sits in identifiers and grows with the distance to their earlier occurrence; the model finds those occurrences as well as its twin and uses them less well.
dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-pm2-pmx16-sr2
Text Generation • Updated • 11Note Python code, 315M tokens, 2 Poisson-mode wells, second order. Settled 5.71: the wells help by 4.7% against SO-CODE (lower at all 38 evaluations), 15.1% behind the matched GPT-2. With the wells its first-order twin leads by 1.6%; resetting the velocity costs +6,300%, so the model leans on it, but first order does better without one.
dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-pm2-pmx16-sr2-fob
Text Generation • Updated • 15Note Python code, 315M tokens, 2 Poisson-mode wells. FIRST ORDER: PM2-CODE trained as a true gradient flow, no velocity anywhere. Settled 5.62, 1.6% ahead of its second-order twin (paired −1.55%, spread 0.14%) and 14.2% ahead of FO-b-CODE without wells: on code the wells turn first order's largest gap (9.2%) into a lead. 13.3% behind the matched GPT-2.
dimitarpg13/semsimula-ladder-ts-d384-gpt2-matched
Text Generation • Updated • 542Note REFERENCE for the TinyStories models. Matched GPT-2 (d = 384, L = 8, tied, 33.7M) on the TinyStories pair's tokens and schedule (315M tokens: 16,250 steps on the ladder's WSD schedule, then 3,000 at its floor): settled 4.47. SO-TS (5.16) is 15.4% behind it, FO-TS (5.13) 14.8%.