dimitarpg13/semsimula-ladder-owt-d384-gpt2-matched-2b
Text Generation • Updated • 418
Semantic Simulation — Gen 3: 2B training tokens
Gen 3 models trained on 2.13B tokens (4x the Gen 3 ladder), with a matched GPT-2 at the same budget as reference.
Note SHARED REFERENCE for the Gen 3 2B-token models. Matched GPT-2 (d=384, L=8, tied, 33.7M) on the Fock ladder's schedule exactly (WSD, peak 1.2e-3), 2.13B tokens: 42.69 settled, 11.7% below the same model at 0.53B tokens (48.37). Matched tokenizer, corpus, batches, width and token budget; depth NOT matched (L=8 against the ladder's L=2). Single seed.