LLM training from scratch: 90M → 2.7B on one GPU
Collection
Data cleaning, custom tokenizer, 167-run speedrun, scaling probes, a 2.67B pretrained on one B200 in 38h, then SFT and reasoning. ~300 W&B runs. • 24 items • Updated
crisp3b-v7 — 2.67B decoder-only LM, GPT-2 tokenizer
Base language model. Same architecture and training recipe as crisp3b-v8 (34 layers, d_model 2560, GQA 40/20, FlexAttention sliding window + global layers, ReLU², value residual, Muon + AdamW, FP8 forward), with the GPT-2 BPE tokenizer (vocab 50,257, padded to 50,304) instead of the v8 tokenizer. Trained on Crisp-165B with loader-side source reweighting (target mix: FineWeb-Edu 32%, DCLM 20%, code 13%, FineMath 9%, Dolmino 8%, peS2o 8%, Cosmopedia 4%, PG19 3%, Wikipedia 3%).
Part of the collection LLM training from scratch: 90M → 2.7B on one GPU.
Runs
| Run | W&B | Steps | Notes |
|---|---|---|---|
| Stable phase | …_0955 |
43,500 | Constant LR after 50-step warmup. Stopped when validation loss plateaued (best 2.535 at step 37k; band 2.54–2.60 for ≈6k steps). ≈2.14B tokens (43,500 steps × 49,152), ≈12h. |
| Warmdown | …_9225 |
13,561 | Resumed model + optimizer from step 40,000 (val 2.555). Linear LR decay to 0 over ≈4h. Final val loss 2.374. ≈0.67B tokens (13,561 steps × 49,152). |
Hardware: 1× NVIDIA B200, Vast.ai (≈$3.15/h). Batch 24 × 2048, ≈50k tokens/s. ≈2.8B tokens and ≈16 GPU-hours (≈$50) across both runs; no token was seen twice.
Evaluation (final warmdown checkpoint)
| Benchmark | Score |
|---|---|
| ARC-easy | 55.8 |
| SciQ | 66.4 |
| PIQA | ≈66 |
| HellaSwag (acc_norm) | ≈37 |
| LAMBADA | ≈34 |
| MMLU | 25.4 |
Validation loss 2.374 (GPT-2 tokenizer units).
Downstream checkpoints
| Stage | Repo | W&B | Data |
|---|---|---|---|
| SFT (ChatML, 3 epochs, batch 16, doc-masked packing) | crisp3b-sft-ckpts | crisp3b_sft_v1 |
crisp-sft-chat — 329,722 conversations, 218M tokens |
Reasoning cold-start (<think>…</think>) |
crisp3b-reason-ckpts | v1, v2 |
crisp-reason-cold — 34,821 short verified math/science traces + 5k no-think |
| GRPO on arithmetic | — | crisp3b-grpo (4 runs, all crashed) | — |
Chat special tokens (<|im_start|>, <|im_end|>, tool-call and think tags) occupy ids 50257+ in the padded vocab; see chat_format.py in the training repo.
Files
| File | Contents |
|---|---|
2669M_..._9225_final_step13561.pt |
Final warmdown checkpoint: model + optimizer states + step |
2669M_..._8830_final_step651.pt |
Earlier short warmdown test |
w_final_weights.pt |
Weights only, bf16, ≈5.3 GB; input of the SFT stage and the MLX playground |
Loading is the same as for crisp3b-v8, with tiktoken.get_encoding("gpt2") as tokenizer and logits for ids ≥ 50257 masked at inference.
Limitations
English only. Base model, not instruction-tuned, not aligned or filtered. ≈3 tokens/parameter; MMLU at chance.