AGILLM-128B-bin8 — exact 1-bit sign packs of a 128B binary-weight MoE in training

Periodic snapshots of the AGILLM-128B run (80 blocks, d=8192, ff=32768, 2–3 experts per block, 126.7 billion binary weights with int8 latents, 21.7 M floating-point parameters, tiktoken o200k_base, context 2048). The model is trained with binary weights: the forward pass reads only the sign of every int8 latent (w = +1 iff latent >= 0), so a pack of those signs is inference-exact and 8× smaller than the native checkpoint (16.1 GB instead of 128.7 GB).

Each packs/step<N>/ directory is one snapshot in format agillm128b-bits-1:

file	contents
model_bits.bin	body records in catalog order: packed sign bits (row-major, little bit order, bit 1 = +1) followed by per-row int64 populations
table_bits.bin	the binary LM table record (200064 × 8192), same layout
meta.json	index (offset/rows/width per matrix), per-record SHA-256 and populations, part SHA-256s, config, pinned source hash, native checksums
ckpt.pt, state.json	the small fp state (scales, norms, routers, local heads, Adam) and the full training state at that step
trainer_source.py	the single-file trainer / inference / pack program that produced the snapshot
SHA256SUMS, EXPORT_COMPLETE.json, README.md	checksums and the export receipt (what was verified)

Files are uploaded whole, never split.

Use

cd packs/step000004566
python3 trainer_source.py pack128 verify --pack .                          # populations, record and part hashes
python3 trainer_source.py pack128 expand --pack . --out /data/agillm128b_step4566   # native dir, ~128.7 GB, ±32 latents
python3 trainer_source.py infer --run /data/agillm128b_step4566 --verify-only
python3 trainer_source.py infer --run /data/agillm128b_step4566 --tokenizer-dir <dir with o200k_base.tiktoken + tokenizer_config.json> --prompt "..." --max-new 64

Expansion is sign-exact for inference. Resuming training from an expansion is a warm restart (the pack does not carry the latent magnitudes).

Snapshots

step	date (UTC)	notes
4566	2026-10-24	first export; training on a pinned multi-dataset mixture (fineweb-edu, GitHub code, FineMath, Wikipedia, Project Gutenberg, arXiv, FineWeb-2 ru/zh/ko/de/fr/es)

Training runs on a single consumer GPU, one 2048-token window per step; snapshots are infrequent by design.