Alpha yi base knowledge model

Alpha yi is a 63.2M-parameter, decoder-only research base model trained from scratch with the Alpha/Helios stack. This repository freezes the terminal pretraining checkpoint at optimizer step 1,180,000, before chat fine-tuning.

This is an exact Alpha native checkpoint, not a standard Transformers export. It includes model weights, AdamW state, RNG state, model configuration, and the embedded 12,288-token byte-BPE tokenizer. The native format is retained because this GPT-form Alpha architecture uses runtime details that an ordinary GPT-2 conversion would not reproduce exactly.

Model

Architecture: GPT-form causal decoder
Parameters: 63,210,496
Layers / width / heads: 16 / 512 / 8
Context: 512 tokens
Vocabulary: 12,288 byte-BPE tokens
Position encoding: learned absolute positions
Normalization / activation: LayerNorm / GELU
Embeddings: untied input and output embeddings
Training stack: Alpha native CUDA backend on an NVIDIA RTX 3070

Pretraining

The model was trained on concordance-v10.txt, a frozen 60,418,346-token knowledge-text corpus. The run performed 1,180,000 optimizer steps with batch 1 and context 512: 604,160,000 token positions, approximately ten corpus passes and 9.56 training tokens per parameter.

AdamW, learning rate 6e-4 to 6e-5
2,000 warmup steps
Weight decay 0.1; gradient clipping at 1.0
Terminal training loss: 0.5848 (training batch, not validation loss)
Source tree commit: d130af1fde31c343f4618655b4103dcdb4f5178a

Checkpoint SHA-256:

6b46769eae17a48378ebc58472242d3a3bbd62d04130ba1e80b70b9da734bc05  alpha-yi-base-step-1180000.alph

Use with Alpha

With a compatible Alpha checkout and built CLI:

hf download ajaxdavis/alpha-yi-base-knowledge \
  alpha-yi-base-step-1180000.alph --local-dir .

node apps/cli/dist/main.js sample \
  --checkpoint=alpha-yi-base-step-1180000.alph \
  --prompt="The capital of France is" \
  --steps=40 --temp=0.7 --topk=40

Evaluation status and limitations

The checkpoint was hash-verified and independently reloaded in a fresh process; tokenizer restoration and text generation both work. A clean validation metric is not reported: the native full-forward evaluation path exhausted the RTX 3070 allocator after training, and Vulkan was unavailable in this pod. The terminal training loss must not be interpreted as held-out perplexity.

Raw generations are often incoherent and sometimes form inaccurate or malformed claims. This is a small, repeatedly exposed base model, not an instruction model or a reliable factual system. It should not be used for high-stakes decisions. Chat fine-tuning is tracked separately so this repository remains an immutable base-model artifact.

Downloads last month: -; Downloads are not tracked for this model. How to track