ryze-ai/flash-archer-150M-1.0

Flash-Archer-150M

Flash-Archer-150M is a compact Llama-compatible causal language model trained from scratch on roughly 975M tokens from FineWeb-Edu.

Why This Model Exists

Flash-Archer-150M is intended as a small, easy-to-load baseline for experimentation with custom pretraining, tokenizer behavior, low-cost inference, and continued fine-tuning. It is not instruction-tuned and should be treated as a base language model rather than a chat assistant.

The main points of interest are:

• 150M parameter Llama-style decoder-only architecture.
• Custom 16k byte-level BPE tokenizer.
• Standard Hugging Face Transformers format with safetensors.
• Small enough to run on CPU for testing, with GPU recommended for faster generation or fine-tuning.

Model Details

Field	Value
Architecture	Llama-compatible decoder-only causal LM
Transformers class	LlamaForCausalLM
Parameters	150,335,232
Layers	18
Hidden size	768
Attention heads	12
Key/value heads	12
MLP intermediate size	2,304
Activation	SiLU
Positional embeddings	RoPE, theta 10,000
Normalization	RMSNorm, epsilon 1e-5
Vocabulary size	16,000
Tokenizer	Custom byte-level BPE, PreTrainedTokenizerFast
Special tokens	<pad>, <unk>, <bos>, <eos>
Context length	1,024 tokens
Checkpoint format	Transformers + model.safetensors
Weight dtype in uploaded checkpoint	float32
License	Apache-2.0 for model weights

Training Data

Flash-Archer-150M was trained on:

• Dataset: HuggingFaceFW/fineweb-edu
• Config: sample-10BT
• Dataset license: ODC-BY
• Documents consumed: 877,735
• Tokens seen: 974,520,320
• Validation holdout: 2,000 documents
• Tokenizer training documents: 200,000

FineWeb-Edu is an English educational web dataset derived from Common Crawl. The model card attributes the dataset here because the training data is released under ODC-BY.

Training Methodology

The model was trained from scratch as a next-token prediction causal language model.

Hyperparameter	Value
Optimizer	AdamW
Peak learning rate	3e-4
Final learning rate	3e-5
Schedule	Cosine decay with 2% warmup
Beta1 / Beta2	0.9 / 0.95
Adam epsilon	1e-8
Weight decay	0.1
Gradient clipping	1.0
Micro batch size	1
Gradient accumulation	128
Training precision	FP16 mixed precision
Max sequence length	1,024
Final checkpoint step	7,435

Validation

These are validation-log results from the training run, not standardized benchmark results. The run used a small validation setting (val_batches=2), so the numbers should be interpreted as rough training diagnostics.

Checkpoint	Step	Tokens seen	Validation loss	Perplexity
Best logged validation	4,200	550,502,400	3.0669	21.48
Final logged validation	7,400	969,932,800	3.3003	27.12

No public benchmark scores are claimed.

Hardware

Training metadata indicates the run was performed on Kaggle with one NVIDIA Tesla T4 GPU with about 15.6 GB VRAM. The configured maximum training time was 7 hours.

Intended Uses

This model is best suited for:

• Educational experiments with small pretrained language models.
• Continued pretraining or supervised fine-tuning.
• Tokenizer and architecture experiments.
• Lightweight local generation tests.
• Baseline comparisons against other small causal LMs.

Limitations

• This is a base model, not an instruction-following chat model.
• It may repeat itself, produce low-quality generations, or fail to follow prompts.
• It has not been evaluated on safety, factuality, coding, reasoning, or domain benchmarks.
• It was trained on web text and may reproduce biases, errors, or unsafe content present in that data.
• The context length is limited to 1,024 tokens.
• Validation metrics are noisy because only a small number of validation batches were logged.

Inference

Install dependencies:

pip install torch transformers safetensors

Load the model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Norman89107/Flash-Archer-150M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()

prompt = "In simple terms, machine learning is"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

For GPU inference:

model = AutoModelForCausalLM.from_pretrained(
    "Norman89107/Flash-Archer-150M",
    torch_dtype=torch.float16,
    device_map="auto",
)

Files

• config.json: Llama-compatible model configuration.
• model.safetensors: Model weights.
• tokenizer.json: Custom byte-level BPE tokenizer.
• tokenizer_config.json: Tokenizer metadata and max length.
• special_tokens_map.json: Special token definitions.
• generation_config.json: Default sampling settings.
• inference.py: Minimal inference script.

Source Code

No public source repository link was provided for this release. The uploaded checkpoint is in standard Transformers format and can be loaded directly with AutoModelForCausalLM.

License

The model weights are released under Apache-2.0. The training data used FineWeb-Edu, which is released under ODC-BY; see the FineWeb-Edu dataset card for dataset terms and attribution details.