Fleck-M-500K

Fleck-M-500K Instruct — the independently instruction-tuned child of Fleck-M-500K-Base.

• 497,288 parameters
• BF16 SafeTensors weights
• Fine-tuned from the Base-100M model on Dolly-15k
• Intended for local conversational experiments on Apple Silicon

Model Details

Architecture	Decoder-only Transformer
Parameters	497,288
Hidden size	128
FFN size	336
Physical blocks	2
Effective depth	4 (A → B → A → B)
Attention	GQA — 4 query heads, 2 KV heads, head dimension 32
Normalization	RMSNorm
Embedding	Factorized tied embedding, rank 64
Vocabulary	2,048
Context length	2,048 tokens
Canonical dtype	BF16
Base model	ML-is-Fun/Fleck-M-500K-Base

Training

Pretraining

The Base parent was initialized from scratch and pretrained for exactly 100,000,000 real tokenizer tokens on a 70/30 FineWeb-Edu/FineWeb mixture.

Instruction Tuning

Base model	Fleck-M-500K-Base
Dataset	databricks/databricks-dolly-15k
Dataset license	CC BY-SA 3.0
Requested budget	250K supervised tokens
Realized supervised tokens	216,070
Optimizer	AdamW
Instruction tuning	Assistant-response supervision
Hardware	Apple M2 (10-core GPU)

Benchmark Results

Corrected zero-shot evaluation, no chat template, FP32 evaluation, and the same benchmark aggregation protocol were used for both variants.

Task	Metric	Shots	Base	Instruct	Δ
HellaSwag	acc_norm	0	25.04%	24.67%	-0.38pp
PIQA	acc_norm	0	51.31%	50.98%	-0.33pp
ARC-Easy	acc_norm	0	26.05%	26.09%	+0.04pp
ARC-Challenge	acc_norm	0	24.15%	24.23%	+0.09pp
LAMBADA OpenAI	acc	0	0.04%	0.08%	+0.04pp
WinoGrande	acc	0	51.30%	50.99%	-0.32pp
BoolQ	acc	0	37.83%	37.83%	0.00pp
MMLU (57-subject macro)	acc	0	23.14%	23.20%	+0.06pp
Eight-task mean	—	0	29.86%	29.76%	-0.10pp

Tokenizer and Chat Format

• Fleck-Tokenizer-2048
• Byte-level BPE
• Vocabulary size: 2,048

The training chat format is:

<bos><|user|>{user}<|eot|><|assistant|>{response}<|eot|><eos>

Special Tokens

Token	ID	Role
<bos>	0	sequence start
<eos>	1	sequence end
<pad>	2	padding
<unk>	3	unknown token
<|system|>	4	system turn
<|user|>	5	user turn
<|assistant|>	6	assistant turn
<|eot|>	7	end of turn

Usage

The bundle includes a self-contained inference.py; it does not import the Fleck-LM checkout. The accompanying config.json, generation_config.json, and tokenizer_config.json describe the custom architecture and generation/tokenizer defaults; standard transformers.AutoModel loading is not supported. The chat_template.jinja file contains the Instruct chat template used by compatible HF tooling. Install the three runtime dependencies:

python -m pip install torch safetensors tokenizers

By default the CLI starts an interactive chat. /exit quits and /clear resets the conversation history:

python inference.py --device cpu --max-tokens 32

For one prompt without interactive mode, pass --no-chat and --prompt:

python inference.py \
  --ckpt model.safetensors \
  --tokenizer tokenizer.json \
  --no-chat \
  --prompt "Explain what a tokenizer does." \
  --max-tokens 32 \
  --device cpu

The script reads and runs the BF16 checkpoint without an FP32 model copy, validates every SafeTensors key, shape, and dtype, and uses FP32 only for attention score/softmax and tied-logit accumulation. It reproduces the factorized tied embedding/logits, effective-depth execution A → B → A → B, half-split RoPE, GQA, physical KV caches, RMSNorms, and greedy generation without repository-local imports. Generation stops on <|eot|> or <eos>.

Limitations

This model is extremely small and is intended for research and local experimentation rather than reliable general-purpose assistance. It may produce repetitions, malformed text, weak factual answers, or incoherent responses. Instruction tuning improves conversational behavior but does not overcome the limits of a 497,288 parameter model.

License

MIT License.

Files

The public bundle contains these files:

• README.md — model card and usage documentation
• inference.py — standalone strict loader and interactive/single-prompt inference CLI
• model.safetensors — BF16 model weights
• tokenizer.json — standalone tokenizer
• config.json — custom architecture configuration
• generation_config.json — greedy generation defaults
• tokenizer_config.json — tokenizer defaults and special-token mapping
• chat_template.jinja — Instruct chat template No training data, optimizer state, or other training outputs are included.