# Tiny Decoders

Tiny decoder experiments for instruction-conditioned Python generation.

[Paper](https://chameleon-lizard.ru/tiny_decoder_paper.pdf) · [Code](https://github.com/chameleon-lizard/tiny_decoder)

Each model branch contains its resolved `config.yaml`, checkpoint, manifest, and tokenizer files.

## V1 architecture sweep

| Branch | Experiment |
|---|---|
| [`ettin17m_default`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/ettin17m_default) | Reference: Ettin-17M encoder, 64d × 2-layer decoder, prefix length 4 |
| [`decoder_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/decoder_only) | Reference decoder with the encoder removed |
| [`prefix_k1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/prefix_k1) | Reference with prefix length 1 |
| [`prefix_k8`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/prefix_k8) | Reference with prefix length 8 |
| [`depth_1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/depth_1) | Reference with one decoder layer |
| [`depth_4`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/depth_4) | Reference with four decoder layers |
| [`wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/wider_d128) | Wider 128d × 2-layer decoder |
| [`ettin32m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/ettin32m) | Reference with Ettin-32M encoder |
| [`k1_d128_L4_ettin17m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/k1_d128_L4_ettin17m) | Prefix 1, 128d × 4 layers, Ettin-17M encoder |
| [`combined_k8_d128_L4_ettin32m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/combined_k8_d128_L4_ettin32m) | Prefix 8, 128d × 4 layers, Ettin-32M encoder |

## V2 core models

| Branch | Experiment |
|---|---|
| [`v2_base_pretrain`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_base_pretrain) | Decoder-LM warm-start for the V2 base shape |
| [`v2_base_scratch`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_base_scratch) | V2 base architecture trained from scratch |
| [`v2_base_warm`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_base_warm) | V2 base architecture with decoder-LM warm-start |
| [`v2_decoder_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_decoder_only) | V2 decoder with the encoder removed |
| [`v2_pretrain`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_pretrain) | Decoder-LM warm-start for the scaled V2 model |
| [`v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_scaled) | Headline V2: 192d × 6 layers, cross-attention, RoPE, SwiGLU, copy head |

## V1 → V2 progression

| Branch | Experiment |
|---|---|
| [`progression-v1_prefix_558k`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_prefix_558k) | 558k matched-budget V1 prefix coupling |
| [`progression-v2_xattn_558k`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v2_xattn_558k) | 558k matched-budget cross-attention |
| [`progression-v1_1p2m_prefix_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_1p2m_prefix_only) | 1.2M prefix-only baseline |
| [`progression-v1_5_xattn`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn) | Adds cross-attention |
| [`progression-v1_5_xattn_rope`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope) | Adds RoPE |
| [`progression-v1_5_xattn_rope_swiglu`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope_swiglu) | Adds SwiGLU |
| [`progression-v1_5_xattn_rope_swiglu_copyhead`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope_swiglu_copyhead) | Adds the copy head |
| [`progression-v1_5_xattn_rope_swiglu_copyhead_warmstart`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope_swiglu_copyhead_warmstart) | Adds decoder-LM warm-start |
| [`progression-v2_base_full`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v2_base_full) | Full V2 base with data interventions |

## V2 leave-one-out ablations

| Branch | Experiment |
|---|---|
| [`loo-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base) | Full V2 base control |
| [`loo-v2_base_abspos`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_abspos) | Learned absolute positions instead of RoPE |
| [`loo-v2_base_gelu`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_gelu) | GELU MLP instead of SwiGLU |
| [`loo-v2_base_maxseq128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_maxseq128) | Maximum sequence length 128 instead of 256 |
| [`loo-v2_base_no_copyhead`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_no_copyhead) | Copy head removed |
| [`loo-v2_base_no_drop_stub_bodies`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_no_drop_stub_bodies) | Stub-body filtering disabled |
| [`loo-v2_base_no_warmstart`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_no_warmstart) | Decoder-LM warm-start removed |
| [`loo-v2_base_uniform_sampling`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_uniform_sampling) | Uniform instead of length-weighted sampling |

## Tokenizer and coupling factorial

| Branch | Experiment |
|---|---|
| [`bpe-bpe_alias`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/bpe-bpe_alias) | Alias-rewritten BPE with prefix coupling at 1.2M |
| [`bpe-bpe_raw`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/bpe-bpe_raw) | Raw-code BPE with prefix coupling at 1.2M |
| [`factorial-bpe_alias_xattn_1p2m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/factorial-bpe_alias_xattn_1p2m) | Alias-rewritten BPE with cross-attention at 1.2M |

## TinyPython dataset generations

| Branch | Dataset | Model |
|---|---|---|
| [`tpy07-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v1_base) | tpy07: initial corpus | V1 base |
| [`tpy07-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v1_decoder) | tpy07: initial corpus | V1 decoder-only |
| [`tpy07-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v1_wider_d128) | tpy07: initial corpus | V1 wider 128d |
| [`tpy07-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v2_base) | tpy07: initial corpus | V2 base |
| [`tpy07-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v2_decoder) | tpy07: initial corpus | V2 decoder-only |
| [`tpy07-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v2_scaled) | tpy07: initial corpus | V2 scaled |
| [`tpy08-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v1_base) | tpy08: name/body-aligned corpus | V1 base |
| [`tpy08-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v1_decoder) | tpy08: name/body-aligned corpus | V1 decoder-only |
| [`tpy08-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v1_wider_d128) | tpy08: name/body-aligned corpus | V1 wider 128d |
| [`tpy08-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v2_base) | tpy08: name/body-aligned corpus | V2 base |
| [`tpy08-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v2_decoder) | tpy08: name/body-aligned corpus | V2 decoder-only |
| [`tpy08-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v2_scaled) | tpy08: name/body-aligned corpus | V2 scaled |
| [`tpy09-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v1_base) | tpy09: semantically aligned corpus | V1 base |
| [`tpy09-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v1_decoder) | tpy09: semantically aligned corpus | V1 decoder-only |
| [`tpy09-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v1_wider_d128) | tpy09: semantically aligned corpus | V1 wider 128d |
| [`tpy09-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v2_base) | tpy09: semantically aligned corpus | V2 base |
| [`tpy09-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v2_decoder) | tpy09: semantically aligned corpus | V2 decoder-only |
| [`tpy09-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v2_scaled) | tpy09: semantically aligned corpus | V2 scaled |
| [`tpy07b-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v1_base) | tpy07b: clean held-out corpus | V1 base |
| [`tpy07b-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v1_decoder) | tpy07b: clean held-out corpus | V1 decoder-only |
| [`tpy07b-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v1_wider_d128) | tpy07b: clean held-out corpus | V1 wider 128d |
| [`tpy07b-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_base) | tpy07b: clean held-out corpus | V2 base |
| [`tpy07b-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_decoder) | tpy07b: clean held-out corpus | V2 decoder-only |
| [`tpy07b-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_scaled) | tpy07b: clean held-out corpus | V2 scaled |
| [`tpy07b-v2_scaled_plus1epoch`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_scaled_plus1epoch) | tpy07b: clean held-out corpus | V2 scaled continued for one epoch |

## V1 clean-holdout reruns

All models below retrain a V1 sweep cell on tpy07b and evaluate it on the frozen V2 benchmark.

| Branch | Experiment |
|---|---|
| [`v1_rerun-decoder_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-decoder_only) | Decoder-only rerun |
| [`v1_rerun-depth_1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-depth_1) | One-layer rerun |
| [`v1_rerun-depth_4`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-depth_4) | Four-layer rerun |
| [`v1_rerun-ettin17m_default`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-ettin17m_default) | Ettin-17M reference rerun |
| [`v1_rerun-prefix_k1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-prefix_k1) | Prefix-length-1 rerun |
| [`v1_rerun-prefix_k8`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-prefix_k8) | Prefix-length-8 rerun |
| [`v1_rerun-wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-wider_d128) | Wider-128d rerun |