Hugging Face's logo

chameleon-lizard
/
tiny_decoders

Model card Files
xet
 Community
chameleon-lizard commited on
Commit
641de9a
·
verified ·
1 Parent(s): 3b83277

Document model experiment branches

Browse files
Files changed (1) hide show
  1. README.md +112 -1
README.md CHANGED
@@ -1 +1,112 @@
1
- TODO
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tiny Decoders
2
+
3
+ Tiny decoder experiments for instruction-conditioned Python generation.
4
+
5
+ [Paper](https://chameleon-lizard.ru/tiny_decoder_paper.pdf) · [Code](https://github.com/chameleon-lizard/tiny_decoder)
6
+
7
+ Each model branch contains its resolved `config.yaml`, checkpoint, manifest, and tokenizer files.
8
+
9
+ ## V1 architecture sweep
10
+
11
+ | Branch | Experiment |
12
+ |---|---|
13
+ | [`ettin17m_default`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/ettin17m_default) | Reference: Ettin-17M encoder, 64d × 2-layer decoder, prefix length 4 |
14
+ | [`decoder_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/decoder_only) | Reference decoder with the encoder removed |
15
+ | [`prefix_k1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/prefix_k1) | Reference with prefix length 1 |
16
+ | [`prefix_k8`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/prefix_k8) | Reference with prefix length 8 |
17
+ | [`depth_1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/depth_1) | Reference with one decoder layer |
18
+ | [`depth_4`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/depth_4) | Reference with four decoder layers |
19
+ | [`wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/wider_d128) | Wider 128d × 2-layer decoder |
20
+ | [`ettin32m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/ettin32m) | Reference with Ettin-32M encoder |
21
+ | [`k1_d128_L4_ettin17m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/k1_d128_L4_ettin17m) | Prefix 1, 128d × 4 layers, Ettin-17M encoder |
22
+ | [`combined_k8_d128_L4_ettin32m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/combined_k8_d128_L4_ettin32m) | Prefix 8, 128d × 4 layers, Ettin-32M encoder |
23
+
24
+ ## V2 core models
25
+
26
+ | Branch | Experiment |
27
+ |---|---|
28
+ | [`v2_base_pretrain`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_base_pretrain) | Decoder-LM warm-start for the V2 base shape |
29
+ | [`v2_base_scratch`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_base_scratch) | V2 base architecture trained from scratch |
30
+ | [`v2_base_warm`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_base_warm) | V2 base architecture with decoder-LM warm-start |
31
+ | [`v2_decoder_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_decoder_only) | V2 decoder with the encoder removed |
32
+ | [`v2_pretrain`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_pretrain) | Decoder-LM warm-start for the scaled V2 model |
33
+ | [`v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v2_scaled) | Headline V2: 192d × 6 layers, cross-attention, RoPE, SwiGLU, copy head |
34
+
35
+ ## V1 → V2 progression
36
+
37
+ | Branch | Experiment |
38
+ |---|---|
39
+ | [`progression-v1_prefix_558k`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_prefix_558k) | 558k matched-budget V1 prefix coupling |
40
+ | [`progression-v2_xattn_558k`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v2_xattn_558k) | 558k matched-budget cross-attention |
41
+ | [`progression-v1_1p2m_prefix_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_1p2m_prefix_only) | 1.2M prefix-only baseline |
42
+ | [`progression-v1_5_xattn`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn) | Adds cross-attention |
43
+ | [`progression-v1_5_xattn_rope`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope) | Adds RoPE |
44
+ | [`progression-v1_5_xattn_rope_swiglu`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope_swiglu) | Adds SwiGLU |
45
+ | [`progression-v1_5_xattn_rope_swiglu_copyhead`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope_swiglu_copyhead) | Adds the copy head |
46
+ | [`progression-v1_5_xattn_rope_swiglu_copyhead_warmstart`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v1_5_xattn_rope_swiglu_copyhead_warmstart) | Adds decoder-LM warm-start |
47
+ | [`progression-v2_base_full`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/progression-v2_base_full) | Full V2 base with data interventions |
48
+
49
+ ## V2 leave-one-out ablations
50
+
51
+ | Branch | Experiment |
52
+ |---|---|
53
+ | [`loo-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base) | Full V2 base control |
54
+ | [`loo-v2_base_abspos`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_abspos) | Learned absolute positions instead of RoPE |
55
+ | [`loo-v2_base_gelu`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_gelu) | GELU MLP instead of SwiGLU |
56
+ | [`loo-v2_base_maxseq128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_maxseq128) | Maximum sequence length 128 instead of 256 |
57
+ | [`loo-v2_base_no_copyhead`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_no_copyhead) | Copy head removed |
58
+ | [`loo-v2_base_no_drop_stub_bodies`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_no_drop_stub_bodies) | Stub-body filtering disabled |
59
+ | [`loo-v2_base_no_warmstart`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_no_warmstart) | Decoder-LM warm-start removed |
60
+ | [`loo-v2_base_uniform_sampling`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/loo-v2_base_uniform_sampling) | Uniform instead of length-weighted sampling |
61
+
62
+ ## Tokenizer and coupling factorial
63
+
64
+ | Branch | Experiment |
65
+ |---|---|
66
+ | [`bpe-bpe_alias`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/bpe-bpe_alias) | Alias-rewritten BPE with prefix coupling at 1.2M |
67
+ | [`bpe-bpe_raw`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/bpe-bpe_raw) | Raw-code BPE with prefix coupling at 1.2M |
68
+ | [`factorial-bpe_alias_xattn_1p2m`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/factorial-bpe_alias_xattn_1p2m) | Alias-rewritten BPE with cross-attention at 1.2M |
69
+
70
+ ## TinyPython dataset generations
71
+
72
+ | Branch | Dataset | Model |
73
+ |---|---|---|
74
+ | [`tpy07-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v1_base) | tpy07: initial corpus | V1 base |
75
+ | [`tpy07-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v1_decoder) | tpy07: initial corpus | V1 decoder-only |
76
+ | [`tpy07-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v1_wider_d128) | tpy07: initial corpus | V1 wider 128d |
77
+ | [`tpy07-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v2_base) | tpy07: initial corpus | V2 base |
78
+ | [`tpy07-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v2_decoder) | tpy07: initial corpus | V2 decoder-only |
79
+ | [`tpy07-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07-v2_scaled) | tpy07: initial corpus | V2 scaled |
80
+ | [`tpy08-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v1_base) | tpy08: name/body-aligned corpus | V1 base |
81
+ | [`tpy08-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v1_decoder) | tpy08: name/body-aligned corpus | V1 decoder-only |
82
+ | [`tpy08-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v1_wider_d128) | tpy08: name/body-aligned corpus | V1 wider 128d |
83
+ | [`tpy08-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v2_base) | tpy08: name/body-aligned corpus | V2 base |
84
+ | [`tpy08-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v2_decoder) | tpy08: name/body-aligned corpus | V2 decoder-only |
85
+ | [`tpy08-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy08-v2_scaled) | tpy08: name/body-aligned corpus | V2 scaled |
86
+ | [`tpy09-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v1_base) | tpy09: semantically aligned corpus | V1 base |
87
+ | [`tpy09-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v1_decoder) | tpy09: semantically aligned corpus | V1 decoder-only |
88
+ | [`tpy09-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v1_wider_d128) | tpy09: semantically aligned corpus | V1 wider 128d |
89
+ | [`tpy09-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v2_base) | tpy09: semantically aligned corpus | V2 base |
90
+ | [`tpy09-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v2_decoder) | tpy09: semantically aligned corpus | V2 decoder-only |
91
+ | [`tpy09-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy09-v2_scaled) | tpy09: semantically aligned corpus | V2 scaled |
92
+ | [`tpy07b-v1_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v1_base) | tpy07b: clean held-out corpus | V1 base |
93
+ | [`tpy07b-v1_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v1_decoder) | tpy07b: clean held-out corpus | V1 decoder-only |
94
+ | [`tpy07b-v1_wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v1_wider_d128) | tpy07b: clean held-out corpus | V1 wider 128d |
95
+ | [`tpy07b-v2_base`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_base) | tpy07b: clean held-out corpus | V2 base |
96
+ | [`tpy07b-v2_decoder`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_decoder) | tpy07b: clean held-out corpus | V2 decoder-only |
97
+ | [`tpy07b-v2_scaled`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_scaled) | tpy07b: clean held-out corpus | V2 scaled |
98
+ | [`tpy07b-v2_scaled_plus1epoch`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/tpy07b-v2_scaled_plus1epoch) | tpy07b: clean held-out corpus | V2 scaled continued for one epoch |
99
+
100
+ ## V1 clean-holdout reruns
101
+
102
+ All models below retrain a V1 sweep cell on tpy07b and evaluate it on the frozen V2 benchmark.
103
+
104
+ | Branch | Experiment |
105
+ |---|---|
106
+ | [`v1_rerun-decoder_only`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-decoder_only) | Decoder-only rerun |
107
+ | [`v1_rerun-depth_1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-depth_1) | One-layer rerun |
108
+ | [`v1_rerun-depth_4`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-depth_4) | Four-layer rerun |
109
+ | [`v1_rerun-ettin17m_default`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-ettin17m_default) | Ettin-17M reference rerun |
110
+ | [`v1_rerun-prefix_k1`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-prefix_k1) | Prefix-length-1 rerun |
111
+ | [`v1_rerun-prefix_k8`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-prefix_k8) | Prefix-length-8 rerun |
112
+ | [`v1_rerun-wider_d128`](https://huggingface.co/chameleon-lizard/tiny_decoders/tree/v1_rerun-wider_d128) | Wider-128d rerun |