Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations
Paper โข 2607.17117 โข Published
Trained checkpoints from the paper Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations.
| Folder | Model | Layer | d_model | Training steps |
|---|---|---|---|---|
pythia-160m-deduped/layer_6 |
EleutherAI/pythia-160m-deduped |
6 | 768 | 244,140 |
pythia-410m-deduped/layer_12 |
EleutherAI/pythia-410m-deduped |
12 | 1024 | 244,140 |
pythia-1b-deduped/layer_8 |
EleutherAI/pythia-1b-deduped |
8 | 2048 | 244,140 |
pythia-1.4b-deduped/layer_12 |
EleutherAI/pythia-1.4b-deduped |
12 | 2048 | 244,140 |
gemma-2-2b/layer_12 |
google/gemma-2-2b |
12 | 2304 | 244,140 |
pythia-2.8b-deduped/layer_16 |
EleutherAI/pythia-2.8b-deduped |
16 | 2560 | 367,648 |
pythia-6.9b-deduped/layer_16 |
EleutherAI/pythia-6.9b-deduped |
16 | 4096 | 367,648 |
llama-3.1-8b/layer_19 |
meta-llama/Llama-3.1-8B |
19 | 4096 | 367,648 |
gemma-2-9b/layer_20 |
google/gemma-2-9b |
20 | 3584 | 367,648 |
pythia-12b-deduped/layer_18 |
EleutherAI/pythia-12b-deduped |
18 | 5120 | 367,648 |
@misc{luo2026persistentsparseautoencoderslearning,
title={Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations},
author={Haoyan Luo and Mateo Espinosa Zarlenga and Mateja Jamnik},
year={2026},
eprint={2607.17117},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2607.17117},
}