Persistent Sparse Autoencoders

Trained checkpoints from the paper Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations.

Checkpoints

Folder Model Layer d_model Training steps
pythia-160m-deduped/layer_6 EleutherAI/pythia-160m-deduped 6 768 244,140
pythia-410m-deduped/layer_12 EleutherAI/pythia-410m-deduped 12 1024 244,140
pythia-1b-deduped/layer_8 EleutherAI/pythia-1b-deduped 8 2048 244,140
pythia-1.4b-deduped/layer_12 EleutherAI/pythia-1.4b-deduped 12 2048 244,140
gemma-2-2b/layer_12 google/gemma-2-2b 12 2304 244,140
pythia-2.8b-deduped/layer_16 EleutherAI/pythia-2.8b-deduped 16 2560 367,648
pythia-6.9b-deduped/layer_16 EleutherAI/pythia-6.9b-deduped 16 4096 367,648
llama-3.1-8b/layer_19 meta-llama/Llama-3.1-8B 19 4096 367,648
gemma-2-9b/layer_20 google/gemma-2-9b 20 3584 367,648
pythia-12b-deduped/layer_18 EleutherAI/pythia-12b-deduped 18 5120 367,648

Citation

@misc{luo2026persistentsparseautoencoderslearning,
      title={Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations}, 
      author={Haoyan Luo and Mateo Espinosa Zarlenga and Mateja Jamnik},
      year={2026},
      eprint={2607.17117},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.17117}, 
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for haoyanl78/persistent-sparse-autoencoders