georglange/qwen3-4b-molt

Qwen3-4B MOLT checkpoints

Final model weights for sparse mixtures of linear transforms (MOLTs), trained independently on Qwen3-4B MLP layers.

All 36 layers: 0–35, each at 100,000 training steps. Layer numbers are zero-based. Training is complete for every layer.

Each layer folder contains checkpoint.safetensors and config.json. The Safetensors file contains model weights and required buffers only: encoder weights and bias, low-rank transform matrices, JumpReLU thresholds, and input/output normalization statistics. No optimizer states, scheduler states, or training-loop state are included.

Original tensor values and dtypes are preserved exactly: learned weights are FP32; normalization buffers retain their saved FP32 or BF16 dtype. Each layer is approximately 4.22 GB, for approximately 151.94 GB in total.

• Target model: Qwen/Qwen3-4B.
• Architecture: 2,480 transforms per layer; N=80; ranks 512, 256, 128, 64, 32.
• Training data: UltraChat 200k train_sft, with the native chat template.
• Recipe: learning rate 4e-5, peak sparsity coefficient 7.5e-5, sparsity ramp through 80% of training, learning-rate decay over the final 20%.
• Inputs: before post-attention normalization; target: raw MLP output.
• Tensor names match Molt.state_dict() (the Lightning model. prefix is removed).

Download and read one checkpoint

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download(
    repo_id="georglange/qwen3-4b-molt",
    filename="layer-22/checkpoint.safetensors",
)
state = load_file(path)

Training and model implementation: crosslayer-transcoder. The target Qwen3-4B language-model weights are available separately from the model repository linked above.

Explanations and scores

Automated explanations, four Delphi scores, and the exact scoring prompts for all 89,280 transforms are available in the separate dataset georglange/qwen3-4b-molt-interpretations. It covers all 36 layers on a shared 10M-token UltraChat evaluation sample. Explanations and aggregate scores can be downloaded separately from the prompts.