| --- |
| base_model: siddharthmb/2026.AP.cpt_ifp |
| datasets: |
| - allenai/tulu-3-sft-mixture |
| --- |
| |
| # 2026.AP.sft_ifp |
| |
| Main IFPruning SFT run: initialized from the cpt_ifp model (init_from models/cpt_ifp) and fine-tuned on tulu-3-sft-mixture with the predictor-generated masks active (llm_lr 1e-5, grad_accum 8, max 500k examples, 1 epoch). |
|
|
| Part of a replication of Apple's **Instruction-Following Pruning for Large Language Models** ([arXiv:2501.02086](https://arxiv.org/abs/2501.02086)) — "AP" = apple-paper-replicate, package `ifpruning` in [Sid-MB/mats_exploration](https://github.com/Sid-MB/mats_exploration) under `code/apple-paper-replicate/` (branch `introspection-causal-test`, merged to main at `a76965b`). |
|
|
| ## Architecture |
|
|
| - LLM: Qwen2.5-3B (Qwen2ForCausalLM, 36 layers, d_ffn 11008) |
| - Predictor backbone: Qwen2.5-0.5B (Qwen2Model) + 2-layer MLP head (`head.pt`) producing per-layer FFN importance scores `[36, 11008]`; per-row SoftTopK selects t_ffn=1536 of 11008 FFN units (~1B activated params). Dense-baseline runs train the same LLM without masking. |
|
|
| ## Contents |
|
|
| - `checkpoints/step_1000/pytorch_model_fsdp_0/` — FSDP2 SHARDED_STATE_DICT model weights (llm + predictor_backbone + head) |
| - `checkpoints/step_1000/optimizer_0/` — optimizer state (for exact training resumption) |
| - `checkpoints/step_1000/random_states_*.pkl`, `scheduler.bin` — RNG/scheduler state |
| |
| ## Performance |
| |
| **No evaluations were run on this checkpoint.** The project reached "scaffold + smoke test + this training grid" before being paused (see `code/mats_exploration/everything we learned.md`); `logs/eval/` is empty and no eval_results directory exists. The only training-quality signal is the loss curves in the wandb runs below. |
| |
| ## Reproduction |
| |
| From `code/` in the mats_exploration repo (paths as of June 2026; `IFP_ROOT=/nlp/scr/siddharth/apple-paper-replicate` set in `slurm/_common.sh`): |
| |
| ```bash |
| sbatch apple-paper-replicate/slurm/train.sbatch apple-paper-replicate/configs/presets/sft_ifp.yaml |
| ``` |
| |
| which runs (8 GPUs, accelerate FSDP2 full-shard bf16, `SHARDED_STATE_DICT`): |
| |
| ```bash |
| srun uv run accelerate launch --config_file apple-paper-replicate/configs/accelerate_fsdp8.yaml \ |
| -m ifpruning.train --config apple-paper-replicate/configs/presets/sft_ifp.yaml \ |
| --ckpt-root $IFP_ROOT/ckpts --out-root $IFP_ROOT/models |
| ``` |
| |
| Preset: `configs/presets/sft_ifp.yaml`. Data: allenai/tulu-3-sft-mixture. Seed 0. Slurm job 15878287 (jagupard39, 8 GPUs, afterok:15878284). |
| |
| ## Weights & Biases |
| |
| - https://wandb.ai/siddharth-stanford/ifpruning-sft/runs/31cgnihz |
| |
| ## Cluster paths (Stanford NLP) |
| |
| - Training log: `/nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/logs/train/sft_ifp_15878287.out` |
| - Original checkpoint dir: `/nlp/scr2/siddharth/apple-paper-replicate/ckpts/sft_ifp` (deleted after this upload was verified; this repo is now the only copy) |
| - Code, configs, paper PDF, research notes: `/nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/` |
| - Research note with the full run grid: `.../research-notes/2026-06-12_setup-and-smoke.md` |
| |
