Add model card

README.md

@@ -0,0 +1,55 @@
+---
+base_model: siddharthmb/2026.AP.cpt_ifp
+datasets:
+- allenai/tulu-3-sft-mixture
+---
+
+# 2026.AP.sft_ifp
+
+Main IFPruning SFT run: initialized from the cpt_ifp model (init_from models/cpt_ifp) and fine-tuned on tulu-3-sft-mixture with the predictor-generated masks active (llm_lr 1e-5, grad_accum 8, max 500k examples, 1 epoch).
+
+Part of a replication of Apple's **Instruction-Following Pruning for Large Language Models** ([arXiv:2501.02086](https://arxiv.org/abs/2501.02086)) — "AP" = apple-paper-replicate, package `ifpruning` in [Sid-MB/mats_exploration](https://github.com/Sid-MB/mats_exploration) under `code/apple-paper-replicate/` (branch `introspection-causal-test`, merged to main at `a76965b`).
+
+## Architecture
+
+- LLM: Qwen2.5-3B (Qwen2ForCausalLM, 36 layers, d_ffn 11008)
+- Predictor backbone: Qwen2.5-0.5B (Qwen2Model) + 2-layer MLP head (`head.pt`) producing per-layer FFN importance scores `[36, 11008]`; per-row SoftTopK selects t_ffn=1536 of 11008 FFN units (~1B activated params). Dense-baseline runs train the same LLM without masking.
+
+## Contents
+
+- `checkpoints/step_1000/pytorch_model_fsdp_0/` — FSDP2 SHARDED_STATE_DICT model weights (llm + predictor_backbone + head)
+- `checkpoints/step_1000/optimizer_0/` — optimizer state (for exact training resumption)
+- `checkpoints/step_1000/random_states_*.pkl`, `scheduler.bin` — RNG/scheduler state
+
+## Performance
+
+**No evaluations were run on this checkpoint.** The project reached "scaffold + smoke test + this training grid" before being paused (see `code/mats_exploration/everything we learned.md`); `logs/eval/` is empty and no eval_results directory exists. The only training-quality signal is the loss curves in the wandb runs below.
+
+## Reproduction
+
+From `code/` in the mats_exploration repo (paths as of June 2026; `IFP_ROOT=/nlp/scr/siddharth/apple-paper-replicate` set in `slurm/_common.sh`):
+
+```bash
+sbatch apple-paper-replicate/slurm/train.sbatch apple-paper-replicate/configs/presets/sft_ifp.yaml
+```
+
+which runs (8 GPUs, accelerate FSDP2 full-shard bf16, `SHARDED_STATE_DICT`):
+
+```bash
+srun uv run accelerate launch --config_file apple-paper-replicate/configs/accelerate_fsdp8.yaml \
+    -m ifpruning.train --config apple-paper-replicate/configs/presets/sft_ifp.yaml \
+    --ckpt-root $IFP_ROOT/ckpts --out-root $IFP_ROOT/models
+```
+
+Preset: `configs/presets/sft_ifp.yaml`. Data: allenai/tulu-3-sft-mixture. Seed 0. Slurm job 15878287 (jagupard39, 8 GPUs, afterok:15878284).
+
+## Weights & Biases
+
+- https://wandb.ai/siddharth-stanford/ifpruning-sft/runs/31cgnihz
+
+## Cluster paths (Stanford NLP)
+
+- Training log: `/nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/logs/train/sft_ifp_15878287.out`
+- Original checkpoint dir: `/nlp/scr2/siddharth/apple-paper-replicate/ckpts/sft_ifp` (deleted after this upload was verified; this repo is now the only copy)
+- Code, configs, paper PDF, research notes: `/nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/`
+- Research note with the full run grid: `.../research-notes/2026-06-12_setup-and-smoke.md`