hbfreed/Qwen3.6-35B-A3B-winnow-keep50-healed

Qwen3.6-35B-A3B · Winnow keep-50 (healed)

Qwen/Qwen3.6-35B-A3B with 50% of routed-expert channels removed by winnow variable-width expert pruning — 18.5B parameters. Requires trust_remote_code=True (ragged variable-width experts, PrunedQwen3_5MoeForCausalLM); transformers ≥ 5.

Instead of deleting whole experts (REAP-style), winnow shrinks each expert's SwiGLU intermediate width according to a per-channel importance score (router weight × |post-SwiGLU activation| × down-projection column L2), allocated by a per-layer global rank in blocks of 128 channels. Attention, embeddings and the router are untouched. Calibration: C4, qwen35_stats1024.

Layout

Raw training run — final weights are checkpoints/step0200/ (load that subfolder). Intermediate checkpoints: step0025, step0100, step0200. trainer_state.pt files are optimizer state for resuming and are not needed for inference. heal_run/ holds args.json, the console log and per-step train_log.jsonl.

Healing recipe

On-policy distillation (scripts/62_qwen_distill_on_policy.py): reverse KL on the student's own rollouts against a bf16 Qwen3.6-35B-A3B teacher, 200 steps × 120k loss tokens, AdamW8bit, rollouts sampled with a 50/50 per-prompt mix of thinking / non-thinking chat frames (max 1024 new tokens), vLLM rollout engine refreshed every step, seed 1224. Trained on 3 × RTX Pro 6000 (Verda), 2026-07-22.

Results (step 0200, non-thinking greedy unless noted)

model	GSM8K (1319)	HumanEval pass@1 (pass@5)
Qwen3.6-35B-A3B (base)	0.957	0.940 (0.945)
keep-50 unhealed	0.840	0.017 (0.024)
keep-50 healed @200 (this)	0.925	0.448 (0.518)

Thinking-mode GSM8K for the healed keep-50: 0.946 (base thinking: 0.959). Evals via openbench (Protocol E), served through vLLM.