bs8-moe8-lr7xct-20260722-iter7651: block-8 MoE-8 lr7x-ct iter_7651 (accept_len 4.704) -- current best

README.md

@@ -9,18 +9,21 @@ tags:
   - code
 ---
 
-# dspark-glm52-bs8-moe8-dflare-20260715
+# dspark-glm52-bs8-moe8-lr7xct-20260722
 
-**Mid-training SFT checkpoint** (step 10001 / 20730, epoch 3 of 6) of the DSpark draft model
+**Mid-training SFT checkpoint** (step 7651 / 10365, epoch 3 of 3) of the DSpark draft model
 for **GLM-5.2** speculative decoding, optimized for code / coding-chat traffic.
 
+Supersedes `dspark-glm52-bs8-moe8-dflare-20260714/20260715` (same architecture; this is the
+high-LR warm-restart continuation, `official_accept_len` 4.59 → **4.70** on the same trend set).
+
 ## Architecture
 
 - **DSpark** (DFlash-family single-shot parallel drafting): block size **8**, 5 draft layers,
   hidden 6144, 64 heads (head_dim 64).
 - **Vanilla MoE-8**: 8 routed experts (top-2, norm_topk) + 1 shared expert per layer,
   moe_intermediate 3072. Jakiro position-MoE **OFF**.
-- **DFlare per-layer context fusion ON** (zero-init gates; bit-identical to base arm at step 0).
+- **DFlare per-layer context fusion ON** (zero-init gates).
 - Markov/GRIFFIN logit-correction head (rank 256, vanilla) + confidence head
   (`confidence_detach_input`), `griffin_fuse` on.
 - Reads target hidden states from GLM-5.2 layers **[8, 23, 39, 55, 70]** (of 78).
@@ -28,7 +31,8 @@ for **GLM-5.2** speculative decoding, optimized for code / coding-chat traffic.
   0.20 B draft lm_head) ≈ **2.7 B active/token**. bf16, 8.3 GB.
 - `draft_vocab_size` **32000** vs target vocab 154880: serving through sglang's DSpark loader
   requires the **d2t full-vocab scatter conversion** (the loader argmaxes the draft lm_head
-  directly as target ids); TorchSpec→sglang key renames apply.
+  directly as target ids); TorchSpec→sglang key renames apply. `d2t` (int64 [32000]) and
+  `t2d` (bool [154880]) ship in the safetensors and must NOT be cast to bf16.
 
 ## Training
 
@@ -38,29 +42,48 @@ for **GLM-5.2** speculative decoding, optimized for code / coding-chat traffic.
   (1,379,922 production DB conversations + 522,176 on-policy OPB rows), seq 4096, GLM chat template.
 - Objective: decay-weighted CE (0.1) + L1 (0.9) on target states, confidence BCE,
   VSD 0.3 (ramp 30@40), MoE aux/z-loss 1e-3, label convention `infill`, 256 anchors.
-  Optimizer muon, lr 5.4e-4 cosine, global batch 512, warm start from the 1-epoch champion
-  (`champion_p7moe_bs16_nvfp4_fullmix_graph_chunk_1ep/iter_0001141`).
-- W&B: `morphllm/torchspec-p7autoresearch/ts-sft-ec6255b7913a4a5b`.
+- Optimizer muon, **peak lr 2.3e-3** cosine → min 6e-6 over a 3-epoch horizon,
+  warmup_ratio 0.02, seed 43, global batch 512 (micro 8 × accum 8 × dp 8), FlashAttention-4.
+- **Weights-only warm restart** from `sft/loadroot_iter9951` (the pre-bump champion of the
+  predecessor dflare run, `official_accept_len` 4.594), fresh optimizer + fresh schedule.
+- W&B: `morphllm/torchspec-p7autoresearch/ts-sft-lr7xct-9951`.
 
 ## Eval (caveat first)
 
-`official_accept_len` **4.59 of 8** at the step-10000 eval on the fixed 96-row trend set.
-The trend set **overlaps the training view** (no holdout exists in the immutable view) and is a
-teacher-forced trend metric — it is NOT a serving acceptance measurement. Curve so far:
-3.34 (step 50) → 4.05 (1250) → 4.25 (2900) → 4.33 (3700) → 4.39 (5250) → 4.49 (7500) →
-4.59 (10000), still climbing log-linearly. Training attention backend switched to
-FlashAttention-4 at ~step 3800 (training-side change only; weight semantics unchanged).
-Keep/kill decisions belong to the sglang serving bench.
+`official_accept_len` **4.7040 of 8** at the step-7650 eval on the fixed 96-row trend set.
+
+**This is a trend metric, not a serving measurement, and it is not a SOTA claim.** The trend
+set **overlaps the training view** (no holdout exists in the immutable view), it is
+teacher-forced, and this checkpoint is in **epoch 3** over that same view — so part of the
+late-curve gain is second/third-pass fit rather than generalization. Per this program's
+methodology, keep/kill belongs to the offline 3-task screen (gsm8k + humaneval + mt-bench,
+×32 samples, temp 1.0) and to an sglang serving bench; neither has been run on this
+checkpoint. Trend-eval noise is ±0.03–0.04, so single-eval deltas below ~0.05 are not real.
+
+Curve (this run, step → `official_accept_len`; step 0 == the 4.594 restart point):
+
+| 50 | 500 | 1500 | 3000 | 4500 | 6000 | 7000 | 7500 | **7650** |
+|---|---|---|---|---|---|---|---|---|
+| 4.566 | 4.082 | 4.207 | 4.353 | 4.479 | 4.595 | 4.653 | 4.676 | **4.704** |
+
+The dip to ~4.03 around steps 250–500 is the expected high-LR warm-restart excursion
+(fresh optimizer state entering a 2.3e-3 peak); it recovered and passed the restart point
+by step ~6300. `official_accept_len` is the published formula (per-block cumprod of
+`alpha_k = Σ min(draft_p, target_p)`, then averaged), verified against the speculators
+library's own `dspark/metrics.py` — the same shape as a reference-harness figure, though
+comparison to any published number still requires matched block size, temperature, and target.
 
 ## Provenance
 
-- Checkpoint: `glm52-dspark-moe8-draft-opd/sft/checkpoints/iter_0010001` (FSDP full-state),
-  converted via `TorchSpec/tools/convert_to_hf.py` (native sglang keys, bf16,
-  embed_tokens injected from the NVFP4 target).
-- Pipeline identity `7b901cf00f421aff…`, draft config
-  `dspark_draft_config_glm_p7_bs8_moe8_plf.json` (d10b170ef lineage).
+- Checkpoint: `glm52-dspark-moe8-draft-opd/sft_lr7x_ct/checkpoints/iter_0007651` (FSDP
+  full-state, best-pinned by `eval/official_accept_len`), converted via
+  `TorchSpec/tools/convert_to_hf.py` (native sglang keys, bf16, embed_tokens injected from
+  the NVFP4 target).
+- Pipeline identity `7b901cf00f421aff…`; draft config
+  `dspark_draft_config_glm_p7_bs8_moe8_plf.json`,
+  sha256 `58f164050a6d70039ef333e9e866c2117380207b7eff302272bd39887f39d6f8` (matches the
+  run's recorded `pipeline_draft_config_fingerprint`).
 - Tokenizer: use GLM-5.2's (not bundled).
 
-Later checkpoints of this run will supersede this snapshot; earlier snapshots (iter_0003701,
-iter_0005251) remain in the repo's commit history. SFT is followed by a verifier-replay
-Draft-OPD stage (8 epochs).
+Later checkpoints of this run will supersede this snapshot. SFT is followed by a
+verifier-replay Draft-OPD stage.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3d294578b4f3011a6eeb75a49784286ffab115ffe8fed5362c03495ab164f809
+oid sha256:409276ce1ee8f40bcf6468c4351411001227a7c2ae8a48317b6e5fae71929640
 size 8876715950