--- license: apache-2.0 library_name: transformers base_model: osk-arr00/qwen-3.8-next-40b-exp-moe-pruned-bf16 pipeline_tag: text-generation tags: - moe - qwen4_exp - router-healing - st-moe - ple-calibrated - research - experimental - not-for-production --- # ⚠️ EXPERIMENTAL WEIGHTS — NOT A FINAL RELEASE ⚠️ > [!WARNING] > ## THESE ARE EXPERIMENTAL RESEARCH WEIGHTS. THIS IS **NOT** A FINISHED OR PRODUCTION MODEL. > > Every `qwen-3.8-next-40b-exp-*` repository is an **intermediate checkpoint of an ongoing > weight-surgery research line** — not a release, not a final version, not a validated product. > > - **Not validated** for end-user deployment, standalone inference, or any downstream product. > - **No safety, alignment, or quality guarantees.** Behaviour can change or regress between checkpoints. > - **Non-standard architecture** (`qwen4_exp`): requires an inference stack that explicitly supports it. > - Published **only** for reproducibility of MoE pruning, router-healing, PLE-calibration and > distillation experiments. > > If you are looking for a model to actually use, use the upstream Qwen release instead. # Qwen-3.8-Next-40B-MoE-Healed (BF16) Full BFloat16 weights of the experimental hybrid model **Qwen-3.8-Next-40B** after memory-pruning surgery and a routing-stabilization run (*router healing*) over the 48 MoE gates $W_{\text{gate}}$, with the PLE associative memory table pruned to 2.0M entries. ## 1. Diagnosis of the pruned base The pruned base checkpoint showed **expert starvation and routing collapse**: 1. **Routing imbalance** — the 48 $W_{\text{gate}}$ routers still carried logits initialized for a much larger expert space, leaving **~38% of experts fully inactive** or below 0.2% traffic. 2. **Syntactic collapse into loops** — critical layers (layer 3, Gini 0.728; layer 44, Gini 0.706) concentrated up to 8.8% of all traffic in a single monopolist expert, inducing infinite lexical repetition in open-ended generation. 3. **PLE hypertrophy** — factual-memory injection ran at $\rho = \|\Delta h_{\text{PLE}}\| / \|h_{\text{residual}}\| = 0.658$, roughly double the stable range ($0.20 \sim 0.35$). ## 2. Router-healing SFT (150 steps) A closed-loop protocol with **triple balancing pressure**: - **ST-MoE auxiliary balance loss**: $\mathcal{L}_{\text{aux}} = N \sum_{i=1}^N f_i \cdot P_i$ - **Logarithmic starvation barrier (KL)**: asymmetric exponential penalty on experts whose share falls below $\frac{0.5}{N}$. - **Inter-expert load variance**: $\mathcal{L}_{\text{var}} = \text{Var}(f)$. - **Parametric surgery**: 99.96% of the model frozen. Only the 48 $W_{\text{gate}}$ routers (15.73M trainable parameters) were optimized with AdamW, warmup and cosine decay ($LR_{\max} = 3 \times 10^{-4}$). ## 3. Quantitative results, pre vs post healing Audited on an NVIDIA RTX PRO 6000 (96 GB VRAM) with a multi-domain battery (concurrent Python, CoT mathematics, legal doctrine, tool calling): | Indicator | Pre-healing | Post-healing | Change | | :--- | :---: | :---: | :--- | | Trigram repetition | Pathological (infinite loops) | **0.69%** | Loops eliminated | | Lexical diversity (Distinct-1) | $<0.30$ | **0.81** | High vocabulary variability | | Shannon routing entropy ($H$) | $0.680$ | **$0.8353$** | **+22.8%** | | Inactive experts (0 hits) | $22.6$ / 128 (17.7%) | **$16.6$ / 128 (12.9%)** | 6 dead experts revived per layer | | Top-1 expert share | $7.20\%$ | **$6.12\%$** | **-15.0%** | | Top-3 expert share | $18.90\%$ | **$15.75\%$** | **-16.7%** | | PLE injection ratio ($\rho$) | $0.658$ | **$0.600$** | Damped toward the stable band | > [!NOTE] > These numbers describe **relative improvement over a broken checkpoint**, not competitive > benchmark results. No standard benchmark (MMLU, GSM8K, HumanEval…) has been run on this > checkpoint, and none should be inferred from the table above. ## Architecture (`qwen4_exp`) | Property | Value | |---|---| | Total layers | 48 hybrid MoE blocks | | Hidden size | 2,560 | | Attention | Gated DeltaNet linear attention, 3 linear blocks per 1 full-attention block (`full_attention_interval: 4`) | | Hyper-Connections | 4 residual streams, rank-320 projections | | Routed experts | 128 per layer, top-10 active (`num_experts_per_tok: 10`) | | Shared expert | 1 dense expert (intermediate size 640) | | PLE memory | Associative n-gram table, 2,000,000 base entries (`ngram_size: 3`) | | Vocabulary | 248,320 | | Native context | 262,144 tokens | | Precision | BFloat16 | - **Total parameters**: 40.22B (1,294 tensors, 80.44 GB BF16) - **Activated per token**: top-10 routed experts + 1 shared dense expert ($\approx$ 2.6B expert parameters per token, on top of the attention and embedding stack) - **Checkpoint class**: text-only (`Qwen4ExpForCausalLM`); the vision tower is not included - **PLE memory**: n-gram table pruned to 2,000,000 entries ## 4. Requirements Non-standard architecture. It needs an inference stack with explicit `qwen4_exp` support (Gated DeltaNet linear attention + 4-stream Hyper-Connections + PLE associative memory). Standard dense-Transformer loaders will **not** work. > [!NOTE] > Shard 1 is ~49.6 GB. Use `hf download` / `snapshot_download` with resume rather than a plain > browser download. ## Experiment line | Repository | Role | |---|---| | [`qwen-3.8-next-40b-exp-clean-bf16`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-clean-bf16) | Clean baseline | | [`qwen-3.8-next-40b-exp-clean-healed-bf16`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-clean-healed-bf16) | Router-healed, text-only | | [`qwen-3.8-next-40b-exp-moe-pruned-bf16`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-moe-pruned-bf16) | 512→128 expert pruning | | [`qwen-3.8-next-40b-exp-moe-healed-bf16`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-moe-healed-bf16) | Pruned + router-healed + PLE 2.0M | | [`qwen-3.8-next-40b-exp-ple-2m-table`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-ple-2m-table) | Routing tables and calibration artifacts | | [`qwen-3.8-next-40b-exp-cartridge-general`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-cartridge-general) | Delta cartridge testbed | | [`qwen-3.8-next-40b-exp-profiling-metrics`](https://huggingface.co/osk-arr00/qwen-3.8-next-40b-exp-profiling-metrics) | Profiling metrics | ## 5. Credits Based on the architectures and research of **David Aylward (`logic65`)**, the **Qwen team (Alibaba Cloud)**, and work on adaptive MoE routing. ## License Apache 2.0, inherited from the upstream Qwen base model.