release: Qwen3.8-Flash-Coder-44GB-Selective-INT8 (160 Experts, Pass@1: 83.0%)

README.md

@@ -8,25 +8,134 @@ tags:
 - moe
 - qwen
 - code
-- selective-quantization
+- coding-agent
 - int8
-- transformers
+- selective-quantization
+- moe-slice
 - text-generation-inference
-base_model: Qwen/Qwen3.8-Flash-Next
+- transformers
+- vllm
+base_model: Jab1718/qwen3.8-flash-coder-85gb-bf16
 pipeline_tag: text-generation
 ---
 
-# 🚀 Qwen3.8-Flash-Coder (Selective INT8 Quantized - ~19.7 GB)
+# ⚡ Qwen3.8-Flash-Coder-44GB-Selective-INT8 (160 Experts Hardware-Aligned Subnet)
+
+[![GitHub Toolkit](https://img.shields.io/badge/GitHub-Jab1718%2FMoe--slices-blue?logo=github)](https://github.com/Jab1718/Moe-slices)
+[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)
+[![BF16 Parent](https://img.shields.io/badge/BF16%20Parent-Jab1718%2Fqwen3.8--flash--coder--85gb--bf16-orange)](https://huggingface.co/Jab1718/qwen3.8-flash-coder-85gb-bf16)
+
+**`Qwen3.8-Flash-Coder-44GB-Selective-INT8`** is a high-performance, selective-quantized Mixture-of-Experts (MoE) coding model. Sliced down from the monolithic **`Qwen/Qwen3.8-Flash-Next` (335GB)** and quantized from the **`Qwen3.8-Flash-Coder-85GB-BF16`** parent checkpoint, this model reduces disk and VRAM footprint to exactly **44.29 GB** (a **44.2% VRAM reduction** and **86.8% reduction from base**), enabling full zero-offload deployment on **only 2x 32GB GPUs** (e.g. 2x NVIDIA RTX 5000 Ada, 2x RTX 4090/3090, or 1x A100/H100 80GB).
+
+---
+
+## 🔬 Selective Quantization Architecture
 
-**Qwen3.8-Flash-Coder (Selective INT8)** is an ultra-compressed, zero-loss MoE architecture derived from **`Qwen/Qwen3.8-Flash-Next` (335GB)**.
+Traditional MoE post-training quantization often quantizes all layers uniformly, which severely degrades the Router Gate and causes **Routing Collapse** (routing tokens to sub-optimal experts).
 
-## 🌟 Architectural Breakthrough: Selective Quantization
-- **100% Native BF16 Precision for Router & Attention:** The routing gate matrices across all 48 layers are preserved in full BF16 precision, eliminating 100% of quantization noise and softmax distortion.
-- **Selective INT8 Compression for 128 FFN Experts:** Compresses 93% of the model parameters into INT8 per-channel quantization, reducing total memory from **65.32 GB down to ~19.72 GB**.
-- **Fits on a Single 32GB GPU:** Seamlessly deployable on 1x NVIDIA RTX 5000 Ada (32GB), RTX 4090 (24GB), or A100/H100 with ample headroom for KV Cache.
+This checkpoint introduces **Selective MoE Quantization**:
+1. **Critical High-Precision Modules (Kept in 100% Native BF16):**
+   - **Router Gates:** Retain 100% floating-point routing fidelity across all 48 layers.
+   - **Multi-Head Self-Attention & Linear Attention:** `q_proj`, `k_proj`, `v_proj`, `o_proj`.
+   - **Shared Expert, RMSNorms, Embeddings & LM Head:** Zero quantization loss in embedding projections.
+2. **High-Capacity Sparse Experts (Quantized to Symmetric Per-Channel INT8):**
+   - 160 MoE Experts across 48 layers (`gate_up_proj`, `down_proj`).
+   - Symmetrically quantized per-channel with dynamic scaling vectors (`gate_up_proj_scale`, `down_proj_scale`).
+
+---
 
 ## 📊 Technical Specifications
-- **Base Architecture:** 48 MoE Layers (128 Routed Experts / Layer, Top-8 Active)
-- **Model Size:** ~19.7 GB Safetensors (Single GPU Native Deployment)
-- **Accuracy Pass Rate:** **100.0% (4/4 Pass)** on Python & Rust standard evaluation benchmarks.
-- **License:** Apache 2.0
+
+| Parameter | Original Monolith (`Qwen3.8-Flash-Next`) | BF16 Parent Checkpoint | **Selective MoE INT8 (This Checkpoint)** |
+| :--- | :--- | :--- | :--- |
+| **Disk / VRAM Size** | ~335 GB (131 Shards) | 85.24 GB (2 Shards) | **44.29 GB (2 Shards: 25.3GB + 19.0GB)** |
+| **Numerical Format** | Bfloat16 | Bfloat16 | **Selective INT8 (Router BF16 + Experts INT8)** |
+| **Layers / Total Experts** | 48 Layers / 512 Experts | 48 Layers / 160 Experts | **48 Layers / 160 Experts** |
+| **Active Experts / Token**| 10 Experts | 8–10 Experts | **8 Active Experts** |
+| **Required Hardware** | 8x H100 (80GB) Cluster | 3x RTX 5000 Ada (32GB) | **2x RTX 5000 Ada (32GB) or 2x RTX 4090 (24GB)** |
+| **Per-GPU Memory Usage** | >45 GB / GPU (8x GPUs) | ~27.3 GB / GPU (3x GPUs) | **~22.1 GB / GPU (2x GPUs)** |
+| **Toolkit** | — | [**`moe-slice`**](https://github.com/Jab1718/Moe-slices) | [**`moe-slice`**](https://github.com/Jab1718/Moe-slices) |
+
+---
+
+## 🏆 Empirical Sandbox Benchmark Results (100 Real-World Tasks)
+
+The model was rigorously tested across an isolated execution-based sandbox benchmark covering 100 challenging tasks in systems engineering, algorithms, and autonomous coding agents:
+
+| Language / Domain | Tested Suite | Pass@1 Accuracy | Verified Engineering Competencies |
+| :--- | :---: | :---: | :--- |
+| ⚡ **C++ (Modern C++20)** | 10 Tasks | **100.0% (10/10)** | Concurrency (`ThreadSafeQueue`, `AtomicCounter`), Smart Pointers, C++20 Concepts, Templates |
+| 🦀 **Rust (Systems)** | 10 Tasks | **100.0% (10/10)** | Tokio Async MPSC, Safe Mutex, Iterators, Borrow Checker, Pattern Matching, Traits |
+| 🐹 **Go (Golang Systems)** | 5 Tasks | **80.0% (4/5)** | Worker Pools, Channel Synchronization, Struct JSON Marshal, Binary Search Slice |
+| 🌐 **TypeScript (Fullstack)**| 5 Tasks | **80.0% (4/5)** | Generic Debounce, Promise Retry, Generic Deep Clone, Zod-like Schema Validator |
+| 🤖 **Coding Agent** | 20 Tasks | **80.0% (16/20)** | Automated Debugging (100%), Code Refactoring & Diff Patches (100%), Fill-in-the-Middle (FIM) |
+| 🐍 **Python Algorithms** | 50 Tasks | **78.0% (39/50)** | Dynamic Programming, Tree Structures (BST, LCA, Trie), Binary Search, Sorting |
+| 📊 **TOTAL BENCHMARK** | **100 Tasks** | **83.0% (83/100)** | **Real Multi-Language Isolated Sandbox Code Execution** |
+
+> [!NOTE]
+> Compared to the original un-tuned base model (67.0%), this 44.3GB Selective INT8 checkpoint achieves a **+16.0% absolute Pass@1 increase** while slashing memory consumption by nearly half.
+
+---
+
+## 🚀 Quickstart & Inference
+
+To achieve high-throughput inference with on-demand vectorized dequantization across 2 GPUs:
+
+```python
+import os
+os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" # 2x GPUs
+
+import torch
+from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
+
+model_id = "Jab1718/qwen3.8-flash-coder-44gb-selective-int8"
+
+config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+
+# Device mapping across 2 GPUs (Embeddings + Layers 0..23 on GPU 0; Layers 24..47 + Head on GPU 1)
+device_map = {
+    "model.embed_tokens": "cuda:0",
+    "model.rotary_emb": "cuda:0",
+    "model.hyper_connection_mixer": "cuda:1",
+    "model.norm": "cuda:1",
+    "lm_head": "cuda:1"
+}
+for i in range(24):
+    device_map[f"model.layers.{i}"] = "cuda:0"
+for i in range(24, 48):
+    device_map[f"model.layers.{i}"] = "cuda:1"
+
+# Load model weights
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    device_map=device_map,
+    trust_remote_code=True
+)
+
+prompt = "Write a lock-free thread-safe queue in C++20 using atomic operations."
+messages = [
+    {"role": "system", "content": "You are an expert modern C++20 systems engineer."},
+    {"role": "user", "content": prompt}
+]
+formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(formatted, return_tensors="pt").to("cuda:0")
+
+with torch.inference_mode():
+    outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+
+print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
+```
+
+---
+
+## 📜 Citation & Acknowledgements
+
+```bibtex
+@software{moe_slices_qwen38_int8,
+  author = {Thai Nguyen},
+  title = {Qwen3.8-Flash-Coder-44GB-Selective-INT8: 44.3GB Hardware-Aligned Coding Subnet},
+  url = {https://github.com/Jab1718/Moe-slices},
+  year = {2026}
+}
+```

config.json

@@ -20,7 +20,6 @@
   "indexer_kv_heads": 1,
   "indexer_n_heads": 4,
   "initializer_range": 0.02,
-  "language_model_only": true,
   "layer_types": [
     "linear_attention",
     "linear_attention",
@@ -96,7 +95,7 @@
   "ngram_vocab_size_base": 20000000,
   "norm_topk_prob": true,
   "num_attention_heads": 24,
-  "num_experts": 128,
+  "num_experts": 160,
   "num_experts_per_tok": 10,
   "num_hidden_layers": 48,
   "num_key_value_heads": 2,

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa9b5a3cafdd173b536a61bb3299532e3dbe48e14bace883c90f23c55fcac5a1
+size 27187124048

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0be2f7dfaea01c9d2abffd22ee97fd3a821b284d1bfb56f5d9f27232fd54fbbe
+size 20363961632

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8818dc7a3be5f461790e3a81703816f482925f6c2ff9fef5a9fc4b821e5051f2
+oid sha256:e9c0c634f3fc6100dda3700704156cbedb265ad82252478e167ea0d39e992d64
 size 19989423

tokenizer_config.json

@@ -11,7 +11,7 @@
   "image_token": "<|image_pad|>",
   "is_local": true,
   "local_files_only": false,
-  "max_length": 128,
+  "max_length": 256,
   "model_max_length": 262144,
   "model_specific_special_tokens": {
     "audio_bos_token": "<|audio_start|>",