--- license: apache-2.0 base_model: Qwen/Qwen3-8B language: - en - de tags: - lora - peft - function-calling - task-planning - moe-orchestration - gguf - lumi - amd-mi250x pipeline_tag: text-generation --- # qwen3-planner-sft A LoRA fine-tune of [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B), trained to act as the **Planner** component of [MoE Sovereign](https://github.com/h3rb3rn/moe-sovereign), a local, sovereign Mixture-of-Experts LLM orchestrator. Given a user request, the model decomposes it into 1–4 subtasks and emits a single JSON array assigning each subtask to a specialist expert category or a deterministic MCP tool — nothing else. It is **not** a general-purpose chat/instruct model; it has one job, and the training and evaluation below are scoped to that job. This card documents the training run in detail, including two significant failure modes from an earlier attempt, because the fixes are only meaningful in that context and because HPC-cluster SFT failures of this shape (silent, no error, normal-looking loss curve) are easy to reproduce elsewhere if the causes aren't spelled out. ## Model summary | | | |---|---| | Base model | Qwen/Qwen3-8B (dense, 8.19B params, Apache 2.0) | | Fine-tuning method | LoRA, r=16, α=32, dropout=0.05 | | LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` | | Trainable params | 43.65M / 8.23B (0.53%) | | Precision | BF16 (full precision base + LoRA; no quantization during training) | | Context length trained | 4,096 tokens | | Formats published | safetensors (merged, BF16, ~16 GB), GGUF F16 (~16 GB), GGUF Q4_K_M (~4.7 GB, this repo) | | License | Apache 2.0 (inherited from base model; no additional restrictions) | ## Intended use Structured task decomposition for a multi-expert LLM orchestrator. Input: a system prompt fixing a 16-category expert taxonomy plus a tool catalog (see "Prompt format" below), followed by a user query. Output: **only** a JSON array, e.g.: ```json [{"task": "Explain the difference between TCP and UDP", "category": "technical_support"}, {"task": "Convert 100 km/h to m/s", "category": "precision_tools", "mcp_tool": "unit_convert", "mcp_args": {"value": 100, "from_unit": "km/h", "to_unit": "m/s"}}] ``` The model is trained against one specific, versioned system prompt (`PLANNER_SYSTEM_PROMPT` in the MoE Sovereign codebase, ~2,500 tokens, 16 expert categories + a 22-tool precision catalog). Using it with a different taxonomy or system prompt is expected to degrade quality — see "Known limitations." ## Training data - **259,829** synthetic `(query, plan)` pairs, 258,529 train / 1,300 held-out eval (`val_split=0.005`). - 5 system-prompt phrasing variants plus ~15% adversarial/negative samples (queries designed to probe category-boundary confusion, e.g. arithmetic phrased as a "technical" question that should still route to `precision_tools`, not `technical_support`). - Teacher model: `meta-llama/Meta-Llama-3.1-405B-Instruct` (project default for this dataset generator; `Qwen/Qwen3-235B-A22B-Instruct` is the documented fallback teacher — the exact teacher used for this specific merged dataset build was not independently re-verified against the original generation logs at model-card-writing time, so treat this as "default per the generation script," not as an independently re-confirmed fact). - Dataset is internal to the MoE Sovereign project and not currently published. ## Training procedure - **Framework:** TRL `SFTTrainer` + PEFT LoRA + DeepSpeed ZeRO-2 (not ZeRO-3 — the 8B model fits in a single GPU's memory without parameter sharding, so ZeRO-2 is faster and, unlike ZeRO-3, composes cleanly with LoRA). - **Loss masking:** `assistant_only_loss=True` — loss is computed only over the assistant's JSON response, not over the system+user portion of the sequence. This is a native TRL 1.4 feature and is the fix for the truncation failure described below, not an incidental setting. - **Schedule:** 3 epochs, micro-batch 4 × grad-accum 4 × 8 GPUs = effective batch 128, cosine LR schedule, peak LR 2e-4, gradient checkpointing enabled. - **Attention:** eager (`attn_implementation="eager"`) — Flash Attention 2 is not available for the ROCm 7.0 / PyTorch 2.10 build used on this cluster, so attention materializes the full O(seq²) matrix rather than a fused/tiled kernel. This is the dominant cost factor behind the per-step latency below, not model size or LoRA overhead. ### Hardware: LUMI-G (EuroHPC) Trained on **LUMI-G**, the GPU partition of the EuroHPC LUMI supercomputer (CSC, Finland), under EuroHPC project allocation `project_465003058`. One `small-g` node: 8× AMD Instinct MI250X GCDs (Graphics Compute Dies — each physical MI250X card exposes two GCDs as separate devices), 512 GB HBM2e, ROCm 7.0, PyTorch 2.10+rocm7.0, executed inside a Singularity container. ### Training run and two prior failure modes This model is the result of the **second** full training attempt against this dataset. Both failure modes below are documented here (rather than only in internal project notes) because they cost real HPC allocation and neither produced an error — both looked like a normal, converging training run right up until post-hoc inspection. **Attempt 1 — silent truncation of the training target (root-caused, not repeated here):** The first run used `max_seq_len=1536`, carried over from an earlier, much shorter prompt format and never re-checked after the system prompt grew to ~2,500 tokens. With `truncation_side="right"` and no completion-only loss masking, the assistant's JSON target — which starts *after* the ~2,500-token system prompt — fell outside the 1,536-token window for essentially all 259,829 examples. The loss converged normally (the model was trivially learning to predict text it was already given in full, i.e. the system prompt prefix), and the run reported no errors. The resulting model passed a superficial capability check (valid JSON, plausible categories) purely because the *base* model already had those capabilities zero-shot — the fine-tune itself had no measurable effect. Root-caused by tokenizing real training examples post-hoc and finding the assistant span started at token ≈2,536, far past the 1,536-token cutoff. **Attempt 2, first sub-issue — GPU device-placement bug (fixed):** early re-runs (with `max_seq_len` corrected) hit repeated out-of-memory errors that looked like a batch-size problem and were initially treated as one (leading to a QLoRA 4-bit detour that "fixed" the symptom without being the actual cause). The real cause: the training script never called `torch.cuda.set_device(local_rank)` before model loading. Because `SFTConfig`/`TrainingArguments` (which normally performs this via Accelerate) is only constructed *after* `from_pretrained()` runs, all 8 DeepSpeed ranks defaulted to the same device and loaded 8 copies of the model onto GPU 0 while the other 7 GPUs sat idle — visible in an OOM trace showing ~59 GB allocated on one GPU by a process that should have used ~5 GB of its own. Fixed with a single explicit `torch.cuda.set_device(local_rank)` call placed before any CUDA/HIP allocation. Once fixed, plain BF16 training (no quantization) ran stably — the QLoRA detour had been solving a problem that no longer existed once each rank owned its own GPU. **Attempt 2, second sub-issue — throughput:** even after the fixes above, eager attention (no Flash Attention 2 on this ROCm build) kept per-step latency around 120–150 s at 8,192-token context. Reducing `max_seq_len` to 4,096 (still comfortably above the measured p99 of 3,628 tokens across the dataset) roughly halved the attention cost without truncating any real example. Even so, throughput meant the full 3-epoch run (6,060 steps) could not fit inside the cluster's 38-hour per-job time limit. The final run was executed as a **chain of 6 SLURM jobs** (`afterany` dependencies, each resuming from the latest checkpoint), spanning **2026-08-01 to 2026-08-10 (≈8 days 17 hours of wall-clock, cumulative queue + compute time)** before the final job completed cleanly within its time limit and wrote the adapter. ## Evaluation **Training-time metrics only — no independent/adversarial evaluation has been run yet against this specific checkpoint.** Treat the numbers below as evidence the fine-tune converged in-distribution, not as a claim about real-world planner quality; a held-out multi-domain benchmark against the pre-fine-tune baseline is planned but not yet executed as of this card's writing. | Metric | Value | Source | |---|---|---| | Final training loss | 0.00017 | last logged training step (step 6,060/6,060, epoch 3.0) | | Held-out eval loss | 0.0235 | 1,300-sample held-out split, epoch ≈2.97 | | Held-out mean token accuracy | 0.9911 | same held-out split | | Held-out mean token accuracy (training split, rolling) | ~0.99 throughout the final epoch | training logs | The gap between the very low final training loss and the higher (but still low) held-out eval loss is consistent with normal LoRA-SFT convergence on a synthetic, template-heavy dataset, not independently confirmed to be free of overfitting on out-of-distribution phrasing — see limitations below. ## Known limitations - **Tied to one specific system prompt.** The model was trained against a fixed, versioned ~2,500-token system prompt with a specific 16-category taxonomy and 22-tool catalog. This is not a hypothetical caveat — a taxonomy/prompt mismatch between training and serving is the documented root cause of a separate, earlier deployment failure in this project. Do not deploy this adapter behind a different system prompt without retraining or at least re-validating. - **No independent evaluation yet.** All numbers above are training/held-out-split metrics from the same synthetic data distribution the model was trained on. No adversarial, out-of-distribution, or human-graded evaluation has been run against this checkpoint at the time of writing. - **Inherits teacher-model biases.** The training data was generated by a large teacher model decomposing queries into the target taxonomy; any systematic blind spots or stylistic biases in the teacher's decompositions carry over. - **Narrow by design.** This is not a chat model and will not behave like one — it is trained to emit only a JSON array, nothing else, and has not been evaluated for any other task. - **Quantized variants trade precision for size.** The Q4_K_M GGUF in this repo (4.90 bits/weight, ~4.7 GB) is the recommended deployment format for constrained VRAM; F16 GGUF and BF16 safetensors are also published for full-precision use or further fine-tuning. ## Files in this repository | File | Format | Size | Use case | |---|---|---|---| | `qwen3-planner-q4_k_m.gguf` | GGUF, Q4_K_M (4.90 BPW) | ~4.7 GB | Recommended for local/edge inference (e.g. Ollama) | | `qwen3-planner-f16.gguf` | GGUF, F16 | ~16 GB | Full-precision GGUF inference | The merged BF16 safetensors checkpoint is published separately at [`h3rb3rn/qwen3-planner-sft`](https://huggingface.co/h3rb3rn/qwen3-planner-sft). ## Acknowledgments Training was performed on **LUMI**, a EuroHPC Joint Undertaking supercomputer hosted by CSC (Finland) and the LUMI consortium, under EuroHPC allocation `project_465003058`. Base model and tokenizer: [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) (Apache 2.0).