Update README.md

README.md

@@ -15,7 +15,7 @@ tags:
 - speculative-decoding
 - multi-token-prediction
 model-index:
-  - name: Ornith-1.0-35B-oQ7-mtp
+  - name: Ornith-1.0-35B-oQ7
     results:
       - task:
           type: text-generation
@@ -135,34 +135,32 @@ model-index:
             type: accuracy
             value: 86.7
 ---
-
-# Ornith-1.0-35B-oQ7-mtp
-
-MLX format quantization of [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B), produced with mlx-optiq and shipped with a grafted Multi-Token Prediction (MTP) head for speculative decoding on Apple Silicon.
-
+ 
+# Ornith-1.0-35B-oQ7
+ 
+MLX format quantization of [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B), produced with mlx-optiq. This repository also ships a grafted Multi-Token Prediction (MTP) head (`mtp.safetensors`) for speculative decoding on Apple Silicon, sourced from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`. **Testing found that this MTP head does not currently work correctly with this oQ7 mixed-precision quantization**, which is why this repository was renamed from `Ornith-1.0-35B-oQ7-mtp` to `Ornith-1.0-35B-oQ7`. The `mtp.safetensors` file has not been removed and remains in the repo for future compatibility work — see [Multi-Token Prediction](#multi-token-prediction-mtp) below for details.
+ 
 ## Model Details
-
+ 
 ### Model Description
-
+ 
 This repository contains a mixed precision MLX quantization of Ornith-1.0-35B, a Qwen3.5-35B-A3B based mixture of experts model. The quantization was produced with mlx-optiq, which performs a per-layer KL sensitivity analysis against the original BF16 checkpoint and assigns each layer either 6-bit or 8-bit precision so that the weighted average lands at a target of 7.5 bits per weight (BPW). Layers that are more sensitive to quantization error are kept at 8-bit, while more robust layers are reduced to 6-bit.
-
-An auxiliary MTP head, grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`, is included as `mtp.safetensors`. It is used as a self-speculative draft model during decoding and is architecturally compatible because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share the same hidden size, layer count, expert count, and block layout, and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.
-
+ 
+An auxiliary MTP head, grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`, is included as `mtp.safetensors`. It is architecturally compatible with the base model because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share the same hidden size, layer count, expert count, and block layout, and Ornith is fine-tuned directly from Qwen3.5-35B-A3B. However, testing concluded that this head does not work correctly as a self-speculative draft model with the oQ7 mixed-precision quantization in this repository, so it should not currently be enabled for inference. The file is kept in the repo for future compatibility work (see the Multi-Token Prediction section below).
+ 
 - **Developed by:** programmer-666
 - **Model type:** Causal decoder only mixture of experts language model (Qwen3.5-35B-A3B architecture, qwen35moe, 40 layers)
 - **Language(s):** Inherited from the base model
 - **License:** Apache 2.0 for this repository. See the Licenses section below for the licenses of the underlying components.
 - **Quantized from model:** [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)
 - **MTP head source:** [mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit](https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit)
-
 ### Model Sources
-
+ 
 - **Base model:** https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B
 - **Quantization tool:** https://mlx-optiq.com
 - **Serving engine used for benchmarks:** https://github.com/jundot/omlx
-
 ## Model Properties
-
+ 
 | Property | Value |
 |---|---|
 | Base model | deepreinforce-ai/Ornith-1.0-35B |
@@ -176,56 +174,55 @@ An auxiliary MTP head, grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`,
 | Layers at 8-bit | 362 |
 | Group size | 32 |
 | Calibration sequences | 40 |
-| MTP head source | mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit |
+| MTP head source | mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit (included, but not functional with this quantization — see [Multi-Token Prediction](#multi-token-prediction-mtp)) |
 | Format | MLX safetensors |
-
+ 
 ## Uses
-
+ 
 ### Direct Use
-
-This model is intended for local text generation on Apple Silicon devices using MLX based inference engines such as `mlx-lm` or `optiq serve`. It is suited for users who want most of the quality of the BF16 checkpoint at roughly half the memory footprint, with an optional speculative decoding path for faster generation.
-
+ 
+This model is intended for local text generation on Apple Silicon devices using MLX based inference engines such as `mlx-lm` or `optiq serve`. It is suited for users who want most of the quality of the BF16 checkpoint at roughly half the memory footprint. Note: the bundled MTP head does not currently provide working speculative decoding with this quantization (see [Multi-Token Prediction](#multi-token-prediction-mtp)).
+ 
 ### Out-of-Scope Use
-
+ 
 This is a quantized derivative of a third party base model. It has not been independently evaluated for safety, factuality, or fitness for any particular downstream task. It should not be used in high stakes settings (medical, legal, financial, or safety critical decisions) without additional evaluation. Refer to the base model card for its intended use cases and known limitations, since these are inherited by this quantization.
-
+ 
 ## Bias, Risks, and Limitations
-
+ 
 Quantization can shift a model's behavior relative to the original weights, even when overall benchmark scores are similar. Mixed precision quantization at 7.5 BPW is expected to be close to BF16 quality, but no independent evaluation of downstream task accuracy, factuality, or bias has been performed for this specific quantized artifact. Users should treat outputs as they would from the base model and are encouraged to run their own evaluations for their use case before deploying it in production.
-
+ 
 The intelligence benchmark results reported below use 30-question samples from each dataset and should be treated as indicative rather than definitive. Full dataset evaluations may yield different figures.
-
+ 
 ## How to Get Started with the Model
-
-### With mlx-optiq serve (enables MTP speculative decoding)
-
+ 
+### With mlx-optiq serve
+ 
 ```bash
 optiq serve \
-  --model programmer-666/Ornith-1.0-35B-oQ7-mtp \
-  --mtp \
+  --model programmer-666/Ornith-1.0-35B-oQ7 \
   --port 8080
 ```
-
+ 
 ### With mlx-lm
-
+ 
 ```bash
 mlx_lm.generate \
-  --model programmer-666/Ornith-1.0-35B-oQ7-mtp \
+  --model programmer-666/Ornith-1.0-35B-oQ7 \
   --prompt "Your prompt here"
 ```
-
-Note: MTP is not available through `mlx_lm.generate`. Use `optiq serve` or `omlx` if you want speculative decoding.
-
+ 
+> **Note on MTP:** This repository still contains a grafted MTP head (`mtp.safetensors`), but testing found it does not work correctly as a speculative decoding draft model with this oQ7 mixed-precision quantization. Do not pass a `--mtp` flag to `optiq serve` for this model. The file is retained for future compatibility work — see [Multi-Token Prediction](#multi-token-prediction-mtp) below for details.
+ 
 ## Training Details
-
+ 
 This repository does not modify the base model's weights beyond quantization; no additional fine-tuning was performed. For training data and training procedure, refer to the [base model card](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).
-
+ 
 ### Quantization Procedure
-
+ 
 Quantization was performed with mlx-optiq, which runs a per-layer KL sensitivity analysis to assign bit widths. Layers with higher sensitivity to quantization error retain 8-bit precision, while more robust layers are assigned 6-bit precision. The reference model used during sensitivity calibration was the original BF16 checkpoint.
-
+ 
 Conversion command:
-
+ 
 ```bash
 optiq convert deepreinforce-ai/Ornith-1.0-35B \
   --candidate-bits 6,8 \
@@ -236,37 +233,38 @@ optiq convert deepreinforce-ai/Ornith-1.0-35B \
   --skip-baselines \
   -o Ornith-1.0-35B-oQ7-mtp
 ```
-
+ 
 ## Multi-Token Prediction (MTP)
-
-The `mtp.safetensors` file contains an auxiliary prediction head grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`. This works because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share an identical structure (hidden dimension 2048, 40 layers, 256 experts, same block layout), and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.
-
-MTP uses this auxiliary head as a draft model for speculative decoding, giving roughly a 1.3x to 1.4x decode speedup on Apple Silicon during greedy generation.
-
+ 
+The `mtp.safetensors` file contains an auxiliary prediction head grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`. Architecturally, the graft is sound: Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share an identical structure (hidden dimension 2048, 40 layers, 256 experts, same block layout), and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.
+ 
+**Known issue — MTP does not work with this oQ7 quantization.** In principle, MTP uses this auxiliary head as a draft model for speculative decoding, giving roughly a 1.3x to 1.4x decode speedup on Apple Silicon during greedy generation. In testing, however, the head did not pair correctly with the oQ7 mixed-precision weights in this repository, so that speedup is not realized here and speculative decoding should not be enabled (e.g. do not pass `--mtp` to `optiq serve`).
+ 
+Because of this finding, the repository was renamed from `Ornith-1.0-35B-oQ7-mtp` to `Ornith-1.0-35B-oQ7` to avoid implying working speculative decoding support. The `mtp.safetensors` file itself has not been deleted and remains in the repo for future compatibility work.
+ 
 ## Hardware Requirements
-
+ 
 | Configuration | Notes |
 |---|---|
 | Recommended | Apple Silicon with 64GB or more of unified memory |
 | Tested on | M4 Max MacBook Pro, 128GB |
 | Disk space | Approximately 37GB |
-
+ 
 ## Evaluation
-
+ 
 ### Testing Setup
-
+ 
 All benchmarks were run locally on an M4 Max MacBook Pro (128GB unified memory) using [oMLX](https://github.com/jundot/omlx), an LLM inference engine optimized for Apple Silicon. Three models were evaluated side by side:
-
-- `Ornith-1.0-35B-oQ7-mtp` (this repository)
+ 
+- `Ornith-1.0-35B-oQ7` (this repository)
 - `Ornith-1.0-35B-bf16` (original BF16 checkpoint, reference)
 - `Qwen3.6-35B-A3B-OptiQ-4bit` (MTP head source, shown for context)
-
 Performance figures come from single request runs unless noted as continuous batching. Intelligence benchmarks were run with thinking mode enabled and use 30-question samples drawn randomly from each dataset; they are indicative and should not be compared directly to full-dataset evaluations published elsewhere.
-
+ 
 ### Intelligence Benchmarks
-
-All results are for this model (`Ornith-1.0-35B-oQ7-mtp`) with thinking enabled.
-
+ 
+All results are for this model (`Ornith-1.0-35B-oQ7`) with thinking enabled.
+ 
 | Benchmark | Sampled | Correct | Accuracy | Time (s) |
 |---|---|---|---|---|
 | MMLU | 30 / 14042 | 26 / 30 | 86.7% | 735.6 |
@@ -282,13 +280,13 @@ All results are for this model (`Ornith-1.0-35B-oQ7-mtp`) with thinking enabled.
 | LiveCodeBench | 30 / 1055 | 19 / 30 | 63.3% | 4501.8 |
 | BBQ | 30 / 10864 | 28 / 30 | 93.3% | 233.9 |
 | SafetyBench | 30 / 11435 | 26 / 30 | 86.7% | 270.2 |
-
+ 
 These results represent 30-question random samples from each dataset evaluated with thinking mode enabled. Sample-based scores carry higher variance than full dataset evaluations and may not reflect performance on the full benchmark. Results are not directly comparable to published leaderboard figures which typically use the full dataset without thinking mode.
-
+ 
 ### Performance Benchmarks
-
-**Ornith-1.0-35B-oQ7-mtp (this repository)**
-
+ 
+**Ornith-1.0-35B-oQ7 (this repository)**
+ 
 | Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
 |---|---|---|---|---|---|---|---|
 | pp1024/tg128 | 787.3 | 10.70 | 1300.6 tok/s | 94.2 tok/s | 2.147 | 536.6 tok/s | 37.36 GB |
@@ -299,17 +297,17 @@ These results represent 30-question random samples from each dataset evaluated w
 | pp65536/tg128 | 94152.4 | 25.95 | 696.1 tok/s | 38.8 tok/s | 97.449 | 673.8 tok/s | 43.13 GB |
 | pp131072/tg128 | 348440.0 | 30.93 | 376.2 tok/s | 32.6 tok/s | 352.368 | 372.3 tok/s | 48.50 GB |
 | pp200000/tg128 | 726434.2 | 36.19 | 275.3 tok/s | 27.9 tok/s | 731.030 | 273.8 tok/s | 54.17 GB |
-
+ 
 Continuous batching, pp1024/tg128:
-
+ 
 | Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT (ms) | E2E (s) |
 |---|---|---|---|---|---|---|
 | 1x | 94.2 tok/s | 1.00x | 1300.6 tok/s | 1300.6 tok/s | 787.3 | 2.147 |
 | 2x | 110.1 tok/s | 1.17x | 549.5 tok/s | 274.8 tok/s | 3726.8 | 6.052 |
 | 4x | 125.5 tok/s | 1.33x | 826.0 tok/s | 206.5 tok/s | 4836.1 | 9.038 |
-
+ 
 **Ornith-1.0-35B-bf16 (reference)**
-
+ 
 | Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
 |---|---|---|---|---|---|---|---|
 | pp1024/tg128 | 834.6 | 16.17 | 1227.0 tok/s | 62.3 tok/s | 2.888 | 398.9 tok/s | 65.62 GB |
@@ -320,17 +318,17 @@ Continuous batching, pp1024/tg128:
 | pp65536/tg128 | 85199.7 | 30.94 | 769.2 tok/s | 32.6 tok/s | 89.130 | 736.7 tok/s | 71.35 GB |
 | pp131072/tg128 | 346498.2 | 36.68 | 378.3 tok/s | 27.5 tok/s | 351.157 | 373.6 tok/s | 76.79 GB |
 | pp200000/tg128 | 630686.4 | 40.49 | 317.1 tok/s | 24.9 tok/s | 635.829 | 314.8 tok/s | 82.47 GB |
-
+ 
 Continuous batching, pp1024/tg128:
-
+ 
 | Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT (ms) | E2E (s) |
 |---|---|---|---|---|---|---|
 | 1x | 62.3 tok/s | 1.00x | 1227.0 tok/s | 1227.0 tok/s | 834.6 | 2.888 |
 | 2x | 43.1 tok/s | 0.69x | 511.9 tok/s | 255.9 tok/s | 4000.4 | 9.941 |
 | 4x | 71.0 tok/s | 1.14x | 938.8 tok/s | 234.7 tok/s | 4212.9 | 11.578 |
-
+ 
 **Qwen3.6-35B-A3B-OptiQ-4bit (MTP head source, shown for reference)**
-
+ 
 | Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
 |---|---|---|---|---|---|---|---|
 | pp1024/tg128 | 765.7 | 9.03 | 1337.3 tok/s | 111.6 tok/s | 1.913 | 602.3 tok/s | 21.72 GB |
@@ -341,42 +339,41 @@ Continuous batching, pp1024/tg128:
 | pp65536/tg128 | 87223.7 | 18.90 | 751.4 tok/s | 53.3 tok/s | 89.624 | 732.7 tok/s | 27.48 GB |
 | pp131072/tg128 | 343907.5 | 27.19 | 381.1 tok/s | 37.1 tok/s | 347.361 | 377.7 tok/s | 32.86 GB |
 | pp200000/tg128 | 675070.1 | 32.83 | 296.3 tok/s | 30.7 tok/s | 679.240 | 294.6 tok/s | 38.52 GB |
-
+ 
 Continuous batching, pp1024/tg128:
-
+ 
 | Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT (ms) | E2E (s) |
 |---|---|---|---|---|---|---|
 | 1x | 111.6 tok/s | 1.00x | 1337.3 tok/s | 1337.3 tok/s | 765.7 | 1.913 |
 | 2x | 148.6 tok/s | 1.33x | 569.3 tok/s | 284.6 tok/s | 3597.4 | 5.320 |
 | 4x | 161.0 tok/s | 1.44x | 873.2 tok/s | 218.3 tok/s | 4564.0 | 7.870 |
-
+ 
 ### Summary
-
+ 
 Compared to the BF16 checkpoint, this quantization roughly halves peak memory usage (37 to 54 GB versus 66 to 82 GB depending on context length) while producing 51% faster token generation at standard context lengths (94.2 tok/s versus 62.3 tok/s at pp1024/tg128). The BF16 model's continuous batching throughput degrades at batch size 2 (0.69x), while this quantization scales consistently to 1.33x at batch size 4. The 4-bit Qwen3.6-35B-A3B-OptiQ-4bit model is faster and lighter on its own; it is included here as the MTP draft head source and as a context point for the tradeoff between model size and throughput.
-
+ 
 ## Licenses
-
+ 
 - Ornith-1.0-35B (base model): MIT License
 - Qwen3.5-35B-A3B (base architecture): Apache 2.0
 - Qwen3.6-35B-A3B (MTP head source): Apache 2.0
 - This repository (quantized weights and MTP head): Apache 2.0
-
 Users should review the license terms of each underlying component before use or redistribution.
-
+ 
 ## Citation
-
+ 
 If you use this model, please cite the original Ornith-1.0-35B model and the mlx-optiq quantization tool.
-
+ 
 ```
-@misc{ornith-1.0-35b-oq7-mtp,
-  title = {Ornith-1.0-35B-oQ7-mtp},
+@misc{ornith-1.0-35b-oq7,
+  title = {Ornith-1.0-35B-oQ7},
   author = {programmer-666},
   year = {2026},
-  note = {MLX mixed precision quantization of deepreinforce-ai/Ornith-1.0-35B with grafted MTP head},
-  howpublished = {\url{https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ7-mtp}}
+  note = {MLX mixed precision quantization of deepreinforce-ai/Ornith-1.0-35B; includes a grafted MTP head retained for future compatibility work, not currently functional with this quantization},
+  howpublished = {\url{https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ7}}
 }
 ```
-
+ 
 ## Model Card Contact
-
+ 
 For questions about this quantization, open a discussion on this repository's Community tab. For questions about the base model, refer to [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).