Update README.md
Browse files
README.md
CHANGED
|
@@ -9,89 +9,40 @@ tags:
|
|
| 9 |
- Ninfer
|
| 10 |
- ThinkingCap
|
| 11 |
- Qwen3.8
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
-
# ThinkingCap-Qwen3.8-27B NInfer
|
| 14 |
|
| 15 |
-
|
| 16 |
|
| 17 |
-
This repository contains
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
* Native NInfer artifact
|
| 24 |
-
* Qwen3.8-27B ThinkingCap
|
| 25 |
-
* MTP (Multi-Token Prediction) support
|
| 26 |
-
* Long-context capable
|
| 27 |
-
* Optimized for NVIDIA Blackwell GPUs
|
| 28 |
-
* Compatible with the current NInfer runtime
|
| 29 |
-
|
| 30 |
-
---
|
| 31 |
-
|
| 32 |
-
## Download
|
| 33 |
-
|
| 34 |
-
Download the complete repository:
|
| 35 |
|
| 36 |
-
|
| 37 |
-
hf download Schestex/ThinkingCap-Qwen3.8-27B-NInfer \
|
| 38 |
-
--local-dir ./ThinkingCap-Qwen3.8-27B-NInfer
|
| 39 |
-
```
|
| 40 |
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
```bash
|
| 44 |
-
hf download Schestex/ThinkingCap-Qwen3.8-27B-NInfer \
|
| 45 |
-
thinkingcap_qwen3_8_27b.ninfer \
|
| 46 |
-
--local-dir models
|
| 47 |
-
```
|
| 48 |
-
|
| 49 |
-
---
|
| 50 |
-
|
| 51 |
-
## Example
|
| 52 |
-
|
| 53 |
-
```bash
|
| 54 |
-
./build/apps/ninfer \
|
| 55 |
-
models/thinkingcap_qwen3_8_27b.ninfer \
|
| 56 |
-
--prompt "Explain speculative decoding." \
|
| 57 |
-
--max-context 32768 \
|
| 58 |
-
--max-new 1024 \
|
| 59 |
-
--spec mtp \
|
| 60 |
-
--draft-tokens 3
|
| 61 |
-
```
|
| 62 |
-
|
| 63 |
-
---
|
| 64 |
|
| 65 |
-
|
| 66 |
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
|
|
|
|
|
|
| 70 |
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
## About ThinkingCap
|
| 74 |
-
|
| 75 |
-
ThinkingCap is a reasoning-focused variant of Qwen3.8-27B designed for high-quality assistant workloads.
|
| 76 |
-
|
| 77 |
-
This repository provides the model packaged in the native NInfer artifact format for efficient local inference.
|
| 78 |
-
|
| 79 |
-
---
|
| 80 |
-
|
| 81 |
-
## Credits
|
| 82 |
-
|
| 83 |
-
Base model:
|
| 84 |
-
|
| 85 |
-
* bottlecapai/ThinkingCap-Qwen3.8-27B
|
| 86 |
-
|
| 87 |
-
Runtime and artifact format:
|
| 88 |
-
|
| 89 |
-
* NInfer
|
| 90 |
-
|
| 91 |
-
---
|
| 92 |
|
| 93 |
-
|
| 94 |
|
| 95 |
-
|
| 96 |
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
- Ninfer
|
| 10 |
- ThinkingCap
|
| 11 |
- Qwen3.8
|
| 12 |
+
- quantization
|
| 13 |
+
- text-generation
|
| 14 |
---
|
| 15 |
+
# ThinkingCap-Qwen3.8-27B — NInfer
|
| 16 |
|
| 17 |
+
NInfer conversions of **ThinkingCap-Qwen3.8-27B**, optimized for NVIDIA Blackwell GPUs.
|
| 18 |
|
| 19 |
+
This repository currently contains two target-model variants:
|
| 20 |
|
| 21 |
+
| Variant | File | Size | Description |
|
| 22 |
+
|---|---|---:|---|
|
| 23 |
+
| Baseline | `thinkingcap-qwen3.8-27b.ninfer` | 17,106,573,056 bytes | Standard NInfer `qwen3_8_27b` conversion |
|
| 24 |
+
| **D48** | `thinkingcap-qwen3.8-27b-d48.ninfer` | **16,705,529,348 bytes** | Quality-first optimized variant with 48 selected MLP Down projections converted to NVFP4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
## Recommended variant
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
+
For current use, **D48 is the recommended quality-first optimized anchor**.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
Compared with the standard NInfer conversion, D48:
|
| 31 |
|
| 32 |
+
- reduces the artifact by approximately **401 MB**
|
| 33 |
+
- reduces model size by approximately **2.34%**
|
| 34 |
+
- improves prefill throughput substantially relative to the unoptimized baseline architecture
|
| 35 |
+
- shows no measurable overall perplexity regression on the development corpus used during optimization
|
| 36 |
+
- keeps the most sensitive MLP Down layers at the precision chosen by the standard NInfer recipe
|
| 37 |
|
| 38 |
+
The optimization is deliberately conservative.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
+
The goal is not minimum file size.
|
| 41 |
|
| 42 |
+
The priority is:
|
| 43 |
|
| 44 |
+
```text
|
| 45 |
+
1. Preserve model quality
|
| 46 |
+
2. Improve inference throughput
|
| 47 |
+
3. Reduce VRAM / model footprint
|
| 48 |
+
4. Keep the configuration reproducible
|