Upload README.md with huggingface_hub

README.md

@@ -1,214 +1,179 @@
 ---
-pipeline_tag: text-generation
+library_name: ninfer
 tags:
-  - qwen
-  - ninfer
-  - quantization
-  - nvfp4
-  - blackwell
-  - text-generation
+- ninfer
+- qwen
+- quantized
+- nvfp4
+- blackwell
+- preview
+license: other
 ---
 
-# ThinkingCap-Qwen3.8-27B — NInfer D48 NVFP4
+# ThinkingCap Qwen3.8 27B — NInfer D48+O44 Preview
 
-Experimental **NInfer** conversion of **ThinkingCap-Qwen3.8-27B**, optimized for NVIDIA Blackwell GPUs with a quality-first mixed-precision strategy.
+This repository contains a **preview NInfer quantization candidate** for
+ThinkingCap-Qwen3.8-27B.
 
-The current recommended artifact is the **D48 anchor**.
+It is not yet the final quantization release.
 
-It starts from NInfer's standard `qwen3_8_27b` recipe and adds model-specific NVFP4 quantization to selected MLP Down projections. The D48 layers were selected from a complete 64-layer single-change sensitivity sweep and then validated cumulatively with full perplexity and runtime benchmarks.
+The cumulative D48+O44 performance candidate has passed conversion and
+target validation. A cumulative Full-PPL evaluation is still pending at
+the time of this preview release.
 
 ## Artifact
 
-Place the model file in the repository root with this name:
+`thinkingcap-d48-o44-preview.ninfer`
 
-```text
-thinkingcap-qwen3.8-27b-d48.ninfer
-```
+SHA256:
 
-| Property | Value |
-|---|---:|
-| Model | ThinkingCap-Qwen3.8-27B |
-| Runtime format | NInfer |
-| Variant | D48 |
-| Artifact size | 16,705,529,348 bytes |
-| Runtime weight capacity | 16,692,422,656 bytes |
-| Workspace | 159,981,568 bytes |
-| Target GPU | NVIDIA RTX PRO 4000 Blackwell 24 GB |
-| Base NInfer recipe | `qwen3_8_27b` |
-| Additional optimization | 48 selected MLP Down projections → NVFP4 |
+`db5f12b03ddc48d1776179c8564144325d967f4e5d5d36deb34e9bd70d1b5e17`
 
-The corresponding standard NInfer baseline artifact was **17,106,573,056 bytes**. D48 reduces the artifact by **401,043,708 bytes (~382.5 MiB / 2.34%)**.
+Artifact size:
 
-Artifact size note: This build is not directly size-comparable to the full upstream Qwen3.8 NInfer/NVFP4 artifacts. It was converted with --components text and currently does not bundle Vision, MTP or DFlash2 companion weights. D48 also retains NInfer's compact groupwise allocation for most tensors and selectively replaces only 48 MLP Down projections with NVFP4. As a result, the optimized artifact is smaller than its own 17.11 GB text-only NInfer baseline.
-The smaller size does not imply that this build uses a more aggressive full-model NVFP4 profile. It is a selective mixed-precision configuration optimized around the measured ThinkingCap sensitivity map.
+`16,575,798,788 bytes`
 
-## What D48 means
+Baseline artifact size:
 
-D48 does **not** mean “layers 0–47”. Every one of the 64 MLP Down projections was tested individually. The 48 most tolerant layers were selected for the cumulative D48 build.
+`17,106,573,056 bytes`
 
-### NVFP4 MLP Down layers
+Reduction versus baseline:
 
-```text
-0, 1, 4, 5, 6, 7, 9,
-13, 14, 15, 16, 17, 19, 20, 22, 23, 24,
-28, 29, 30, 31,
-33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
-53, 54, 55, 56, 58, 59, 60, 61, 62
-```
+`530,774,268 bytes` (~506.2 MiB, ~3.10%)
 
-### Protected MLP Down layers
+## Quantization layout
 
-```text
-2, 3, 8, 10, 11, 12, 18, 21,
-25, 26, 27, 32,
-51, 52, 57, 63
-```
+The candidate was converted directly from the original BF16/FP16 source.
+It was not requantized from another `.ninfer` artifact.
 
-These remain at the precision selected by the standard NInfer recipe.
+It combines:
 
-## Quantization strategy
+- **D48**: 48 selected MLP Down projections converted to NVFP4.
+- **O44**: 44 selected output projections converted to NVFP4.
+  - 35 GDN output projections
+  - 9 Attention output projections
 
-New artifacts are generated directly from the original BF16/FP16 ThinkingCap checkpoint. Existing `.ninfer` files are **not** requantized.
+Total validated NVFP4 conversion jobs:
 
-```text
-ThinkingCap BF16/FP16 Safetensors
-        │
-        ▼
-NInfer qwen3_8_27b recipe
-        │
-        ▼
-standard mixed-precision NInfer baseline
-        │
-        ▼
-selected MLP Down projections → NVFP4
-        │
-        ▼
-ThinkingCap D48 NInfer
-```
+**92**
 
-D48 is still a mixed-precision model. Only the selected MLP Down projections receive the additional NVFP4 override; all other tensors continue to use the formats chosen by the standard NInfer recipe.
+### D48 MLP Down layers
 
-## Full perplexity validation
+`0,1,4,5,6,7,9,13,14,15,16,17,19,20,22,23,24,28,29,30,31,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,53,54,55,56,58,59,60,61,62`
 
-Development corpus configuration:
+### O44 GDN Output layers
 
-```text
-context:        4096
-stride:         2048
-KV cache:       BF16
-scored tokens:  782,535
-streams:        12
-```
+`0,5,9,12,13,14,16,17,18,20,22,28,32,33,34,36,37,40,41,42,44,45,46,48,49,50,52,53,54,56,57,58,60,61,62`
 
-The corpus contains English long-form text, English reference text, and code. It is a **development/sensitivity corpus**, not an untouched final holdout.
+### O44 Attention Output layers
 
-| Domain | Standard NInfer baseline | D48 |
-|---|---:|---:|
-| English long-form | 7.918420 | 7.944685 |
-| English reference | 6.264029 | 6.219786 |
-| Code | 1.651905 | 1.654696 |
-| **Overall** | **4.351060** | **4.348019** |
+`11,19,27,31,39,51,55,59,63`
 
-Overall D48 delta versus the baseline is approximately **−0.070% PPL** on this development corpus.
+## Size impact
 
-The negative aggregate delta must not be interpreted as evidence that quantization improves the underlying model. The individual domains move in different directions; the result is best read as **no measurable aggregate quality loss on this development corpus at the tested resolution**.
+D48 artifact:
 
-## Runtime benchmark
+`16,705,529,348 bytes`
 
-Measured on an **NVIDIA RTX PRO 4000 Blackwell 24 GB** with FP8 KV cache, prefill chunk 1024, 2 warmups and 5 measured repetitions.
+D48+O44 artifact:
 
-| Test | Throughput |
-|---|---:|
-| PP512 | 1293.37 tok/s |
-| PP2048 | 1333.02 tok/s |
-| PP8192 | 1310.12 tok/s |
-| TG256 | 32.46 tok/s |
-| TG1024 | 32.03 tok/s |
-| PP2048 + TG512 — prefill | 1323.95 tok/s |
-| PP2048 + TG512 — decode | 31.92 tok/s |
-| PP8192 + TG512 — prefill | 1297.68 tok/s |
-| PP8192 + TG512 — decode | 31.43 tok/s |
+`16,575,798,788 bytes`
 
-These values are intended for comparisons within the same NInfer/hardware/software setup. They should not be treated as directly comparable to unrelated runtimes or benchmark methodologies.
+O44 therefore removes another:
 
-## Why D48 is the current anchor
+`129,730,560 bytes` (~123.7 MiB)
 
-More aggressive cumulative candidates were also tested:
+from the D48 candidate.
 
-| Variant | Extra NVFP4 Down layers | Full PPL | Artifact size |
+## Performance
+
+The meaningful comparison below uses the same physical GPU0.
+
+| Test | D48 | D48+O44 | Change |
 |---|---:|---:|---:|
-| Standard baseline | 0 | 4.351060 | 17,106,573,056 B |
-| **D48** | **48** | **4.348019** | **16,705,529,348 B** |
-| D50 | 50 | 4.350115 | 16,688,818,692 B |
-| D53 | 53 | 4.355314 | 16,663,752,708 B |
+| pp512 prefill | 774.13 tok/s | 840.81 tok/s | +8.61% |
+| pp2048 prefill | 782.36 tok/s | 850.04 tok/s | +8.65% |
+| pp8192 prefill | 746.28 tok/s | 790.06 tok/s | +5.87% |
+| tg256 decode | 18.74 tok/s | 19.06 tok/s | +1.68% |
+| tg1024 decode | 18.28 tok/s | 18.72 tok/s | +2.44% |
+| pp2048+tg512 prefill | 735.53 tok/s | 765.80 tok/s | +4.11% |
+| pp2048+tg512 decode | 18.53 tok/s | 18.70 tok/s | +0.94% |
+| pp8192+tg512 prefill | 723.44 tok/s | 754.14 tok/s | +4.24% |
+| pp8192+tg512 decode | 18.31 tok/s | 18.52 tok/s | +1.16% |
+
+The D48 control used 2 warmups and 3 measured repetitions.
+The D48+O44 run used 2 warmups and 5 measured repetitions.
+
+The result is therefore suitable as a performance screening measurement,
+not as the final release benchmark.
+
+Approximate mean change over the measured cases:
 
-D50 was only about **16 MiB smaller** than D48 and measured roughly **+0.31% mean prefill / −0.12% mean decode** versus D48 in the same benchmark setup. D53 produced a clearer prefill gain, but also a clearer perplexity regression.
+- Prefill: **+6.3%**
+- Decode: **+1.6%**
 
-For the current **quality-first** target, D48 is therefore the working Down anchor.
+## Quality status
 
-## Current optimization status
+The individual sensitivity sweeps used a 782,535-token English/code
+development corpus with:
 
-```text
-1. Standard NInfer conversion
-2. Full 64-layer MLP Down sensitivity sweep
-3. D48 quality-first Down anchor        ← this release
-4. ThinkingCap-specific IMatrix
-5. Gate / Up sensitivity analysis
-6. Combined mixed-precision optimization
-7. Final task-level quality validation
-8. MTP and/or DFlash2 integration
-```
+- context: 4096
+- stride: 2048
+- KV cache: BF16
 
-### Not yet included
+D48 previously showed no measurable degradation on this development
+corpus.
 
-This D48 artifact does **not** yet include:
+The individual O44 projections were selected from the completed Output64
+sensitivity map using the `ΔNLL <= 0.0002` boundary.
 
-- ThinkingCap-specific IMatrix optimization
-- Agentic-workload importance calibration
-- dedicated Gate optimization
-- dedicated Up optimization
-- final Gate/Up + Down joint optimization
-- MTP speculative decoding model
-- DFlash2 speculative decoding model
-- final task-level evaluation suite
+**Important:** individual layer deltas are not assumed to add linearly.
 
-## Planned IMatrix work
+The cumulative D48+O44 Full-PPL measurement is still pending for this
+preview. This artifact should therefore be treated as an experimental
+performance candidate rather than the final quality-validated release.
 
-A ThinkingCap-specific importance matrix is planned from the original BF16/FP16 weights using an agentic/code-oriented calibration corpus.
+## Runtime benchmark configuration
 
-The IMatrix will be used to prioritize Gate/Up candidates, identify important tensors/channels, reduce the search space, and cross-check the existing empirical sensitivity results. It will **not** replace direct single-change and cumulative-group quality measurements.
+The throughput benchmark uses the public NInfer `ninfer_bench` route with:
 
-## Reproducibility
+- KV cache: FP8 E4M3 row256
+- maximum context: 16384
+- prefill chunk: 1024
+- prefill tests: 512, 2048, 8192 tokens
+- decode tests: 256, 1024 tokens
+- combined tests:
+  - 2048 prompt + 512 generation
+  - 8192 prompt + 512 generation
 
-Converter checkout base commit:
+Raw JSON benchmark reports are included in `benchmarks/`.
 
-```text
-9e163eee4b8acec21ab0ac765107b6a3f287b217
-```
+## Tokenizer and chat resources
 
-The local build also contains experimental changes required for the NVFP4 path, so exact compatibility can depend on the corresponding NInfer runtime implementation.
+Tokenizer and chat-template resources are copied from the original
+ThinkingCap-Qwen3.8-27B source directory.
 
-See `PROVENANCE.md`, `quantization/d48-layers.txt`, and `benchmarks/` for compact experiment metadata.
+They are provided for use with the NInfer artifact and do not imply that
+this repository contains Hugging Face Transformers weights.
 
-## Intended use
+## Provenance
 
-This release is primarily intended for:
+The conversion was performed from the original BF16/FP16 source using a
+local NInfer development checkout.
 
-- ThinkingCap-Qwen3.8-27B experiments with NInfer
-- NVIDIA Blackwell inference
-- mixed-precision / NVFP4 research
-- quality/performance trade-off analysis
-- further model-specific optimization work
+The conversion metadata, artifact checksum and available build provenance
+are included with this preview.
 
-The optimization priority is:
+## Status
 
-```text
-1. Preserve model quality
-2. Improve real inference throughput
-3. Reduce persistent model/VRAM footprint
-4. Keep experiments reproducible
-```
+**Preview / experimental**
 
-## License and upstream terms
+Planned before the final release:
 
-This repository does not redefine the licenses of the upstream ThinkingCap model, its base model, NInfer, or NVIDIA components. Users should review and comply with the applicable upstream licenses and terms before redistribution or use.
+1. cumulative D48+O44 Full-PPL
+2. completed Input64 sensitivity analysis
+3. cumulative Down + Output + Input candidate
+4. Full-PPL validation
+5. final Prefill/Decode benchmark
+6. frozen final quantization anchor
+7. MTP work after the final quantization anchor