MTP Hybrid-A1 Controlled OLD W8-MTP vs Hybrid-A1 benchmarks
MTP Hybrid-A1
qwen3_8_27b_coldfusion_nvfp4_mtp-a1-s256-seed38027.ninfer is an experimental MTP memory-optimized variant of the ColdFusion NVFP4 NInfer artifact.
Hybrid-A1 changes only the MTP MLP weights:
mtp/layer/mlp/gate_up: W8 → NVFP4mtp/layer/mlp/down: W8 → NVFP4
The main model weights are unchanged.
The conversion uses calibrated NVFP4 input and weight scale divisors derived from 256 agentic coding calibration samples / 2,064,282 MTP calibration pairs.
Memory result
| Artifact | Result |
|---|---|
| Original no-vision artifact | 23,423,727,616 bytes |
| MTP Hybrid-A1 artifact | 23,290,035,200 bytes |
| Artifact reduction | 133,692,416 bytes / 127.499 MiB |
| Old W8-MTP max verified KV capacity | 120,704 tokens |
| Hybrid-A1 verified KV capacity | 123,904 tokens |
| KV capacity gain | +3,200 tokens / +2.65% |
Tested with k8v4, MTP enabled, one concurrent sequence on an NVIDIA RTX PRO 4000 Blackwell 24 GB.
Controlled OLD W8-MTP vs Hybrid-A1 benchmarks Blackwell SSF!
Same upstream NInfer build, same prompts, same GPU, k8v4, max-context=32904, kv-capacity=32904.
| Benchmark | OLD W8-MTP | Hybrid-A1 |
|---|---|---|
| Code / MTP3 | 42.624 tok/s / 83.029% acceptance | 43.562 tok/s / 86.247% |
| Code / MTP4 | 52.104 tok/s / 74.514% acceptance | 55.761 tok/s / 80.351% |
| Text / MTP3 | 41.340 tok/s / 77.567% acceptance | 42.865 tok/s / 81.680% |
| Text / MTP4 | 49.664 tok/s / 67.492% acceptance | 48.416 tok/s / 64.849% |
MTP3 is the recommended configuration for this artifact. It improved both measured code and text throughput while also improving MTP acceptance versus the original W8-MTP implementation.
Runtime requirement
Hybrid-A1 currently requires the included NInfer MTP Hybrid-A1 runtime patch. Unpatched upstream NInfer does not understand the additional calibrated NVFP4 MTP MLP objects used by this artifact.
This is an experimental NInfer artifact, not a Transformers, GGUF, or Safetensors model distribution.
Source of ninfer changes: https://github.com/Xtravaganz/ninfer/tree/mtp-hybrid-a1
Update — v3 is now stable on upstream NInfer
Both the NoVision and MTP Hybrid-A1 artifacts have now been updated to v3 and are running stable with the current upstream NInfer implementation from Neroued/ninfer.
The previously referenced custom Hybrid-A1 runtime branch is no longer required. The additional NVFP4 MTP MLP objects used by Hybrid-A1 are now supported by Neroued's upstream implementation, so there is no need to use my separate NInfer runtime/compatibility branch anymore.
Verified runtime state
Repository: Neroued/ninfer
Branch: master
Commit: 9e163eee4b8acec21ab0ac765107b6a3f287b217
Message: perf(ops): route the q4/q5 a16 input projections by column band
Commit:
https://github.com/Neroued/ninfer/commit/9e163eee4b8acec21ab0ac765107b6a3f287b217
This commit is the known-good reference point for the current NoVision and Hybrid-A1 v3 artifacts. Newer upstream revisions may work as well, but the commit above is the exact upstream state against which the current v3 builds were verified.
Qwen3.8 Sharp chat-template test
I also maintain a small test branch for running the same upstream runtime with the Qwen3.8 Sharp v22.5.0 chat template:
https://github.com/Neroued/ninfer/compare/master...Xtravaganz:ninfer:neroued-v3-sharp
At the time of verification, this branch is based directly on the known-good Neroued master above and is:
0 commits behind
2 commits ahead
The two additional commits are:
539bbe4f1f468e59c3e1e7373f05e10c7b382888
feat(chat): add Qwen3.8 Sharp v22.5.0 template
54eb0cd47daec4b42ba094b933773cf16e7f2772
build(docker): include Qwen3.8 Sharp chat template
The comparison only adds:
tools/chat_templates/qwen3_8_sharp.jinja
and makes that template available inside the Docker image.
There are no additional NInfer engine, CUDA kernel, model-loading, MTP, or inference-runtime changes in this branch. This makes it useful as a clean test of the Sharp chat template while keeping the actual inference runtime identical to the verified Neroued upstream state.
So for normal use, Neroued/ninfer master is sufficient.
The neroued-v3-sharp branch is only needed if you specifically want to test/use the bundled Qwen3.8 Sharp chat template.
Thanks to Neroued for bringing the required model/runtime support upstream and making the previous separate compatibility runtime unnecessary.
IMatrix is in progress!