24 GB optimized NVFP4 NInfer artifact
I'm currently running Qwen3.8-27B with NInfer on 2× RTX PRO 4000 Blackwell, 24 GB each, with one NInfer worker per GPU.
The current NVFP4 artifact works very well, but on a 24 GB card the available VRAM headroom is naturally quite limited.
Would it be technically possible to create a slightly smaller / more aggressively quantized NInfer variant specifically targeting 24 GB GPUs?
For my use case, a text-focused variant would be completely fine. I would happily trade things like Vision or DFlash2 support for some additional VRAM headroom, as long as MTP speculative decoding remains available.
Another interesting option, if supported by NInfer, might be pushing more of the currently FP8 weights to NVFP4.
The goal would not necessarily be maximum compression, but rather finding a good sweet spot for 24 GB Blackwell cards:
- Qwen3.8-27B
- NVFP4
- MTP support
- text-only is fine
- enough VRAM headroom for a useful KV cache / context
- minimal quality loss
Is something like this feasible with the current NInfer model target?
And if so, would you or someone else familiar with the conversion process perhaps have time to build such a variant?
I would be happy to test it extensively on my 2× RTX PRO 4000 Blackwell setup and provide VRAM usage, context limits, performance and stability results.
hi, 5090 user here.
this model is my daily driver, i cannot thank you enough for the absolute joy you gave me. seriously. changed my life.
i was interesting into pushing efficiency even further.
would it be possible to support something like this https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF ? maybe mix with nvfp4?
hi, 5090 user here.
this model is my daily driver, i cannot thank you enough for the absolute joy you gave me. seriously. changed my life.
i was interesting into pushing efficiency even further.
would it be possible to support something like this https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF ? maybe mix with nvfp4?
I'm currently trying out different approaches to see which one results in the least loss of quality.
You're welcome to follow along here, but I've already specified my area of expertise there.
I'm currently running Qwen3.8-27B with NInfer on 2× RTX PRO 4000 Blackwell, 24 GB each, with one NInfer worker per GPU.
The current NVFP4 artifact works very well, but on a 24 GB card the available VRAM headroom is naturally quite limited.
Would it be technically possible to create a slightly smaller / more aggressively quantized NInfer variant specifically targeting 24 GB GPUs?
For my use case, a text-focused variant would be completely fine. I would happily trade things like Vision or DFlash2 support for some additional VRAM headroom, as long as MTP speculative decoding remains available.
Another interesting option, if supported by NInfer, might be pushing more of the currently FP8 weights to NVFP4.The goal would not necessarily be maximum compression, but rather finding a good sweet spot for 24 GB Blackwell cards:
- Qwen3.8-27B
- NVFP4
- MTP support
- text-only is fine
- enough VRAM headroom for a useful KV cache / context
- minimal quality loss
Is something like this feasible with the current NInfer model target?
And if so, would you or someone else familiar with the conversion process perhaps have time to build such a variant?I would be happy to test it extensively on my 2× RTX PRO 4000 Blackwell setup and provide VRAM usage, context limits, performance and stability results.
Perhaps your memory issues will be resolved by the fork https://github.com/ValerioDolci/ninfer-tp2
I’m running this model with kv = nvfp4, on two RTX 5080 cards with 16 VRAM, in p2p mode = ON, with a context of 360k + (2 concurrent requests simultaneously) and MTP = 3 + vision = ON.
The speeds are good.
If it’s a dflash2 speculation, the speed is higher, but the context is much smaller.
I’m very happy!
2026-10-03 14:11:09.094 INFO starting engine
2026-10-03 14:11:09.371 INFO loading weights | 15.2 GiB
2026-10-03 14:11:11.633 INFO weights ready | 15.2 GiB | 2.3s | 6.71 GiB/s
2026-10-03 14:11:11.639 INFO pinning host state | 293.6 MiB
2026-10-03 14:11:11.690 INFO host state pinned | 293.6 MiB | 51.3 ms
2026-10-03 14:11:11.691 INFO pinning host KV | 2.00 GiB
2026-10-03 14:11:11.969 INFO host KV pinned | 2.00 GiB | 278 ms
2026-10-03 14:11:12.702 INFO CUDA graphs ready | 733 ms
2026-10-03 14:11:12.703 INFO engine ready | thinkingcap-target-o44attn-dx6-mtp-gateupdx15p650-combined-147590 | total 3.6s | weights 15.2 GiB
2026-10-03 14:11:12.703 INFO capacity | KV 350,592 tokens, nvfp4, explicit | pages 5,478/5,478 | runtime 7.55 GiB | free 706.6 MiB
... soon RC2 around, with a better decode result 😀