This is exactly I searched for!
Many thanks!
I have dual 3090, and searched exactly this! So excited to try it out tomorrow!
I used https://github.com/noonghunna/qwen36-dual-3090 for Qwen 3.6. but it's not updated yet for 3.8...
Thanks legend!, what speed you guys getting?
Honestly I figure there will be an optimization on this by https://github.com/noonghunna/club-3090 in less than 24 hours.
In Open WebUI - starts to responds for questions, much much later! - because Thinking is ON - by default, but roughly the same token speed as Qwen3.6. ( ~ 35-80 tps. eg:- Drafted throughput: 74 tokens/s. ) Maybe 3.6 was a bit higher average: ~80tps?? )
BUT 3.8 is much much cleverer than 3.6! :)
Hermes Agent works fine with it, - i feel the same speed. However it solves very complex tasks !
Im very happy with this - Many thanks to you @lued and for QWEN. 😃
Im using latest official VLLM version 0.27.1. - and all requirements works. (compressed-tensors Marlin path, aligned Mamba cache, and MTP speculative decoding together.)
There definitely will be some template updates and better documentation coming in the future. Thanks for the support!
I'm certain the model itself is preserving the draft. I haven't spent too much time on this since I posted this quant 10 minutes after release.. HAHA
FYI. Vision understand is also working as 3.6 (or even better :) )
There definitely will be some template updates and better documentation coming in the future. Thanks for the support!
Many thanks for your work again. 😀
Yes. you were right Qwen3.8 fast lane is available: https://github.com/noonghunna/club-3090/blob/master/models/qwen3.8-27b/vllm/compose/dual/autoround-int4/mtp.yml
it is much faster Drafted throughput: ~114t/s.
Here is a 3-way comparison between AutoRound INT4, FP8, and INT8 on your dual
RTX 3090 setup for qwen3.8-27b.
- 3-Way Benchmark Comparison Table
| Metric | 🥇 AutoRound INT4 | 🥈 INT8 | 🥉 FP8 | INT8 vs INT4 | INT8 vs FP8 |
|---|---|---|---|---|---|
| Narrative Decode TPS | 86.83 tok/s | 75.14 tok/s | 74.21 tok/s | -13.5% | +1.3% |
| Narrative TTFT | 105 ms | 136 ms | 92 ms | +29.5% (slower) | +47.8% (slower) |
| Code Decode TPS | 117.19 tok/s | 97.62 tok/s | 98.13 tok/s | -16.7% | -0.5% (Tied) |
| Code TTFT | 121 ms | 130 ms | 116 ms | +7.4% (slower) | +12.1% (slower) |
| Prefill 10k Throughput | 2,507.16 tok/s | 1,524.25 tok/s | 1,516.76 tok/s | -39.2% | +0.5% (Tied) |
| Prefill 10k TTFT | 4.09 s | 6.64 s | 7.01 s | +62.3% | -5.3% (faster) |
| Prefill 90k Throughput | 1,699.30 tok/s | 1,191.19 tok/s | 1,184.49 tok/s | -29.9% | +0.6% (Tied) |
| Prefill 90k TTFT | 53.63 s | 76.49 s | 73.27 s | +42.6% | +4.4% |
- VRAM & Hardware Utilization Comparison
| Metric | AutoRound INT4 | INT8 | FP8 |
|---|---|---|---|
| Peak VRAM Allocated | ~45.28 GB | 45.47 GB | 46.07 GB |
| GPU 0 Post-run VRAM | 22.72 GB | 22.91 GB | 23.22 GB |
| GPU 1 Post-run VRAM | 22.43 GB | 22.54 GB | 22.82 GB |
| Decode Mem Controller Load | 66.1% | 76.5% | 74.6% |
| Decode PCIe Peak RX | 4,646 MB/s | 6,026 MB/s | 1,900 MB/s |
Key Insights & Takeaways
INT8 vs. FP8: Functionally Identical Throughput
- Both INT8 and FP8 use 1 byte per parameter. Because single-batch decode
is strictly memory-bandwidth bound, their generation speeds are
virtually identical:- Narrative: ~75 tok/s (INT8) vs ~74 tok/s (FP8)
- Code: ~98 tok/s on both
- Prefill: ~1,520 tok/s (10k) and ~1,190 tok/s (90k) on both
- INT8 allocated slightly less peak VRAM than FP8 (45.47 GB vs 46.07 GB),
giving a tiny bit more safety margin on 24 GB cards.
- Both INT8 and FP8 use 1 byte per parameter. Because single-batch decode
Why AutoRound INT4 Dominates on RTX 3090s:
- Weight footprint cut in half (~0.5 bytes/param): Halving the memory
bandwidth required per token results in ~15–17% higher decode speeds (87
vs 75 tok/s in narrative, 117 vs 98 tok/s in code). - Much faster Prefill: AutoRound INT4 is ~39% faster on 10k prefill and
~30% faster on 90k prefill. - Less Memory Controller Saturation: 66% load vs ~75–76% on 8-bit formats.
- Weight footprint cut in half (~0.5 bytes/param): Halving the memory
Thanks for the benchmark. I'm sure there is more optimizations to be had! At the moment I am just experimenting on a different method of INT4 quantization.