Minachist

/

Qwen3.8-Flash-Next-INT4-Mixed-AutoRound

Like 26

4× RTX 3090 (24 GB), 4x3090 build — running stable since Sep 25, long-context validated to 859k

Setup:

• Proxmox LXC container, 4× RTX 3090 + 768 GB, ddr4 RAM, dual skylake. Note the PCIe topology: one card on direct x16, three on Oculink x4 links (Gen3 x4 / one x16). Did not seem to hurt decode/PP (see below) — your PP numbers look in the same ballpark.
• Image: patched vllm/vllm-openai:nightly-dc36fcce9 (0.29.1rc1.dev47+gdc36fcce9) with your rel-20260925 patch suite, PP4 with VLLM_PP_LAYER_PARTITION=13,13,13,9, --kv-cache-memory-bytes 1566000000, --max-num-seqs 2, --max-num-batched-tokens 1024, VLLM_QSA_KV_OFFLOAD=1, PLE in RAM mode (VLLM_PLE_MMAP_PATH unset), engram cpu_offload. Boot: 4m47s with warm page cache, ~9.5min cold; QSA pins ~181 GiB across the 12 QSA layers, KV pool 5,562,011 tokens = 5.30× concurrency at 1M.
• Served behind llama-swap as a 1M ctx lane. MTP currently off in our compose variant (see below).

Measured (fresh, Sep 26, warm):

• Decode: 84.8 t/s wall / 92.6 t/s ITL single-stream (count-100, temp 0); 87.5 t/s prose-512 temp 0.7
• Concurrency ×2: 164.8 t/s aggregate (82.6 + 82.4 per stream) — only ~2.6% per-stream penalty at bsz=2, much less than the -21% @4-streams you report for MTP
• Prefill: 2 176 t/s @1.2k, 4 520 t/s @9.3k (salted prompts, no prefix cache)
• Long context: needle test at 859 482 prompt tokens — both needles (50% and 95% depth) retrieved correctly, no NaN/degeneration. Prefill 953 s = 906 t/s. RoPE extrapolation far beyond the native 262k window holds for this ckpt in our runs.
• Prefix caching: on, converges to ~79–80% hit rate under steady agentic traffic; --enable-per-request-metrics works as advertised (nice side benefit: drives our proxy's t/s telemetry).

Changes vs provided config:

• --max-model-len 1048576 (with VLLM_ALLOW_LONG_MAX_MODEL_LEN=1) instead of 512k — the 5.56M-token KV pool absorbs it fine at 15% usage for a single 859k sequence.
• Added --enable-per-request-metrics + --enable-prompt-tokens-details.
• MTP draft head commented out. When n=2 was enabled we saw it net-negative under 2-stream concurrency and it shrinks the effective prefill chunk (max_num_scheduled_tokens=512 warning), so we're riding no-MTP for the agent fleet.

Issues / observations:

1. No VRAM margin at load: boot log shows expandable_segments: memory mapping failed with OOM on all 4 cards during weight load (12–16 MiB free) — survives, but anything co-resident on the GPUs would kill it. Initial free memory 23.14 GiB, reserved 1.46 GiB KV + manual kv_cache_memory_bytes means gpu_memory_utilization is effectively ignored; worth a README note.
2. Cold-request JIT spike: first request after boot runs ~17 t/s while _w8a16_gemm_kernel, _qsa_sparse_staged_gqa_kernel and _fill_num_accepted_kernel compile during inference (jit_monitor warns). Extending warmup to cover those shapes would remove the spike.
3. Boot log warns Unknown vLLM environment variable detected: VLLM_PLE_MMAP_PATH / VLLM_QSA_KV_OFFLOAD / VLLM_DEFAULT_THINKING_BUDGET — the patches read them via os.environ so it's cosmetic, but it does scare people who don't read the patch code.
4. Minor: duplicate --enable-prefix-caching in the rendered args; --model deprecation warning.
5. Vision is warmed up (23 s mm warmup, limit 4 images) but we haven't exercised it under real workloads yet — can't confirm/deny vision quality.

TL;DR: 4x3090 build starts clean, runs rock-steady for a day+ of agentic traffic at 1M ctx, ~85–92 t/s decode, 4.5k t/s prefill at 9k, needles pass at 859k. Main sharp edge is the zero-VRAM-headroom load path.

Thank You!

Thanks for the quant.
I haven’t tried it yet , but knowing your previous quants’ for the 27B models excellent quality/size ratio I’m curious to try it too. But I have dual 5090s and 64GB of system RAM (can be expanded to a slower 128GB) so I’m not sure it’s the best choice , cause it seems that with 2x32GB VRAM, a significant part of expert weights would have to be offloaded to system RAM , especially with full context at BF16/F16 (even when its compressed) . Also I’d lose the NVFP4 / FP8 advantage of the Blackwell. So is it even worth trying? I thought if only small part would have to be offloaded and active parameters is only 6B then probably the TG should still be acceptable?

Given that it's for vLLM, you'll need to patch vLLM to get it working. In theory, it should be entirely possible to run it within 64 GB of VRAM, and I think TG could still be quite high, probably around 100 tok/s. I'll try putting together a patch later.

As for the NVFP4 / FP8 advantage, you aren't actually losing anything here. This quant is A16, which means only the weights are quantized and the activations stay in 16-bit. The weights are dequantized inside the kernel, and the matmuls run on the BF16/FP16 tensor cores. Blackwell's FP8 and FP4 tensor cores are only used when the activations are also in FP8 or NVFP4 (WnAFP8 or WnANVFP4). So as long as the activations are 16-bit, those tensor cores are never used in the first place, no matter which GPU you're on. At that point, whether a given weight format runs fast is a question of kernel support, not hardware. Fun fact, even a 3090 can run FP8 / FP4 A16 weight-only quants without a speed penalty, despite having no FP8 or FP4 tensor cores at all. Similarly, vLLM can run INT1-3 and INT5-7 through HummingKernel, even though no native tensor cores exist for those precisions.

Thanks, very much for clear explanation .
Yeah it would be nice to have it working on Blackwell too
I'm quite surprised regarding :

Fun fact, even a 3090 can run FP8 / FP4 A16 weight-only quants without a speed penalty, despite having no FP8 or FP4 tensor cores at all

because I read otherwise several times, but probably that was just me not understanding the issue and some other quants having activations at FP8 or even NVFP4 (instead of keeping all of them at 16 bits).

I honestly think this is one of the most widespread misconceptions in the LLM space. People often see a weight format called FP8 or FP4 and assume the GPU must have native FP8/FP4 tensor cores for it to run efficiently, even when the activations are still FP16/BF16.
Strictly speaking, sm89 and newer GPUs do have hardware instructions for converting FP4/FP8 to and from FP16. 3090 is sm86 so it doesn't, so the kernel does the conversion with regular integer and bit operations instead. Either way, for A16 weight-only quantization the dequantization cost is very, very small compared with the rest of the kernel, especially during decoding, which is mostly bound by memory bandwidth. On a 3090 level of GPU, I'd expect other parts of the kernel to become the bottleneck long before the missing native FP8/FP4 matmul path becomes noticeable.
So for WnA16, what matters much more in practice is whether there is a good kernel for that particular weight format and architecture, rather than whether the GPU advertises native tensor cores for the storage datatype.

qwen originally overthinking. might be like thinkingcap or swift will be cut the thinking?

I also have trouble with over thinking even when giving it a thinking cap. Its my biggest gripe with Qwen models

Thanks for the feedback. Overthinking is a known issue with this model, and I've heard it from a few people now.
I'm currently downloading ukisai/Swift1.5-Qwen3.8-Flash-Next, which claims to cut thinking length with RL. I'm usually skeptical of finetunes, but I've seen many people suggesting it elsewhere, so I might give it a try. I will quantize it with the same recipe and it should be up in few days.
Please note that Swift is a third-party finetune, so I can't guarantee its quality. Also, Swift has its own license (Swift Open License v1.0) on top of the Qwen license, so check that if you plan to use it commercially.

I can't wait to see the Swift version, even better if we can uncensor it. 😃

@Minarhist
Yeah, I’d expect that there will be some downsides as I vaguely remember that was the case even with a previous one by ukisai (Qwen 3.6 or 3.8 27b) which was quite promising, but when enough people try using them it turns out there are tradeoffs - which is not that surprising. But if overall, or even only for some use cases it’s better , then it’s worth it.

Btw: I haven’t tried the current quant ( based on the original one for dual 5090) yet, because I’m quite busy , but when I eventually do I plan to run some benchmarks and maybe compare it with some other quant( probably some that used NVFP4 expert weights quantization ) and report. Can’t promise when, but hopefully within a couple weeks .

The AutoRound quantization is now available here: Minachist/Swift1.5-Qwen3.8-Flash-Next-INT4-Mixed-AutoRound.