Feedback: Qwen 3.8 Flash Next on 3X3090+MTP
Field report: Qwen3.8-Flash-Next-INT4-Mixed-AutoRound, 3×3090 + MTP (prefix3)
Rig: 3× RTX 3090 24GB (71 GB VRAM total), host 54.7 GiB RAM. Build: 3x24GB+MTP from HF main @ f6f90dd + 7 local patches (qsa-fp8 ×4, prefix-eagle-flag, prefix-mtp-annotate, 52244-replay). Image tag: flash-next-vllm:prefix3. vLLM base 43f13b4c. Config: PP=3 (16,17,15), TP=1, moe marlin, KV fp8_e4m3 550 MB/GPU, --gpu-memory-utilization 0.80, --max-num-seqs 2, --max-num-batched-tokens 512, --max-model-len 262144, mamba align, prefix-caching ON, MTP speculative 1 token, vision tower host-streamed, PLE table on NVMe mmap. Change vs stock recipe: removed --limit-mm-per-prompt (defaults to 999). Uptime at report: fresh build, canonical rebench + 1 day production use on previous prefix2 image (same recipe minus HF fix #5).
It works
Canonical rebench-full.sh (club-3090) against :8000, all green:
verify-full8/8 OK (tool prefill, IDE-agent one-shot, multi-turn, LCB-coding, reasoning-heavy, NIAH 60K/90K, ceiling ladder to 241K).bench.sh: narrative 115.8 tok/s (CV 0.7%, TTFT 92 ms), code 129.1 tok/s (CV 0.6%, TTFT 91 ms). Prefill 10K: ~3410 tok/s (TTFT ~2.9 s). Decode @90K depth: 103 tok/s.bench-agentic(2 sess × 12 turns → 35K ctx): turn-12 TTFT 8.8–9.0 s, decode ~120–128 tok/s. No timeouts.concurrency@16K ctx, seqs=4 (forced explicitly; auto-detect only ran N=1): N=1 → 107 tok/s/stream; N=2 → 66 tok/s/stream (PASS, clean); N=3 → 30 tok/s/stream (PASS, clean). Aggregate caps at ~40 tok/s — shared chunked-prefill (512) is the ceiling, not decode. seqs=2 is the sweet spot; reverted to it after the test.soak20 sess × 5 turns: verdict PASS, all HTTP 200, p50 decode 126.1 tok/s, p95 TTFT 1284 ms, no degradation.- F0 smoke: "Paris" correct (thinking off; with thinking on, the reasoning tokens consume a small max_tokens budget — harness-side expectation, not a model bug).
Prefix caching + MTP (the interesting part)
Production traffic (Pi coding agent, append-style multi-turn): prefix_cache hit rate 54% on a short session, 83–84% sustained (12.7M/15.3M tokens). Attention block 9776, kv lcm 9776 — reuse lands on whole blocks as expected.
Findings from A/B testing (documented, reproducible):
- MTP ON vs OFF (same image,
--speculative-configremoved for OFF): identical 0-hit on synthetic repeats → MTP is not the cause of anything cache-related here. MTP accept rate ~0.77, no decode regression. - prefix1 (annotate only) vs prefix2 (annotate + 52244-replay): bench-agentic turn-12 TTFT within 2% (noise) → the replay patch is neutral on this workload: doesn't help, doesn't hurt.
- Synthetic A→A repeats below one GDN block (9776 toks) always give 0 hits by design (0 whole blocks to reuse), on every image and endpoint tested. Real append-style traffic above ~10K toks reuses fine. This initially looked like a bug; it is block arithmetic.
- One harness-side cache killer found (not a server bug): when Pi rewrites its system prompt mid-session (tool list changes), the token LCP collapses (9348 → 103) and the whole prefix invalidates. Freezing the toolset per session fixes it. Mentioning because other agent harnesses may do the same and blame the server.
Vision
6 images in one request → 200 OK after removing the --limit-mm-per-prompt '{"image":4}' cap (untested upstream default, not a VRAM limit — the streamed tower costs ~0 fixed VRAM per image). mm_cache 61/65 hits (94%). 2 MP/image cap kept.
Our 7 patches on top of the stock recipe
QSA FP8 compat (4 patches, one concern: run QSA KV in fp8_e4m3 instead of BF16 — halves the QSA host pool 41.2 → 20.6 GiB): backend dtype support, host-pool pinning with the real dtype size (no hardcoded BF16 ×2), and FP8 dequant in the staging + decode Triton kernels (exact 256-entry E4M3→BF16 LUT — triton/SM86 has no fp8e4nv IR type). Without these, QSA KV stays BF16 at 2× host RAM.
Prefix-cache under MTP (3 patches: un-veto attention reuse): per-group eagle flag, MTP draft-group annotate (releases the other 20 of 21 groups from the :2215 flag-all veto), and a minimal port of upstream PR #52244 rule 2. Measured neutral on bench-agentic (±2%, noise); kept because production prefix-hit rate is 84% with them (0 without, under MTP).
PLE fix #5
Applied via the updated flash-next-vllm.patch (+ decode-01/06, ttft-01). No sparse/incomplete warnings in logs on our existing table; build applied all 22 patches with 0 hunk failures. Can't confirm the fix triggers (our cache was already dense), but it doesn't break anything.
Issues / tweaks
- None so far on prefix3. Previous prefix2 ran 1 day in production (Pi agent traffic) with zero errors, zero OOM, zero preemptions.
- Note:
verify-stressVRAM probe warns "could not determine model GPU(s) on 3-GPU host" — cosmetic, uses min-free fallback (255 MB reported at ladder start; nvidia-smi shows ~1 GB free/card in idle, which is the normal pre-allocated state, not pressure). rebenchswap check FAILs on our box (987 MB of server paged out; 54 GB host RAM, 16.7 available) — numbers above are still tight (CV <1%), but noted for honesty.
Verdict
3×3090+MTP recipe works as advertised on a second rig. Decode ~115/129 tok/s narrative/code, long-ctx to 241K, prefix cache 84% on real agent traffic, MTP 0.77 accept, vision OK without the 4-image cap. No crashes, no tweaks needed beyond the image-limit removal. I'm a big fan of this work — the recipe is the only sane way to run this model on 3090s. Thanks for the quant + patches.
P.S. Have you considered putting the recipe (Dockerfile + patches + compose) in a GitHub repo? Right now it lives as loose files on HF plus these local diffs — a repo would make it much easier to track fixes like #5, accept PRs (e.g. the QSA-FP8 compat above, happy to contribute it), and let others report per-commit results.
Thank you for a report.
About N=3 at 30tok/s per stream, it is a CUDA graph capture-size issue.
With MTP every stream verifies 2 rows per step, so N=2 is 4 rows, N=3 is 6 and N=4 is 8. The shipped cudagraph_capture_sizes is [1,2,4], so N=3 and N=4 fall back to eager decode when you raise --max-num-seqs to more than 3. The compose comment says this ("2 x --max-num-seqs must be one of cudagraph_capture_sizes"). For --max-num-seqs 4 use [1,2,4,8].
Even with graphs, this build is for one or two streams. MTP is a latency tool. An extra verified row costs little while a step is latency-bound (N=1: 10.1 to 14.0 ms per step for 1.7 tokens), and a lot once the step already carries many rows. From my tests, MTP vs no MTP are N=1 +27%, N=2 +0.9%, N=4 −22.5%. The step time is flat up to 2 rows (10.1 to 10.6 ms), then grows with rows (16.5 ms at 4, 35.7 ms at 8). MTP doubles the rows per step and gains 1.7 tokens for them (acceptance 0.71), so past one stream it pays for rows that no longer come for free.
In general, speculative decoding spends spare compute to cut latency, so it helps most at low batch and costs throughput when the batch is already saturated. For 3+ streams 3x24GB is the right build.
About vision, image cap is just a conservative number and it's not a measured limit. You can go more than 6, but I don't see any benefits for this. I'll measure the limit later though.
About GitHub, I'm looking into putting them in a GitHub repo.
About FP8 QSA KV, I haven't tested FP8 KV and I wouldn't make it the default (I don't like to quantize KV cache as it degrades the quality badly), but halving the pool matters on hosts like yours, so an opt-in switch is reasonable. I may also ship calibrated KV scales as a separate overlay so FP8 KV does far less damage.
This is how I actually run the model on my system, with Pi, long sessions, MTP, prefix caching, vision, and all the experiments around it. It’s not meant to be a pristine stock setup.
The FP8 QSA KV change is a good example. I didn’t do it because I specifically wanted FP8 KV; my rig is pretty RAM-constrained, so I had to make these changes. Otherwise I simply wouldn’t be able to run the setup properly. Going from ~41 GiB to ~20.6 GiB for the QSA host pool is what makes the whole thing practical on this machine.
Same with the image cap and the prefix patches: those changes came from things I hit in real daily use.
So from my side, this really is the field report: this is the configuration I actually use, with the constraints and modifications that were necessary to make it work well on my 3×3090 box.
Happy to support in any way. Let me know if you need anything else from me as a user and I can't wait for the Github to appear! :)