Feedback: Qwen3.8 Flash Next on 4× RTX 3090
We spent the last couple of days trying to make the Minachist Qwen3.8 Flash Next/MTP stack work well on four RTX 3090s. We separated three questions:
Can the native PP4 configuration start and fit?
If it starts, does it improve real interactive/concurrent performance?
Can the established PP3 (original 3 card recipe) MTP configuration simply be spread across four cards instead?
Native four-GPU PP4 configuration: initial fit failure
The native four-card configuration used:
PP=4 / TP=1
Partition 12,14,14,8
262K context
MTP-1
1.03 GB KV-cache reservation
QSA host-KV offload and Engram CPU offload
Vision-capable model path
The initial native PP4 candidate did not reach API readiness. It failed during post-load Marlin MoE repacking, rather than during KV allocation or vision profiling:
PP1: Tried to allocate 800 MiB; only 738.62 MiB free
PP2: Tried to allocate 800 MiB; only 758.62 MiB free
The direct rank-1 shortfall was only about 61 MiB, but it was a hard CUDA OOM during gptq_marlin_moe_repack.
We ruled out:
KV/context sizing as the immediate cause—the failure happened before KV-cache creation.
Vision replication—the vision streaming patch was active.
Xid, reset, uncorrected PCIe error, kernel OOM, or Docker OOM-kill.
A single uniquely bad rank—other ranks also had allocator mapping-OOM warnings during the peak.
The required fix to make native PP4 start
The smallest successful change was adding:
--cpu-offload-gb 0.5
That gave enough peak-VRAM margin for the Marlin repack and allowed the PP4 engine to start.
We then ran a controlled no-UVA retry where the only serving-policy change was removing that exact --cpu-offload-gb 0.5 argument. It failed again at the same Marlin-repack step before API readiness.
So the conclusion is narrow but solid:
For this exact PP4 bundle—partition 12,14,14,8, MTP, 262K ceiling, 1.03 GB KV reservation, QSA/Engram policy—explicit UVA parameter offload is a startup-admission requirement.
That is not a claim that every four-way pipeline layout must use CPU/UVA offload. It is a fit result for this particular PP4 configuration.
What we saw after native PP4 was running
The native PP4 arm could serve requests, but it did not reproduce the useful two-stream interactive behavior of the established PP3 MTP arm.
Important correction: the first “terrible C1 tok/s” reading was partly a measurement artifact. It included roughly 26 seconds of cold 90K-token prefill and only 128 generated tokens.
For the PP4 arm’s cold 90K C1 runs:
Wall-output rate: about 4.15–4.50 tok/s
TTFT: 26.47 s
Actual post-first-token decode: about 52.61 tok/s
Short warm-prompt decode: about 59–62 tok/s
So the issue was not literally a 4 tok/s decoder. The important problems were cold-prefill TTFT and two-stream scheduling behavior.
On the PP4 C2 runs:
One request typically reached about 45–60 tok/s decode.
The other often saw only roughly 4.5 tok/s early decode.
First tokens were staggered around 26.7 s and 53.8–54.2 s.
waiting_max=1 occurred in all C2 rounds.
In other words, the second request was not being admitted and serviced like a true simultaneous interactive decode stream.
That contrasts sharply with the established PP3 MTP arm, where we had both:
Real coding tasks observed at approximately 100 tok/s and 80 tok/s simultaneously.
A measured 60-second two-stream window with running=2, waiting=0, and 170.97–187.75 aggregate tok/s.(2/6)
Why “less offload on PP4 should be faster” was not borne out
The native PP4 setup was not actually a clean “same model, one extra GPU, less offload” comparison.
The good PP3 MTP arm:
PP=3, partition 16,17,15
No --cpu-offload-gb
Engram CPU offload enabled
QSA host-KV offload enabled
The native PP4 arm:
PP=4, partition 12,14,14,8
--cpu-offload-gb 0.5
QSA host-KV offload enabled
More/larger host-backed state
In practice, PP4’s nominal 0.5 GB CPU-offload setting produced approximately 4.15 GiB of rank-summed UVA weight exposure:
PP0: about 1.63 GiB
PP1–PP3: about 0.84 GiB each
The PP3 MTP arm that performed well did not use this parameter-offload path at all.
Likely contributors to PP4’s poor interactive concurrency are therefore:
The slowest PP stage determines token latency; PP0 had the largest UVA-backed payload.
PP4 adds a fourth pipeline stage: three PCIe stage boundaries per token instead of two.
There is no NVLink on this system.
The fit-oriented 12,14,14,8 partition is not necessarily a decode-balanced partition.
PP4 has much more QSA host-KV state than PP3.
The MTP-constrained scheduler gave fresh 90K prompts a shared 512-token budget, creating prefill waves and queueing.
We did not establish a single root cause or prove that PP4 topology itself is inherently bad. The configurations were still confounded by partitioning, offload policy, host-cache size, and scheduler policy.
Running the established PP3 runtime across four cards
Next, we took the known-good PP3 MTP image/runtime policy and changed only the card geometry:
GPUs: 0,1,2 → 0,1,2,3
Pipeline stages: 3 → 4
Partition: 16,17,15 → 13,13,13,9
max_num_seqs: 2 → 4
We preserved MTP-1, QSA host-KV, Engram CPU offload, 262K context, and no explicit --cpu-offload-gb.
The 13,13,13,9 split was intentional: the last stage owns the BF16 MTP draft head, so equal transformer-layer counts would not balance VRAM residency.
This PP3-policy-on-four-cards configuration did fit and pass its short functional C4 smoke:
4/4 requests HTTP 200
Four active requests observed
No queueing in that short fixture
No OOM, container loss, fatal kernel event, or thermal violation
It also completed a 90K C1–C4 capacity matrix. At C4:
All requests completed.
Median TTFT: 36.46 s
running_max=4
waiting_max=3
Peak KV usage: 84.0%
It had enough capacity to admit four long contexts, but fresh 90K prefills still arrived in waves. That is capacity/admission, not proof of four useful simultaneous decode streams.
We tried scheduler changes:
Baseline 512/no-cap: C4 median TTFT 36.46 s
2048/no-cap: improved C4 median TTFT to 29.82 s and aggregate wall output from 8.58 to 9.85 tok/s
But even the best tested scheduler setting still had waiting_max=2
So increasing the scheduler budget improved cold-prefill admission, but did not create true four-stream long-context concurrency.
We also raised the PP3-on-4 KV reservation from 550 MB to 1 GB per worker:
Reported token capacity: 718,781 → 1,306,491 tokens
Advertised full-262K capacity: 2.74× → 4.98×
That improved context capacity, but it did not solve the decode/scheduling issue. More KV is not the same thing as more parallel decode throughput.
MTP correctness issue found and fixed on four cards
While the PP3 runtime was spread across four cards, we saw the old repeated-output failure again under concurrent MTP:
ductductduct...
We found that the active four-card image was based on a Dockerfile that did not include the aligned-Mamba persistent request-slot fix.
We rebuilt the four-card image from the updated source and applied the upstream vLLM PR #55506 backport. The patch makes the aligned-Mamba speculative-decode tables use the persistent request slot rather than a transient batch-local index.
After an image-only swap, the patched PP4 arm passed a bounded MTP-active two-stream reliability test:
50 synchronized pairs / 100 completions
Every response HTTP 200 and non-empty
Every pair reached running=2 with no observed queue
No pathological repeated-token run
threshold: 12 identical tokens
observed maximum: 2
MTP was active:
9,069 drafts
6,874 accepted
75.80% acceptance
So the request-slot/Mamba bug was fixed as a reliability matter.
However, that test was deliberately a short bounded correctness gate. It does not establish PP4 as a throughput or routing winner.
Why we returned to the three-GPU PP3 topology
We ultimately restored the old PP3 MTP arm because it remains the only configuration with demonstrated useful real-world two-stream coding behavior:
PP=3
GPUs 0,1,2
Partition 16,17,15
MTP-1
262K context
No explicit CPU parameter offload
The PP4 candidate was stopped cleanly and preserved for future work. GPUs 3 and 4 are currently unused.
This was not a claim that the patched PP4 cannot work. It was a practical decision based on what was actually demonstrated:
PP4 required explicit CPU/UVA parameter offload merely to start in its native configuration.
The native PP4 C2 fixture showed queueing and highly asymmetric stream behavior.
Spreading the PP3 policy over four GPUs demonstrated capacity, but not the sustained concurrent coding-task behavior we already had on PP3.
The specific matched real coding-task replay that would fairly settle PP3 vs PP4 was not run, because we need the original workload artifact rather than synthesizing a replacement.
What would be most useful from Minachist
Is there a recommended PP4 partition/offload policy that avoids or reduces the Marlin-repack startup peak without placing a large UVA-backed parameter payload on a token-critical stage?
Is the PP4 source warning—“one stream decodes faster, several streams slower”—expected behavior for this topology, and is there a known improved configuration for multi-stream decode?
For MTP with long fresh prompts, is there a scheduler configuration that avoids a single request consuming the entire early prefill budget and forcing the other request into a later TTFT wave?
Should the request-slot/aligned-Mamba fix be treated as mandatory in every PP+MTP topology Dockerfile/release image, not just the three-card image?
Is there a recommended way to compare PP3 and PP4 fairly while holding image, quantization, MTP, context, batch policy, and host-offload policy constant?
Bottom line: we solved the four-card MTP corruption bug and proved that PP3-style configuration can physically span four cards. But neither native PP4 nor PP3-on-4 yet matched the established PP3 arm’s real two-stream coding behavior, so we returned to the proven three-GPU topology while keeping the patched PP4 candidate intact for a controlled, matched-workload follow-u
Same hardware: 4× RTX 3090, 128 GB of host RAM, no NVLink. PCIe gen4 x16 btw.
Four days of pain, a day for each card, I guess ;) - and I have been running this model on native PP4 for a while now.
Throughput (cards power-limited to 230 W):
- 1 request: ~80 tok/s (75 at 250K context)
- 2 requests: 140–150 tok/s aggregate, 70–75 per stream
MTP is off, so the partition below is the no-MTP one.
Runtime: patched image flash-next-vllm:local, built on top ofvllm/vllm-openai@sha256:43f13b4c624ab9e9e6753d0eeb5953268bff334f2a826239e6f4a2197d47bb96,
following the official 4x3090 compose recipe. Model: the 4x3090 branch of this repo.
environment:
# --pipeline-parallel-size below is what enables PP; this variable alone does not
VLLM_PP_LAYER_PARTITION=12,13,13,10
VLLM_PLE_MMAP_PATH=/ple/ngram_table.bin
VLLM_QSA_KV_OFFLOAD=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
VLLM_MARLIN_USE_ATOMIC_ADD=1
VLLM_USE_BREAKABLE_CUDAGRAPH=0
NCCL_P2P_DISABLE=1
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
OMP_NUM_THREADS=1
# The PLE offload path uses the pidfd_getfd syscall, blocked by the default
# seccomp policy; the PLE table is a ~96 GB mmap, hence unlimited memlock.
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
ulimits:
memlock:
soft: -1
hard: -1
command:
- --tensor-parallel-size
- "1"
- --pipeline-parallel-size
- "4"
- --engram-config
- '{"cpu_offload":true}'
- --max-model-len
- "262144"
- --max-num-seqs
- "2"
- --max-num-batched-tokens
- "512"
- --kv-cache-memory-bytes
- "450000000"
- --mamba-cache-mode
- "align"
- --enable-prefix-caching
- --compilation-config
- '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2]}'
- --limit-mm-per-prompt
- '{"image":2,"video":0}'
- --mm-processor-kwargs
- '{"max_pixels":2097152}'
--trust-remote-code
Host RAM usage ~50 GB.
Cold prefill is not fast, but not catastrophically so.
Thanks for the reports. I'm sorry about the original compose setup. I calculated it from three card measurements and did not leave sufficient margin for the Marlin repack peak on the middle ranks.
I have recalculated the allocation and updated both four-card compose files locally. MTP is now commented out by default. Optional MTP now uses INT4 g128 draft experts. The updated files have passed CPU/config checks and image builds, but I have three GPUs and cannot claim a successful four-GPU boot yet.
To answer your five questions:
Is there a recommended PP4 partition/offload policy that avoids or reduces the Marlin-repack startup peak without placing a large UVA-backed parameter payload on a token-critical stage?
For the 4x3090 weights, the revised split is 12,13,13,10.
MTP is disabled by default. I recommend trying this before adding --cpu-offload-gb. For optional MTP, quantize the draft experts to INT4 g128 and keep this revised split. The revised allocation should reduce the estimated loaded weights on each middle rank by about 1.39 GiB.
Is the PP4 source warning—“one stream decodes faster, several streams slower”—expected behavior for this topology, and is there a known improved configuration for multi-stream decode?
Yes. MTP should generally be thought of as a latency optimization rather than a throughput optimization. It can reduce per-request decode latency by accepting multiple tokens from the draft path, but this comes at the cost of additional draft and verification work. As concurrency increases, that extra work can reduce total throughput
For concurrent workloads, I therefore recommend starting with MTP disabled. In my three-card measurements, GPU-resident INT4 MTP improved single-stream decode by about 18-43%, while four-stream throughput was about 22% lower. MTP also changes KV cache capacity.
PP4 should mainly be evaluated for concurrent throughput or for fitting the higher-precision 4x3090 quantization. Adding another pipeline stage does not make single-request latency shorter. If anything, PP introduces additional pipeline communication and synchronization. MTP can compensate for some of that latency at low concurrency, but it should not be expected to improve PP4 aggregate throughput.
For MTP with long fresh prompts, is there a scheduler configuration that avoids a single request consuming the entire early prefill budget and forcing the other request into a later TTFT wave?
Try --long-prefill-token-threshold. For example:
--max-num-batched-tokens 512
--long-prefill-token-threshold 256
The pinned scheduler caps each long request's contribution to an iteration, leaving budget for the other request. For four requests at the same total budget, 128 is another starting point. Increasing the total budget without a per-request cap can still let the first prompt take the whole budget.
This can make prefill progress and TTFT more even, but it does not reduce the total work for two 90K prompts. The first request may get its first token later because it shares the work with the second. Available KV capacity and Mamba alignment still affect admission and scheduling. I checked that this setting exists in the pinned scheduler, but have not benchmarked it on four cards. This scheduling behavior can also occur with MTP disabled.
Should the request-slot/aligned-Mamba fix be treated as mandatory in every PP+MTP topology Dockerfile/release image, not just the three-card image?
Yes, for the affected vLLM base. It is already included few days ago (more info) in both the distributed four-card Dockerfile and the three-card MTP Dockerfile as flash-next-upstream-01-mamba-align-req-slot.patch. The current release also includes the structured-output draft fix. The image you identified as missing the request-slot fix was an older build. Please rebuild from the current distributed Dockerfile. There is no additional four-card-specific request-slot patch to apply.
Is there a recommended way to compare PP3 and PP4 fairly while holding image, quantization, MTP, context, batch policy, and host-offload policy constant?
Yes. Use the main checkpoint on both, and use vllm-patch/4x24GB/main/compose.yaml for the four-card run. That compose is already supplied for running the same weights on four cards; its updated default split is 12,12,12,12, with MTP disabled. The 4x3090 checkpoint is a different, higher-precision quantization, so comparing it with the three-card main run does not isolate PP scaling.
For a baseline, use the same built image and main weights, with MTP off on both, and match the prompts, sampling, simultaneous request count, batching and usable context capacity. The KV byte values may need to differ because PP changes the cache layout. If you then compare MTP, use INT4 g128 draft experts on both and match those same conditions again. You can use the four-card image for both PP3 and PP4; the PP size and corresponding layer partition are runtime settings.