Qwen3.8 Flash-Next 3×3090+MTP — CUDA illegal access / PP rank failure under two live streams
We hit a fatal failure in the upstream 3×3090+MTP vLLM profile during real
two-stream use.
Environment
- Source repository: Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound
- Source path: vllm-patch/3x3090+MTP
- Source commit: 66964969fff959a86364eb3efc3d01f85741e802
- Locally built image ID:
sha256:dfde87df539110a16eb677370199fc421313ab396b9e3b5aa398b26942f96a34 - vLLM: v0.29.1rc1.dev47+gdc36fcce9
- Host driver: 595.84
- Host CUDA reported by nvidia-smi: 13.2
- Three RTX 3090s, CUDA_VISIBLE_DEVICES=0,1,2
- PP=3 / TP=1
- MTP: {"method":"mtp","num_speculative_tokens":1}
- max-model-len=262144
- max-num-seqs=2
- max-num-batched-tokens=512
- gpu-memory-utilization=0.97
- kv-cache-memory-bytes=550000000
- prefix caching enabled
- mamba-cache-mode=align
- EnGRAM CPU offload enabled
- QSA host-KV offload enabled
- PLE mmap enabled
- compilation:
{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY",
"cudagraph_capture_sizes":[1,2,4]} - VLLM_USE_BREAKABLE_CUDAGRAPH=0
Observed user symptom
- Under two real concurrent requests, one response began looping a repeated
fragment ("ductductduct...") rather than producing valid text. - A retry then failed in the client as:
[502]: Unknown error (reset after 24s)
Engine timeline
07:13:41.343Z:
rank 1 NCCL process-group watchdog reported:
"CUDA error: an illegal memory access was encountered"The error also appeared for the PP broadcast group:
PG GUID 35(pp_broadcast), Rank 1.The CUDA error is asynchronous according to PyTorch's own log, so the
reported NCCL watchdog site is not necessarily the kernel that wrote the
invalid address.The engine continued in a broken state for roughly five minutes:
repeated "No available shared memory broadcast block found in 60 seconds"
messages from shm_broadcast.07:18:41Z:
EngineCore failed fatally:
"TimeoutError: RPC call to sample_tokens timed out."
This propagated as vllm.v1.engine.exceptions.EngineDeadError.07:18:44.625Z:
container exited. Docker reports exit code 0 and OOMKilled=false because
the API process shut itself down after EngineDeadError; this was not a clean
successful inference completion.
Scheduler state
- The failure occurred with two requests running and zero waiting.
- KV-cache use was approximately 52%.
- Just before the first CUDA fault, the engine reported normal-looking
concurrent throughput and speculative acceptance. This was not a capacity
queue or prefill-admission failure. - At final EngineCore failure, the scheduler dump still reported:
num_running_reqs=2, num_waiting_reqs=0. - One dumped active request had 39,107 computed input tokens and 167 output
tokens. That is a crash-time snapshot, not proof that this request alone
triggered the original illegal access.
What did NOT occur
- No Docker OOM kill.
- No NVIDIA Xid in the captured incident window.
- No GPU disappearance/reset.
- GPUs 0–2 were idle and visible after the container exited.
- Corrected PCIe AER records appeared later in the failure window, but each
had aer_uncor_status=0x00000000. They are corrected telemetry only and do
not explain or supersede the CUDA illegal-memory-access event. - Caddy was not the origin: it initially logged an incomplete upstream
response, then logged connection-refused errors to 127.0.0.1:18133 after
the candidate had exited. Its 502s are downstream symptoms.
Patch-specific concern
- The source includes
flash-next-mtp-03-structured-output-drafts.patch. - Its header explicitly targets MTP + PP>1 + two or more simultaneous
requests, and changes DraftTokensHandler state/copy handling to retain
drafts across requests. - It is specifically intended for structured-output/JSON-schema cases.
We did not capture request bodies, so we cannot prove whether the failing
requests used response_format/json_schema or attribute the CUDA fault to
this patch. It is nevertheless directly in the affected concurrency regime.
Current conclusion
- This is a fatal candidate-engine failure in the PP/MTP execution path.
- The first load-bearing error is CUDA illegal memory access on logical
pipeline rank 1 / NCCL pp_broadcast, not the later sample_tokens timeout,
proxy 502, or corrected AER telemetry. - The repeated-gibberish output was observed by the client before the outage
and is consistent with a corrupted execution path, but raw generated-token
traces were not captured, so causality is not proven.
Questions / requested guidance
- Is CUDA illegal address on rank 1 / pp_broadcast known with this vLLM
revision under PP=3 + MTP1 + two concurrent requests? - Is FULL_DECODE_ONLY cudagraph mode supported for this exact combination of
MTP, PP=3, QSA host-KV, PLE host gather, and the streamed vision patches? - Is there a required patch, launch flag, or known-safe graph mode for this
profile? - Has flash-next-mtp-03-structured-output-drafts.patch been stress-tested
with two long live requests, and is there a known interaction with tool
calling or JSON-schema generation? - For a controlled reproducer, which single variable should be changed first:
MTP disabled, FULL_DECODE_ONLY disabled/enforce-eager, or the structured-
output draft patch isolated?
Evidence retained
- /media/shaxs/data/evals/qwen38-flashnext-minachist-3x3090-mtp-live-20260924T041500Z/incident-20260925T0721Z/
- container.log
- container-inspect.json
- kernel-since-0700.log
- caddy-since-0700.log
- nvidia-smi-q-after-crash.txt
- SHA256SUMS.txt
If you want any of that evidence above, let me know please.
Thank you very much for the detailed report. While I can't say for sure yet, I suspect this is likely an upstream bug in vLLM.
Issue: https://github.com/vllm-project/vllm/issues/55951
Proposed fix: https://github.com/vllm-project/vllm/pull/55506
These symptoms closely match what you're seeing, though I haven't been able to reproduce your crash yet. I'm currently investigating the issue.
I have successfully reproduced the issue completely. I will send a detailed report shortly, so please give me a moment.
Thank you for waiting, and thank you again for such a careful report.
I reproduced your incident end to end on my own 3 × RTX 3090 box with the shipped 3x3090+MTP image and your settings. It is an upstream vLLM bug (vllm#55951), and vllm#55506 fixes it. I added that PR as a new patch, flash-next-upstream-01-mamba-align-req-slot.patch, to all three build directories (3x3090/, 3x3090+MTP/, 4x3090/) in commit 955532e. Please rebuild your image from the updated vllm-patch/3x3090+MTP/ directory. The model files and the MTP INT4 files are unchanged.
With --mamba-cache-mode align, vLLM copies each request's GDN/Mamba recurrent state into the block-aligned cache slot when the request crosses a cache block boundary (9,776 tokens in this configuration). With MTP, the copy position depends on how many draft tokens were accepted, so the copy runs after sampling.
The copy kernels read the block table through the batch row. The runner re-gathers that table in batch order every step. With pipeline parallelism, a non-last rank (ranks 0 and 1 here) processes a step's sampled output pp_size steps late. By then the batch-ordered table already belongs to a later batch. The copy then combines one request's counters with another request's block IDs. The first request's boundary state is not saved, and the other request's live state is overwritten. According to the PR's analysis, the corrupted state produces all-NaN logits, and the sampler then keeps emitting token 1023 (duct). The illegal access comes from the same misdirected copies. On some GPUs they fault immediately (see vllm#54173), and on the 3090s they can run silently for a while before the engine faults.
Reproducing this needs PP > 1, MTP, align mode, and at least two requests whose batch rows change between steps. PR #55506 binds the copies to the persistent per-request tables and indexes them by request slot, as the V1 runner already does. The PR's credit goes to its author and to zebgop-ops, who found the root cause first. I only rebased it (one docstring line differed) and tested it on this model.
To answer your questions:
- Is the CUDA illegal address on rank 1 / pp_broadcast known with this vLLM revision under PP=3 + MTP1 + two concurrent requests?
Yes. The bug is known upstream as vllm#55951. The fix (#55506) is not merged upstream yet, which is why I ship it as a patch for now. The pp_broadcast watchdog only noticed the asynchronous error. The bad write did not happen where you suspected. With the patch, the fault did not come back in my runs.
- Is FULL_DECODE_ONLY supported for this combination of MTP, PP=3, QSA host-KV, PLE host gather, and the streamed vision patches?
Yes. It is the shipped and tested configuration. The bug is not a CUDA graph problem. The state copies run outside the captured graphs, so eager mode has the same bug. My before/after runs both used FULL_DECODE_ONLY [1,2,4] with QSA host KV, the PLE host gather and vision enabled, and the patched run did not have the issue.
- Is there a required patch, launch flag, or known-safe graph mode for this profile?
The required patch is flash-next-upstream-01-mamba-align-req-slot.patch. The updated 3x3090+MTP/Dockerfile already applies it. No launch flag or graph mode avoids the bug. If you cannot rebuild right away, the safe fallback is the 3x3090/ build without MTP. --max-num-seqs 1 should also avoid it in principle, but I have not tested that. Please do not remove --mamba-cache-mode align. The server needs it for long prompts.
- Has flash-next-mtp-03-structured-output-drafts.patch been stress-tested with two long live requests, and is there a known interaction with tool calling or JSON-schema generation?
mtp-03 is not involved in this crash. My reproduction used no structured output and no tools. The faulty code is the Mamba state copy, which never touches DraftTokensHandler. The earlier minimal reproduction does not call it either. On testing: mtp-03 was verified with 30 short JSON-schema requests, and with 15 schema side requests running next to 3,000-token main answers. The fix run above also had mtp-03 in the image, with two long live streams for 100 requests, but it did not exercise structured output. So I have not stress-tested tool calling or JSON schema with two long streams, and I know of no interaction. If you see anything structured-output-specific after updating, please tell me.
- For a controlled reproducer, which single variable should be changed first?
Among your three candidates, disabling MTP is the one makes it go away. enforce-eager / disabling FULL_DECODE_ONLY would not help, and isolating the mtp-03 patch would not help either.
To reproduce:
I used the same image (vLLM 0.29.1rc1.dev47+gdc36fcce9, the digest-pinned base in the Dockerfile) and the same flags as your list: PP=3 / TP=1, VLLM_PP_LAYER_PARTITION=16,17,15, MTP with 1 speculative token, max-model-len 262144, max-num-seqs 2, max-num-batched-tokens 512, kv-cache-memory-bytes 550000000, prefix caching, mamba-cache-mode align, EnGRAM CPU offload, QSA host KV, PLE mmap, vision, and FULL_DECODE_ONLY with capture sizes [1,2,4].
The load was two concurrent streams of ordinary requests. I requested to explain a Python file, with 6k-45k tokens of source code as context, 400-900 output tokens, temperature 0.7. No response_format, no tools. Because the prompt lengths are mixed, one request is usually prefilling while the other decodes.
- The 12th request degenerated in the middle of a normal sentence:
… **Supported Modules Detection:** It calls … * **Moductductductduct…**. It didn't recover, and an earlier trial run looped the same way on the 9th request. - About 8 minutes in, the rank 1 NCCL watchdog reported
CUDA error: an illegal memory access was encounteredonPG GUID 35(pp_broadcast) Rank 1, thenEngineDeadError. - At that moment the engine reported
Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.5%.
That is your signature exactly. That is a same rank, same process group, the same ~52% KV use, and the same order (a looping response first, the fatal fault later). Only difference is my engine died right away, where yours hung for 5 minutes until the sample_tokens RPC timed out. I think that difference is only where the asynchronous error surfaced.