primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8

The 180B Flash-Next on one GPU, house-mixed.

Same single-96 GB-Blackwell deployment as our NVFP4 build,
with the full-attention and linear-attention projections at FP8 — 13% faster single-stream than the plain build.

---

Why this quant

• 🖥️ One 96 GB GPU, same recipe as the sibling: n-gram table in host RAM, three load-bearing serve flags, stock image.
• 🎯 92.2 knowledge / 84.8 call / 56.7 abstain under the pinned 1,370-item protocol (knowledge n=1, tool-calling n=3: 80.5/79.0/78.0 pooled) — statistically identical to the plain build, whose tie band is ±1.0.
• ⚡ 84.4 tok/s single-stream (10.8 ms/token), 526.0 tok/s at concurrency 32, prefix-cache-free — 13% and 8.7% ahead of the plain build on the same box. The gain is the FP8 GDN projections: 36 layers of decode weight traffic cut in half.
• 🧪 VLLM_GDN_DECODE_KERNEL=triton is REQUIRED. With the default cuda kernel, FP8 GDN projections hang the engine deterministically once concurrency reaches ~32 (no error, requests stall). We bisected this module class by module class; the triton kernel serves the same weights cleanly at identical accuracy and this speed.
• 🔀 MTP speculative decoding preserved, all 31 tensors byte-identical. Tool calling with thinking on and the qwen3_coder parser is verified working on this exact stack (a known sglang looping bug does not affect this image).
• 🧩 Compressed-tensors, stock vLLM image — the same format family as our Laguna, Ornith and Qwen3.8-27B mixed builds.

---

Serve it

docker run --gpus all --ipc=host -p 8000:8000 \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \
  -e VLLM_GDN_DECODE_KERNEL=triton \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Three flags are load-bearing on a single GPU. --distributed-executor-backend mp: the default executor never starts the n-gram offload worker and the server hangs silently. VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800: the worker loads a 95 GB table at startup. VLLM_GDN_DECODE_KERNEL=triton: required by this build's FP8 linear-attention projections — the default cuda kernel stalls under concurrent load. Host needs about 100 GB of free RAM. For long context beyond the native 262K, use --hf-overrides '{"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' (the old --rope-scaling flag no longer exists in recent vLLM).

---

Not enough host RAM? Put the table on NVMe

The serve command above wants ~100 GB of free host RAM for the n-gram table. With fast local storage you can skip that: this repo ships a one-file overlay (worker_image_disk.py) that maps the table from a file instead. First boot writes 95.4 GB into the store directory; every later boot maps it instantly and skips the table's checkpoint reads.

hf download primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 worker_image_disk.py --local-dir .
mkdir -p pledisk_store

docker run --gpus all --ipc=host -p 8000:8000 \
  -v $PWD/worker_image_disk.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
  -v $PWD/pledisk_store:/pledisk_store \
  -e VLLM_PLE_DISK_OFFLOAD_DIR=/pledisk_store \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
  -e VLLM_GDN_DECODE_KERNEL=triton \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Measured on this build, same box as the table below: 8K in / 512 out, prefix-cache-free, two seeds per cell (shown a / b):

config	boot	tok/s @ 1	tok/s @ 32	median TTFT @ 1
table in RAM (command above)	302 s	84.5 / 84.4	516.8 / 523.6	569 / 573 ms
disk, container capped to 48 GB RAM — recommended	263 s	79.4 / 76.8	427.0 / 435.8	571 / 573 ms
disk, uncapped 176 GB host	303–344 s	50.4–62.1	196.7–396.7	1.8–2.9 s
disk, cold page cache	404 s	40.8 / 37.0	134.2 / 290.3	4.6 / 5.2 s
disk, first boot (writes the file)	504 s	—	—	—

Net cost of the disk path, run capped: −8% single-stream, −17% at concurrency 32, TTFT parity with the RAM baseline. The counterintuitive row is the uncapped one, and it reproduces across two boots and four seeds: without a container memory cap, the boot's own 172 GB checkpoint streaming flows through the global page cache and evicts the table it is about to need, so gathers fault back to NVMe mid-decode. A memory cap makes reclaim cgroup-local — the container's checkpoint reads can only evict the container's own cache, and the table stays resident. So on the disk path, always cap the serving container (48 GB is what we validated; --memory 48g --memory-swap 48g).

Accuracy is unaffected — the mapping serves the same bytes. Inside the 48 GB cap the 200-item tool-calling suite scored 78.5 with zero request errors and zero truncations (repeat spread on this suite: 78.0–80.5), and the generation-sanity gate passed on the first-boot and capped configurations. Cold cache is a floor, not a steady state: the two cold @ 32 runs went 134 → 290 tok/s back to back as the cache refilled. Boot times share one caveat: all were measured with the checkpoint at least partially page-cache-resident; a truly cold first read of the 172 GB weights adds its own disk time to any of them.

The overlay targets this exact image. The same change is a draft PR to vLLM — vllm-project/vllm#54070, branch feat/ple-disk-offload — stacked on the PLE CPU-offload PR (vllm-project/vllm#53899).

Known image bug, fix included. The current image (sha256:fc120ece…) predates an upstream fix for a startup race in the PLE offload path: vLLM can hang right after CUDA graph capture, looping No available shared memory broadcast block found in 60 seconds (vllm-project/vllm#53960, fixed upstream 2026-08-29). This repo ships the fixed connector as connector_mrv2.py — mount it alongside any of the serve commands here, with or without the other overlays:

  -v $PWD/connector_mrv2.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/connector.py:ro ```

Verified on this build with the fix mounted: normal boot, sanity gate passed, accuracy and
throughput unchanged.

---

## Quantized PLE tables: 49 / 32 / 28.8 GB instead of 95 GB

The table itself also quantizes well. We publish it in **FP8 per-row (49 GB)**, **INT4
group-16 (32 GB)**, and **NVFP4-style e2m1 group-16 (28.8 GB)**, served memory-mapped by a
two-file overlay — host RAM cost becomes page cache only, no container cap needed. Accuracy
holds on both suites for all three (knowledge 92.2 / 92.9 / 92.2 vs 92.2 BF16; tool-calling
n=3 means 77.7 / 78.2 / 78.7 vs 79.2, one ±1.5 band), throughput lands within 5–6% of the
in-RAM BF16 baseline, and MTP keeps most of its speed-up (129.6 tok/s single-stream with the
INT4 table, 128.8 with NVFP4, vs 142.6 in-RAM). Validated end to end inside a 48 GB container:
tool-calling 80.5, 79.4 / 486 tok/s — a 64 GB-RAM host serves this model. Tables, overlay
files, serve command, format spec, and the full measurement table:
[primitive-ai/Qwen3.8-Flash-Next-PLE-quant](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant).

---

## Speculative decoding (MTP)

The MTP tensors are preserved byte-identical, so vLLM's built-in draft path works — add:

```bash
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Real-prompt A/B, single stream, thinking on (60-item subset of our eval, decode rate = output tokens over wall time — not comparable to the bench-serve numbers above, and measured on real prompts because random-token benches overstate speculative gains):

speculative config	decode tok/s	strict score
none	91.2	91.7
num_speculative_tokens: 1	does not boot	—
num_speculative_tokens: 2	133.2	93.3
num_speculative_tokens: 3	142.6 (+56%)	93.3
num_speculative_tokens: 3, table on NVMe	77.5–82.3	93.3

Score differences are single-run noise on 60 items; the speed difference reproduces. num_speculative_tokens: 1 hangs this image at startup — the engine core blocks in shm_broadcast before the API server binds, reproduced on three boots (one waited 80 minutes). Use 3.

---

The other 4-bit quants, measured on the same box

Every serious vLLM-lane quant of this model is experts-only 4-bit. We ran the four public ones under our exact protocol (one RTX PRO 6000, 200-item tool-calling suite, cache-free 8K/512 bench, table in host RAM):

build	expert format	tok/s @ 1	tok/s @ 32	TTFT @ 1	tool-calling
our mixed build	NVFP4 g16 + FP8 attention	84.4	520	570 ms	79.2 (n=3)
Intel AutoRound	INT4 g128 sym, iters=200	82.6	482	683 ms	80.5 (n=1)
lvkaokao RTN	INT4 g128 sym, RTN	82.4	481	683 ms	80.0 (n=1)
cyankiwi AWQ	INT4 g32 asym, AWQ	81.6	422	680 ms	78.0 (n=1)
wtdcode AWQ	INT4 g128 sym, AWQ	82.6	484	682 ms	76.0 (n=1)

Single tool-calling runs carry a ±1.5 spread, so the middle of this column is one band; the AWQ-vs-RTN-family gap at the edges is larger than that. Two patterns hold regardless: every 4-bit expert format decodes at ~82 tok/s single-stream with ~680 ms TTFT — the speed difference in the top row is the FP8 attention, not the expert format — and the W4A16 builds run on pre-Blackwell GPUs, which NVFP4 does not serve natively. All four keep the n-gram table in BF16, so the quantized-table sidecars apply to them as well.

MTP and the disk-backed BF16 table do not combine well: speculation multiplies table-gather traffic, the working set outgrows the page cache, and the +56% collapses to roughly the no-speculation rate (capped 77.5, uncapped 82.3). With the BF16 table in RAM, use MTP; on a low-RAM host, pair MTP with the INT4 quantized table instead, which keeps 129.6 tok/s.

---

Measured

1,370 items across fourteen public benchmarks, identical protocol to the sibling card: the 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, 16,384-token budget, no reasoning parser, last ANSWER: scored, concurrency 32, one RTX PRO 6000 Blackwell. Auto-scored, no LLM judge.

build	size	overall	knowledge	call	abstain	runs k/a	finished	out/answer	tok/s @ 32	tok/s @ 1
this repo (v2)	183.7 GB	90.3	92.2	84.8	56.7	1/3	99.5%	686 tok	526.0	84.4
v1 of this repo (QSA-only FP8)	185.8 GB	90.3	92.2	85.0	56.7	1/3	99.5%	646 tok	491.8	76.4
our plain NVFP4	186.4 GB	90.2	92.2	84.6	56.7	2/3	99.4%	664 tok	483.8	74.4

overall pools both suites (1,370 items, 85.4%/14.6% by count). call is accuracy on the 160 tool-calling items requiring a call; abstain the 40 where calling nothing is correct — reported separately, never blended. Accuracy across all three rows is one tie; the throughput column is the real difference and reproduced across seeds to 0.2%.

Tool-calling is a mean of 3 runs per build; knowledge single runs sat within 0.1 of each other across all three builds. Throughput is prefix-cache-free with distinct seeds per run. Repeat spread on this suite runs to ±0.5 knowledge, ±1.5 tool-calling.

---

Which of the two should you use

Both serve identically on one 96 GB card and score identically. This one is 13% faster single-stream, 8.7% faster at concurrency 32, and 2.7 GB smaller; the plain build carries the modelopt-format metadata some non-vLLM tooling expects and needs one fewer env var. If you are serving with vLLM, use this one.

---

What's quantized to what

tensors	format
all 48 layers' routed experts (gate/up/down_proj, 120.8B params)	NVFP4 (group 16)
QSA full-attention q/k/v/o_proj (12 layers)	FP8 E4M3, per-channel
GDN linear-attention in_proj_qkv / in_proj_z / out_proj (36 layers)	FP8 E4M3, per-channel — requires VLLM_GDN_DECODE_KERNEL=triton
n-gram embedding table (51.2B, 128 shards)	BF16, pre-scaled — the offload worker loads no other format
MTP, vision, embeddings, lm_head, shared experts, routers, norms, indexers	BF16, byte-identical to the source

Weights-only round-to-nearest, no calibration. v2 (2026-08-27) moved the GDN projections BF16 → FP8 after the kernel workaround was found; v1 (QSA-only FP8) remains in the repo history.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from one fixed protocol.

The same 1,370 items: a 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call). temperature 0.6, top_p 0.95, top_k 20, thinking forced on, a 16,384-token budget, no reasoning parser, scoring the last ANSWER: in the reply. Concurrency 32 on one RTX PRO 6000 Blackwell, each model's rows in one sitting. Auto-scored, no LLM judge. Both halves are means of at least three runs per build.

model	shape	size	overall	knowledge	call	abstain	finished	out	tok/s @ 32
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5	98.9%	1097 tok	1523
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5	97.9%	1429 tok	2204
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0	99.3%	760 tok	1469
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5	99.7%	800 tok	1176
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5	99.7%	651 tok	908
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0	97.3%	995 tok	670
Qwen3.8-Flash-Next (this repo)	180 B MoE (6 B active)	183.7 GB	90.3	92.2	84.8	56.7	99.5%	686 tok	816

Read overall with finished. overall scores an answer that overran the token budget as wrong, but it cannot say whether the model needed the room or failed to stop; finished and out separate those. A gap under 1.0 is a tie. The tok/s column comes from each model's own sitting and drifts a few percent between sittings, so read it as a bracket.

call and abstain are the tool-calling suite's two halves, reported separately. call is accuracy on the 160 items that require a tool call; abstain is the 40 whose correct action is to call nothing. They used to be pooled into one agentic number, and the pooling misled: a model with ordinary call accuracy and unusual abstention discipline outscored models that are better at actually making calls. Weight them by your own workload's mix.

Per-benchmark detail, both halves

benchmark	Laguna-XS-2.1	Nemotron-3.5-Lightning-30B-A3B	Ornith-1.5-35B-A3B	Muse-Glimmer-30B	Qwen3.8-27B	Laguna-S-2.1	Qwen3.8-Flash-Next
knowledge
mmlu_pro	79.0	82.0	89.5	89.0	89.5	81.0	89.0
math_500	76.0	80.0	88.0	74.0	82.0	83.0	86.0
gsm8k	97.0	96.0	99.0	98.0	98.0	99.0	97.0
arc_challenge	94.0	97.3	98.7	98.0	98.0	96.0	98.7
hellaswag	70.7	78.0	86.7	83.3	82.0	76.7	88.7
winogrande	84.7	89.3	89.3	90.7	89.3	86.0	95.3
commonsenseqa	80.8	83.3	86.7	82.5	84.2	85.8	86.7
boolq	90.0	92.0	92.0	90.0	91.0	92.0	94.0
openbookqa	94.0	99.0	98.0	97.0	97.0	94.0	96.0
tool calling
bfcl_v4	83.3	96.7	76.7	86.7	96.7	90.0	96.7
xlam	64.4	73.3	55.6	66.7	77.8	48.9	82.2
toolace	37.8	71.1	71.1	64.4	68.9	48.9	73.3
glaive	87.5	97.5	92.5	100.0	100.0	85.0	100.0
when2call	75.0	77.5	52.5	55.0	55.0	75.0	55.0

---

_{primitive · more models · inference economics for production LLM systems}