kaushikvira/Qwen3.8-Flash-Next-swift15-nvfp4-basalt

Swift-1.5 Qwen3.8-Flash-Next — NVFP4 for the Basalt engine (single-file pack)

A quality-first NVFP4 pack of ukisai/Swift-1.5-Qwen3.8-Flash-Next (~130 B MoE) for the Basalt engine (Strata fork, sm_120 Blackwell). The experts that don't fit in VRAM stream from pinned host RAM; the PLE n-gram table lives in RAM. One file, one command: basalt serve --config config.json with the OpenAI/Anthropic-compatible API.

Why this exists: Flash-Next is a ~130 B MoE — it does not fit a 72 GB card in BF16, and even NVFP4 weights alone exceed it. Basalt's RAM-expert offload makes it servable on a single consumer Blackwell card, and NVFP4 is the right quant for that path: Blackwell has native FP4 tensor cores, MoE decode is memory-bandwidth-bound, so the higher bit-width costs nothing in speed and buys a large quality margin over the 3-bit tier.

Who this is for

• You have a Blackwell card (sm_120, RTX 50 / PRO 5000 class) with 72 GB VRAM and a lot of fast RAM (the box we measured on: 246 GiB DDR5, 323 GB/s measured bandwidth). The engine is Blackwell-only in practice (sm_120 kernels).
• You want the Swift 1.5 reasoning-efficiency profile (63.4 % fewer thinking tokens than base Flash-Next at <1 % accuracy loss) at 4-bit weight cost, with the quality headroom to spare.
• You serve long-context / agentic traffic on a shared card. The KV cache is 4-bit (k8v4) and elastic, and idle conversations back out to RAM — see KV cache: k8v4, elastic VRAM, and the host K/V path, and pick a profile in Deployment setups.

It is not a drop-in for NInfer/vLLM/llama.cpp — the .basalt container format is Basalt-specific. If you need the raw NVFP4 weights, the source checkpoint is ukisai/Swift-1.5-Qwen3.8-Flash-Next-NVFP4.

Credits & provenance

All credit for the model itself goes to the authors below — this is a format conversion plus packaging of their work, with no fine-tuning, no abliteration and no distillation of our own:

Role	Model	Author
Source weights (NVFP4)	ukisai/Swift-1.5-Qwen3.8-Flash-Next-NVFP4 (gated)	ukisai (UkisAI)
Swift 1.5 finetune (BF16)	ukisai/Swift-Qwen3.8-Flash-Next	ukisai (UkisAI)
Upstream base	Qwen/Qwen3.8-Flash-Next	Qwen team
MTP drafter	base Qwen/Qwen3.8-Flash-Next BF16 MTP tensors (SHA256-verified at pack build)	Qwen team
Engine	jesdga95/basalt (Strata fork) + kaushikvira/basalt kv-host-offload patch	jesdga95 / Strata; kaushikvira

Swift 1.5 is UkisAI's reasoning-efficient derivative of Qwen3.8-Flash-Next: RL + OPD post-training aimed at long-horizon agentic and coding work, with pathological overthinking penalized.

What we did (approach — full recipe in RECIPE.md)

The problem. Flash-Next is a ~130 B MoE. Even at NVFP4 the weights are ~122 GB — more than a 72 GB card holds. Basalt solves this with resident experts: a pinned-RAM copy of every expert, and a VRAM cache that holds the hot subset. Decode is memory-bandwidth-bound on a MoE, and our box's RAM bandwidth (323 GB/s measured) is a large fraction of the card's HBM bandwidth (1124 GB/s measured), so streaming the cold experts from RAM is cheap — the quality question is purely which weight format the hot path uses.

The quant choice: NVFP4, not 3-bit. Blackwell has native FP4 tensor cores, so NVFP4 (4.5 bpw effective with block scales) costs nothing in speed vs a 3-bit format on this hardware — while carrying ~25 % more bits per weight. Measured against a Q8_0 reference pack of the same model: KLD 0.0680 vs 0.1547 for the IQ3_S tier (2.3× closer to the reference), top-1 agreement 91.46 % vs 87.25 %. The 3-bit pack remains the speed/size fallback if you need 81 GiB instead of 122 GiB; on this card the NVFP4 pack is the no-regret default.

The build, step by step:

1. Section layout from a sibling pack. The Basalt container format needs a section layout (which tensors go where, in which storage format). We built a GSQ-RCO IQ3_S pack of the same model first, then used it as --sections-from for the NVFP4 pack — the layout is model-shaped, not quant-shaped, so one layout serves both. This avoids hand-authoring the layout for a 131-shard MoE.
2. Packed the ukisai NVFP4 checkpoint into the container with tools/convert/convert.py. Every expert/dense tensor is the ukisai NVFP4 value, re-stored in the container's NVFP4 section format — no re-quantization, no calibration, no weight modification.
3. PLE n-gram table bridge. The model's PLE (n-gram) table is 102 GB in BF16 — too big to ship as-is and too big for the pack's Q8_0 section directly. We quantized it to e4m3 in a 16-worker bridge pass (tools/basalt_ple_fp8.py); the converter then re-quantizes the e4m3 values to Q8_0 inside the pack (~48 GB). Two gotchas: the e4m3 NaN codes (0x7F/0xFF) must be skipped — the converter decodes 0x7F as NaN (standard e4m3fn), so mapping the table's max value to 0x7F would put a non-finite value in the table; and the bridge is single-threaded per shard on purpose (OpenBLAS thread contention made it slower with more threads).
4. Embedded the MTP drafter — the base Qwen/Qwen3.8-Flash-Next BF16 MTP tensors, SHA256-verified at fetch time. The drafter is base-model (not Swift-tuned); measured draft acceptance is 77 %, which is what spec: 8 exploits for the ~1.5–2.5× decode.
5. Embedded the Swift 1.5 chat template from the checkpoint — thinking on by default, reasoning_effort default xhigh, preserve_thinking on. The template is part of the SHA-verified pack, so serving behavior is reproducible from the artifact alone.

Verification. Boot + basalt-bench --quality (KLD scan vs the Q8_0 reference pack), the 12-position needle gate, and the 4-prompt quality probes — all green at the numbers in Benchmarks. The full GPQA-Diamond run (792 samples) is the task-accuracy check.

Artifact

File	Format	Size	SHA-256
Swift-1.5-FN-NVFP4-v2.basalt	.basalt v2 — default, needs Basalt engine ≥ v0.2.0	131,677,782,416 bytes (122.6 GiB)	1c8d3cb996ac0771edb3b5eb45ca4528d6aebd0a1a0f7cb0633b007daa06db2e
Swift-1.5-FN-NVFP4.basalt	.basalt v1 — legacy, for engine < v0.2.0	130,966,848,307 bytes (121.9 GiB)	f1e2acdf085b2080573d6e2fa7a6bc5c0b0f9a5acbbc4d644c12b6c5359524c1

Both files carry the identical weights: the v2 file is a byte-for-byte section copy of the v1 pack with the 8-bit hyper-connection table (hc8) precomputed into the file and the learned expert profile dropped (the engine ships its own; --expert-profile-save still places a learned one) — see the file format changelog. Components in both: Text + Vision (mmproj) + MTP drafter + PLE n-gram table (Q8_0) + Swift 1.5 chat template. Hash manifest: SHA256SUMS. Build recipe: RECIPE.md. Serving config: config.json.

Serving

Any Basalt build with the sm_120 kernels (we used the kaushikvira/basalt main build branch = upstream mirror 876e47e8 + kv-host-offload patch → ba8a25f5, Release, CUDA 13.1, in Docker). The v2 pack needs Basalt ≥ v0.2.0 (upstream a04edb96); the v1 file is what older sm_120 builds read. The config below is the one we ran; config.json in this repo is the same file with the model path pointed at the downloaded pack.

{
  "server": {
    "model": "Swift-1.5-FN-NVFP4-v2.basalt",
    "gpu": "0", "host": "127.0.0.1", "port": 8090,
    "parallel": 1, "slot-full-context": true,
    "vision": "dynamic",
    "reasoning-budget-tokens": 65536,
    "thinking-loop-tokens": 1024
  },
  "engine": {
    "expert-cache": "auto",
    "prefill": "auto",
    "spec": 8,
    "max-context": 262144, "kv": "k8v4",
    "resident-experts": true, "vram-reserve-mib": 2048,
    "prompt-cache": 16,
    "conversation-cache-mib": 65536, "conversation-cache-slots": 32,
    "conversation-cache-min-free-mib": 8192
  }
}

• spec: 8 — MTP speculative decode (the pack ships the drafter); ~1.5–2.5× decode, 77 % draft acceptance measured.
• expert-cache: auto — the engine pins as many experts to VRAM as fit and streams the rest from pinned RAM; with a hard VRAM cap it sizes the pin to the cap automatically.
• kv: k8v4 — 4-bit KV (0.77× the bytes of int8 KV). Only 12 of 48 layers (QSA) grow KV, so KV stays small: ~0.6–3.3 GiB elastic.
• reasoning-budget-tokens: 65536 — the template defaults reasoning_effort to xhigh and thinking is on by default, so the budget matches; answer-room (on) still reserves cap/4 for the answer.
• vision: dynamic — image input at ~0 standing VRAM (borrows expert-cache slots only while encoding an image, ~100 ms, then refills from RAM).
• preserve_thinking is ON by default (chat-template variable, no server flag): every past assistant turn's thinking block is re-injected on each request — reasoning continuity + a byte-stable prefix for the prompt cache. Pass "preserve_thinking": false per request for long agentic sessions if context gets tight.

Deployment setups — choose by workload

The pack is one file; the card economics change with how many streams you run and how deep your contexts go. Two engine profiles, both on the same build; the lever between them is where the deep-context KV lives.

	Setup A — deep context, one stream at a time	Setup B — concurrent streams, VRAM-first experts
Workload	agentic coding sessions, single-user pilots, long-document chats, needle-heavy retrieval	several users/agents at once, chat serving, batch scoring; contexts mostly < 100 k
kv	k8v4 (4-bit, 0.77×)	int8 (1×, marginally fastest) or q4_0 (smallest)
max-context	262 144	131 072 (or 262 144 at parallel: 2)
parallel	1	2–4
host K/V (mid-decode stream pin)	off by design — the feature serves int8/fp16/q4_0 states only; under k8v4 a 99 k+ request keeps its K/V in VRAM (see the KV section)	active — a request decoding at ≥ 99 328 tokens pins its K/V in host RAM; attention reads through a 32 768-cell VRAM pool
Context at rest between turns/sessions	conversation cache: idle sessions park to host RAM (64 GiB / 32 slots) and resume without a re-read — independent of the kv mode and of host K/V	same, plus the streamed sessions' pins
expert-cache	auto (sizes to its VRAM slice; needs the hard cap in co-residency)	auto, same
Effect	the full 200–262 k KV stays in VRAM (elastic, 0.6–3.3 GiB); experts keep full residency	deep streams' KV leaves VRAM → more expert slots; resident KV costs ~1.3× k8v4's bytes/tok
RAM budget	~95 GiB	~95–120 GiB (the pin adds per deep stream)

Setup A — the one we run on our PRO 5000 (this is our production lane)

The config.json in this repo, verbatim. The measured contract: 225–244 t/s decode, wall 0.583 s/pass, 200 k prefill in 32.9 s, deep-context (193 k prompt) decode 24–76 t/s task-dependent, and 113 t/s / 1.2 s/pass co-resident on a 45 GiB slice.

Sequential sessions, by the way, hand off with no re-read at all: session 1 goes idle → its context parks into the RAM conversation cache; session 2 starts with a prompt-cache hit on the shared prefix (it is byte-stable — preserve_thinking on); session 1 comes back → parked context returns, only the new increment is prefilled. All RAM-backed, none of it needing host K/V. "k8v4 never offloads" is about the streamed decode-time K/V pin (a --serve-only feature, int8/fp16/q4_0 modes only, gated off under --batch, --kv-resident, --layer-split, --vram-elastic), not about where a resting conversation lives.

The decision we made and why — copy the reasoning, not just the JSON: at 99 k+ we keep the KV in VRAM; we do not offload it. A 221 k request measured pcie = 0.0 % — on this architecture (GDN linear attention in 36 of 48 layers) the deep-context floor is attention compute, not KV bandwidth, so host-offloading the KV buys ~0 decode and spends PCIe you may need for experts. And the wiring enforces the choice: host K/V only applies to non-k8v4 KV types, so under k8v4 it is off by design and --exp kv_host=0 is a no-op. Do not "fix" deep-context speed by moving KV to RAM — we measured it. Go Setup B when concurrency, not depth, is the pressure.

Setup B — sketch (not the lane we operate)

"kv": "int8", "max-context": 131072, "parallel": 2, "exp": "kv_host=1", "vram-reserve-mib": 2048

Trade-offs, plainly: every concurrent stream pays its KV bytes in the same VRAM that feeds the experts — that is exactly why we do not run it. int8 costs ~1.3× k8v4's KV bytes/token, and host K/V is what keeps a deep stream affordable — its cost is a PCIe round-trip per decode step (the pcie telemetry goes off zero) plus a streamed read that can move a short read's last bits. 16–24 GB class cards: the IQ3_S pack (81 GiB) with int8 + host K/V, accepting the quality class (KLD 0.1547 vs 0.0680). Two Blackwell cards with all-resident weights and 50+ streams is a full-VRAM-engine shape (e.g. SGLang) — outside this pack's design point, by the numbers on its own recipe.

For a shared card, in whichever setup: the lever is a hard VRAM cap, not config tuning. On Linux, cgroup-v2 dmem (driver r615+) partitions one card per container (set memory.dmem on the container cgroup); our capped co-residency measured faster than uncapped (decode +22 %, wall +44 %).

Benchmarks

Conditions: RTX PRO 5000 72 GB (Blackwell, sm_120), one GPU, Basalt ba8a25f5 (Release, CUDA 13.1), config.json above, 262 k context, thinking budget 65 536. The box: EPYC 9275F, 246 GiB DDR5 @ 323 GB/s measured, PCIe Gen5 @ 57 GB/s measured.

Quality — quantization fidelity (KLD scan)

Measured with basalt-bench --quality: 100 chunks of wikitext-2 (512 tokens each), scored through the decode path with adaptation off, against a Q8_0 reference pack of the same model. KLD/top-1 describe closeness to the Q8_0 reference; PPL is each pack's own perplexity on the same text (not a distance measure).

Pack	PPL	KLD mean (vs Q8_0)	Same top-1	Size
NVFP4 (this pack)	3.308	0.0680	91.46 %	121.9 GiB
GSQ-RCO IQ3_S (3-bit)	3.383	0.1547	87.25 %	81 GiB
Q8_0 (reference)	3.286	—	—	178.3 GiB

NVFP4 sits 0.022 KLD above the Q8_0 reference vs IQ3_S's 0.097 — 2.3× closer to the reference, +4.2 pp top-1 agreement.

Quality — task accuracy (GPQA-Diamond)

Full set: 198 questions × 4 answer orderings = 792 samples, inspect_ai + inspect_evals (gpqa_diamond, cot: true), single pass, thinking budget 16 384.

Run	Score
This pack, all 198 q (792 samples)	87.1 % ± 2.0 pp
This pack, first 50 q (200 samples)	89.0 % ± 3.7 pp
This pack, remaining 148 q (592 samples)	86.5 % ± 2.4 pp
Published Swift 1.5 BF16 (ukisai card, xhigh, 5 seeds)	89.60 %
Published base Flash-Next BF16 (ukisai card, xhigh)	89.80 %

Read this honestly: the 87.1 % full-set number is ~2.5 pp under the published BF16 xhigh number, and the first-50 subset (89.0 %) is inside the noise of a 50-question estimate. The published BF16 run used a 100 k output cap (effectively uncapped xhigh thinking); our run capped thinking at 16 384. Treat it as "within the quantization+protocol noise of the BF16 reference, clearly not a regression" — the KLD scan above is the tighter quantization signal.

Gate (correctness, zero-tolerance)

Gate	Result
Needle retrieval (12 depths/positions, 64 k)	12/12 exact, 0 corrupted
Quality probes (4 fixed prompts)	4/4

Speed

Single-stream, same card. decode_tok_s swings ±30 % run-to-run (cold draft state) — the wall-clock dimensions are the stable ones.

Metric	Solo on the card	3 workloads co-resident (45 GiB VRAM cap)
Engine decode (MTP on, spec 8)	225–244 t/s (77 % draft accept)	113 t/s
Gate bench decode (3 samples, cold)	131 t/s mean (83.7 / 135.7 / 173.8)	—
Wall clock per pass (5-task suite, median-of-3)	0.583 s	1.2 s
Tokens per pass	130.8	130.6
Prefill TTFT	12.5 k → 1.94 s · 50 k → 7.52 s · 100 k → 15.43 s · 200 k → 32.9 s	—

Co-residency is the dominant speed variable, not the config: without a hard VRAM cap the engine sizes its pin + KV against the full 72 GB and competes with its neighbors; with the cap, time-slicing is cleaner and the capped setup measured faster than the uncapped one (decode +22 %, wall +44 % in our controlled A/B).

Memory layout (measured, live)

VRAM (72 GB card, 45 GiB hard cap for this workload):
  expert-cache ~35.7 GiB (auto-pinned to the cap) + dense/head/MTP
  + KV k8v4 elastic 0.6–3.3 GiB
RAM (of 246 GiB):
  PLE n-gram table (Q8_0 in pack, ~48 GB) + expert tail (pinned)
  + conversation cache 64 GiB (32 slots)

The RAM requirement is real: budget ~95 GiB of host RAM for this workload (PLE + expert tail + conversation cache). The conversation cache is what makes 262 k context practical on a 72 GB card — idle conversations migrate out of VRAM after 15 s.

KV cache: k8v4, elastic VRAM, and the host K/V path

Three mechanisms, and one fact about how they compose that decides Setup A vs B above. The host path comes from the kv-host-offload patch on kaushikvira/basalt; the other two are the Strata-fork engine's:

• k8v4 KV quantization — INT8 keys (unrotated, exact scores) + rotated 4-bit values, 0.77× the bytes of int8 KV. Only 12 of 48 layers carry KV (QSA — the GDN layers keep a fixed recurrent state), so a full 262 k context costs ~3.3 GiB instead of ~4.3.
• Elastic KV — with max-context above 32 768, each session backs its first 32 768 cells in VRAM at startup and the rest of the KV pool is lent to the expert cache until a request needs it; a request backs its prompt + 16 384 cells before the prompt is read. A short request decodes with the cache a 32 k context would have, whatever max-context is.
• Host K/V (--exp kv_host) — and k8v4 turns it OFF by design. A request decoding at 99 328 tokens or deeper keeps its K/V in pinned RAM, attention reading through a 32 768-cell VRAM pool, so the expert cache keeps its tail slots. The pin is taken only when it leaves the RAM guard's floor free. The engine enables it by default only with a KV other than k8v4 (verified in ba8a25f5 generate/sessions.cpp): under k8v4 the full 200–262 k KV stays in VRAM (elastic) with experts resident, and --exp kv_host=0 is a no-op. It is also off with --parallel, a layer split, --kv-resident and --vram-elastic.

We run the k8v4/VRAM side of that trade on purpose: the offload's payoff is VRAM headroom for experts, and on our box 262 k of k8v4 KV fits without it, so the exchange is worthless (measured: 221 k request at pcie = 0.0 %; deep decode 24–76 t/s is compute-bound). On a tighter card, Setup B's int8/q4_0 + host K/V is the lever that keeps deep streams servable, paid in PCIe per decode step.

Net effect: the KV cache is the elastic resource on the card, not the expert cache — VRAM goes to the experts (the bandwidth-bound part), and in Setup B the deep context's KV is backed by RAM bandwidth (323 GB/s on our box).

Notes

• Quality numbers are KLD + GPQA, not a full eval suite — the KLD scan is 512-ctx wikitext fidelity; GPQA-Diamond is the task-accuracy check. If you need AIME/LiveCodeBench numbers for your workload, run them — the harness is engine-agnostic (OpenAI API).
• The MTP drafter is base-model, not Swift-tuned — it is the Qwen/Qwen3.8-Flash-Next BF16 MTP tensors. Draft acceptance is 77 %; if a future Swift-tuned drafter appears, the pack format supports swapping it.
• parallel: 1 is a throughput choice, not a limit — requests queue (FIFO, no preemption, no 429/503). Bump to 2–4 under concurrent pressure; the elastic KV makes every slot's VRAM share proportional to its context.
• Boot time is dominated by the PLE table load from disk (~48 GB) — expect a couple of minutes on fast NVMe.

Acceptable Use Policy & Legal Notice

Permitted use

Research, evaluation, personal and commercial deployment subject to the upstream licenses of the base models: Qwen3.8-Flash-Next is under the Qwen Community License 1.0 (notably: commercial "Model as a Service" / "AI Work Assistant" use requires a separate license from Qwen; >100 M MAU products must prominently display the model name), and the Swift 1.5 derivative is under the Swift Open License v1.0 as published by UkisAI. You must independently verify that your use complies with those licenses — the gated source checkpoint's access terms apply to the weights this pack was built from.

Prohibited use

No use intended to cause harm; no illegal activity; no generating content that violates applicable law; no circumventing the upstream models' own usage terms; no misrepresenting this artifact as an official release of ukisai, the Qwen team, jesdga95 or Strata.

Your responsibility

You are responsible for what you generate and how you deploy it, including any jurisdiction-specific obligations, disclosure requirements and downstream safety measures.

No warranty / limitation of liability

Provided as-is, without warranty of any kind. The authors of this conversion accept no liability for any use, output, or damage arising from it. No support commitment is implied.

Measurement conditions (for reproducibility)

Scores in the tables above were produced on a single RTX PRO 5000 72 GB (Blackwell, sm_120) with the flags quoted in Serving, on an EPYC 9275F / 246 GiB DDR5 box (323 GB/s measured RAM bandwidth, PCIe Gen5). The KLD scan is basalt-bench --quality (100 × wikitext-2 512-token chunks, decode path, adaptation off) against a Q8_0 reference pack of the same model. GPQA-Diamond is inspect_ai/inspect_evals gpqa_diamond (full 198 questions × 4 orderings, cot: true, thinking budget 16 384). Speed figures come from the engine's own bench and a 3-sample gate run on the same box; they are not comparable across different GPUs, RAM bandwidths, or with other workloads resident — the co-resident column states its conditions explicitly.