sakamakismile/DSV4-Flash-Vision-EXL3-MixedK-D2-K2x3-Dense6

DSV4-Flash-Vision — EXL3 MixedK · D2 · K2×3 · Dense6 (380K context on 8 × 16 GB)

A serving-tuned derivative of vcruz305/DSV4-Flash-Vision-EXL3-MixedK (DeepSeek-V4-Flash-Vision-Exp, 305B MoE, vision). Everything that was still BF16 or fp8 in the original has been carved into EXL3 so that a 380K-context, 4-stream, DSpark3-speculative seat fits on eight RTX PRO 2000 16 GB under vLLM. Baked and measured by Lna-Lab (YUKI) on 2026-09-03; house model of SAZANAMI ("オオタニ").

Recipe, patches and the serve script: github.com/lna-lab/dsv4-carve (mirrored in lna-lab/ here).

Composition

Part	Here	Original MixedK
Routed experts	EXL3 2/3-bit mixed; layers 3, 21, 41 → K2 (wrldsuksgo2mars K2-v1)	EXL3 2/3-bit
MTP draft experts (3 layers, DSpark)	EXL3 K2 (wrldsuksgo2mars K2.2-D2-v1)	fp8
Attention + shared experts (790 linears)	EXL3 6-bit, calibrated (250 × 2048)	BF16
lm_head / embeddings / indexer weights_proj	BF16 (unchanged)	BF16

86 GB on disk, 51 shards. Per GPU at TP8: 12.0 GiB weights → 396,656 tokens of fp8 KV at --max-model-len 389120.

Measured (vLLM 0.28.1rc1.dev337 + vllm-exl3 0.2.3, TP8, CUDA graphs, RTX PRO 2000 × 8)

ppl wikitext-2 (512 × 16)	6.7159 — original MixedK 6.6271 (+1.34 %); 4-bit dense was +3.8 % and rejected
single stream, no speculation	en 42.1 / ja 42.0 / code 41.9 tok/s
single stream, DSpark3 (3 MTP tokens)	en 62 / ja 55 / code 86 tok/s
4 streams, DSpark3, aggregate	en 136 / ja 116 / code 186 tok/s
needle at 166K tokens	found; TTFT 167 s (prefill ≈ 1.0k tok/s), decode at that depth 37 tok/s
vision	works (shapes, colours, text read correctly)

2026-09-04 update — house kernel lna2 for the routed experts

Same weights, same pack. The routed-experts decode kernel was replaced by a bespoke one (lna-lab/lna2/): resident expert teams with a dynamic ticket scheduler outside, small-R inner loops (1–8 rows per expert) inside — the incumbent's M=16 tile wastes 15/16 of its MMA when each expert sees ~1 row. Adopted as the house kernel by Ken on 2026-09-04.

(TP8, DSpark3, CUDA graphs, RTX PRO 2000 × 8)	exllamav3 exl3_moe	lna2
routed kernel per launch	232.6 µs	104.0 µs (2.24×)
single stream code / en / ja	86 / 60 / 56 tok/s	108.6 / 78.6 / 71.1
4 streams code / en / ja	125–179 / 96 / 83	193.8 / 146.4 / 129.0
KV window @ --gpu-memory-utilization 0.97	396,656 tok	395,069
needle @166k	found, TTFT 167 s	found, 119.6 s
ppl (wikitext-2, 4k ctx, speculation off, matched control)	4.7630	4.7647

Launch = the recipe above plus NATIVE_SO=lna-lab/lna2/vllm_exl3_c.cpython-312-x86_64-linux-gnu.so MOE_KERNEL=lna2 and -e VLLM_EXL3_MOE_STRICT=1; the seat log must show LNA2 MoE kernel ACTIVE on every rank (the first delivery silently fell back to the incumbent and "passed" its gates — hence strict mode). Desk parity 171 cases (rel ≤ 1.64e-3, bitwise repeatable); a null-decode variant of the kernel pins the card at its measured 270 GB/s cold-read ceiling, so the remaining ceiling is the trellis decoder itself. Canon: LNA-CANON.md. Design reviews and every gate report: lna-lab/orders/. Code: github.com/lna-lab/dsv4-carve.

You need the patches

Stock exllamav3 1.4.5 / vllm-exl3 0.2.3 will not serve this pack: dense EXL3 linears in attention trigger (a) the cooperative-GEMM autotuner inside CUDA-graph capture and (b) a two-stream deadlock between vLLM's shared-experts stream and the routed EXL3 kernels (both use exllamav3's per-device lock buffer). The lna-lab/ folder carries the patched sources, the vLLM patches, the prebuilt extension for sm_120 (exllamav3_ext.cpython-312-x86_64-linux-gnu.so, torch cu13) and the serve script; the README there explains each. Minimum: VLLM_DISABLE_SHARED_EXPERTS_STREAM=1, VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128, aux streams off, --kv-cache-dtype fp8.

AUX_STREAMS=0 MODEL=. UTIL=0.97 MAXLEN=389120 BT=512 SEQS=4 \
SPEC='{"method":"dspark","num_speculative_tokens":3}' bash lna-lab/serve-dsv4-tp8.sh

Thinking is on by default; pass chat_template_kwargs: {"thinking": false} or a large max_tokens.

Files

• model-*.safetensors, model.safetensors.index.json, config.json (quantization_config.non_routed_exl3 lists the 6-bit dense modules), tokenizer, generation_config.json
• lna-lab/ — Dockerfile, serve/bench/ppl scripts, tools/ (overlay + dense bake), patches/, prebuilt extension, upstream model card

Credits and license

• Base weights: DeepSeek (MIT). This pack follows the DeepSeek model license like the upstream MixedK.
• MixedK, vllm-exl3 and its DSV4 patches: vcruz305 · K2 packs: wrldsuksgo2mars · ExLlamaV3: turboderp · vLLM
• Bake, patches, measurements: Lna-Lab / YUKI with Ken (@Tono_Ken3)