Rabbit-Hole-Ai/DeepSeek-V4-Flash-0731-MXFP4-GGUF

DeepSeek-V4-Flash-0731 — MXFP4-expert trunk GGUF for one RTX 5090 + a big-RAM workstation

A single-file GGUF of deepseek-ai/DeepSeek-V4-Flash-0731, converted in-house and served at 196,608 context on one 32 GB RTX 5090 with the MoE experts on CPU. 145.63 GiB / 4.400 BPW / 284.3 B trunk parameters.

• The routed experts are bit-exact to the release. 0731 ships "expert_dtype": "fp4"; the conversion repacks those weights into GGUF MXFP4 losslessly. There is nothing left to quantize in the 94% of the model that matters, so there is no imatrix and no expert-fidelity trade to argue about.
• Everything else is Q8_0 (attention/dense FFN/shared expert), with token_embd + output kept at BF16 and norms at F32.
• 26.4 tok/s single-stream generation, ~28.3 GB peak VRAM, on one 5090 + a 24-core CPU (measured 2026-08-17 — full methodology and the unflattering numbers are below).
• Requires ik_llama.cpp for this configuration — the 192K context only fits because of ik's --swa-compress. Stock llama.cpp will load the file but cannot reproduce the serving envelope. Minimum ik build: upstream main c46ffaa5 (reports version: 4821 (c46ffaa5)) or later.

Who this is for

You have one 32 GB GPU and a workstation with a lot of RAM (this was solved on 255 GB; the weights alone are ~146 GiB and the MoE tensors live in system memory). You want a 300B-class agentic model at near-release fidelity and a real 192K context, and you are willing to accept ~26 tok/s to get it.

If you have less than ~192 GB of RAM, or you are not running ik_llama.cpp, this file is not the right pick — take a smaller-BPW community quant instead.

What the quant actually is

ggml type	tensors	size	what
MXFP4	129	~137.1 GiB (94%)	routed experts — bit-exact repack of the release's native FP4
Q8_0	365	~5.8 GiB	all non-expert weights ("Q8-rest")
BF16	190	~2.6 GiB	token_embd + output
F32	641	~0.1 GiB	norms
I32	3	~0	ffn_gate_tid2eid hash routing

Three honest notes on the recipe:

1. "Trunk" means the MTP/DSpark drafter tensors are not in this file. 0731 replaced the Preview's 1-block MTP head with a 3-block DSpark drafter; the conversion skips all 4,705 mtp.* tensors, and the arithmetic checks out exactly. A separately converted DSpark drafter was measured and rejected — draft acceptance collapses at block positions ≥ 2 for architectural reasons, not weight reasons. No speculative decoding is recommended here (see Serving notes).
2. --outtype only governs the non-expert tensors. The converter writes MXFP4 experts unconditionally because the release already is FP4. So "Q8-rest" is a description of the 6% you can actually choose, not of the model.
3. BF16 non-experts were tried and rejected. BF16-rest costs −14.7% generation throughput and +5.2 GB VRAM for a KL divergence improvement of 0.005294 mean vs the Q8_0 trunk — real, but with a median of 0.000174 (half of all tokens untouched) and invisible in perplexity. Q8_0 is the ship.

External correctness reference: built with BF16 non-experts, this same conversion path produces a tensor table byte-identical to Unsloth's UD-Q8_K_XL on all four tensor types (MXFP4 experts 147,169,738,752 bytes on both sides, 1,328 tensors). That is an independent check on the conversion, not a self-graded one.

⚠️ The YaRN flags are REQUIRED, not optional

The conversion lost all five deepseek4.rope.scaling.* metadata keys. Without them YaRN is inert on the compressed-attention path and long context is silently wrong. You must pass rope scaling on the command line:

--rope-scaling yarn --rope-scale 16 --yarn-orig-ctx 65536 --yarn-beta-fast 32 --yarn-beta-slow 1

Verify it landed: the load banner's llama_init_from_model: freq_scale must read 0.0625, not 1. Nothing else in the log tells you.

Serving

Tested configuration, ik_llama.cpp upstream main c46ffaa5 or later, CUDA, one RTX 5090:

llama-server \
  -m ./dsv4f-0731-trunk-q8rest-mxfp4moe-00001-of-00004.gguf \
  -c 196608 --swa-compress \
  --ctx-checkpoints 2 --ctx-checkpoints-tolerance 5 --ctx-checkpoints-interval 0 \
  --jinja --chat-template-file ./dsv4f-0731-chat-template.jinja \
  --chat-template-kwargs '{"reasoning_effort":"high"}' \
  -t 24 --parallel 1 \
  --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
  -fa 1 -ctk f16 -ctv f16 -ngl 99 --n-cpu-moe 39 -b 7168 -ub 7168 \
  --rope-scaling yarn --rope-scale 16 --yarn-orig-ctx 65536 \
  --yarn-beta-fast 32 --yarn-beta-slow 1 \
  --spec-type none

-t 24 is the physical core count of the test machine — set it to yours.

Serving notes, in order of how much they matter

• --swa-compress is the enabling flag, not a tweak. Without it, this exact configuration loads to 31,856 of 32,607 MiB, spills, and collapses to PP 93 t/s / TG 14.8 t/s. With it, the same configuration sits at 27,009 MiB and runs normally. It compacts all 43 layers, not just the two sliding-window ones. Look for SWA compress = 43 of 43 layers compacted in the load banner; if the line is absent the flag did not land. Quality cost: greedy output was byte-identical with the flag on and off on the same binary, and the configuration was deterministic 18/18 across three depths.
• --n-cpu-moe 39 puts all 39 MoE layers' experts on CPU (4 dense + 39 MoE = 43 layers). Moving one layer back to GPU costs 3,264 MiB and buys ~+2.3% generation — measured, and a bad trade.
• -b/-ub 7168 is where the --swa-compress saving was spent instead: +31.1% prefill at 90K depth (995 → 1,305 t/s) with generation flat. -ub 8192 is faster still but leaves under 1.3 GB free, which is not safe if a desktop shares the card. Note --swa-compress rows = n_ubatch + 256, so ubatch drives KV size too.
  • Expected and benign: changing -ub changes greedy output text. Prefill chunk size changes the floating-point reduction order. Each configuration is self-consistent (3/3 byte-identical across reloads); this is not non-determinism.
• Context checkpoints are worth turning on for interrupt-then-resend clients (agentic editors). --ctx-checkpoints 2 --ctx-checkpoints-tolerance 5 --ctx-checkpoints-interval 0 takes exactly one checkpoint per turn (~350 ms). On a cancelled 16K generation followed by a divergent resend: 13.37 s of re-prefill → 0.98 s, a ~13.6× speedup on that pattern. Cost measured at −0.8% to −4.1% prefill, generation flat to +3.6%, and a byte-identical compute buffer.
• Speculative decoding is off — for three stacked reasons, not one.
  1. This file has no drafter. 0731 replaced the Preview's MTP head with a 3-block DSpark drafter, and the trunk conversion deliberately excludes all 4,705 mtp.* tensors (see the recipe notes). No flag can enable MTP/DSpark against this GGUF.
  2. It was measured anyway, and it loses where this build lives. With a separately converted DSpark drafter, acceptance collapses at block positions ≥ 2 for architectural reasons. A 7-stage --spec-type matrix found the best drafter-free stage (ngram-map-k) gains +22–27% at 16K depth but degrades to +6.2% on copy-heavy work and −15.8% on deliberative reasoning at 90K (acceptance 0.48). For a long-context reasoning workload that is a net loss.
  3. The flags conflict regardless: ik refuses MTP together with --swa-compress, and --swa-compress is what makes 192K fit on 32 GB. So --spec-type none ships on merit, not just compatibility.
• Usable context was measured, not assumed: 4 mechanical pass/fail task types × 5 depths × 5 reps, 120/120 passed out to a real n_past of 187,118. The 196,608 allocation is not decorative.
• KV cache f16. q8_0 KV is smaller but adds graph splits; the KV cache is a rounding error next to the compute buffer here (311 MiB KV vs 5,254 MiB compute at this configuration).

Benchmarks — read them honestly

Measured 2026-08-17 on: RTX 5090 (32 GB) + AMD Threadripper PRO 9965WX (24c/48t) + 255 GB DDR5, Windows 11, ik_llama.cpp version: 4821 (c46ffaa5), the serving command above.

metric	value	conditions
Generation	26.43 tok/s	median of 3 (26.50 / 26.43 / 26.42), 512 tokens out
Prefill	174.5 tok/s	at ~420-token prompt depth — see the caveat below
Prefill (deep)	~1,305 tok/s	at 90K depth, batch-saturated, -b/-ub 7168
TTFT	3,055 ms	~420-token prompt, streamed, first non-empty content chunk
Peak VRAM	28.31 GB	whole-card nvidia-smi, includes ~605 MiB desktop baseline
Peak GPU power	126.7 W	max of 0.5 s samples during generation
Speculative decoding	none	--spec-type none

llama-bench baseline for cross-checking, ik defaults (that binary supports neither -c nor --swa-compress nor the YaRN flags, so it is not the serving configuration): pp512 295.9 / tg128 27.3, r=3.

Methodology. Per model: a llama-bench pp512/tg128 ×3 no-spec baseline, then a throwaway llama-server using the exact serving flags except sampling forced greedy (temp 0 / top-k 1 — production serves temp 1.0; disclosed): a 96-token warmup on unrelated text, then 3 non-streamed measured runs on 3 distinct ~420-token prompts (expository / narrative / technical), each run the model's first sight of its prompt (cache_prompt: false, fresh server, no prompt reuse); medians published. A single streamed TTFT run last, so it cannot warm the measured runs. Peak VRAM and power from 0.5 s nvidia-smi sampling during generation.

Caveats that matter more than the headline:

• 26.4 tok/s is single-stream at shallow depth. At 90K depth generation measures 25.2 tok/s — the decay is graceful, but it is a decay, and this number is one request at a time with --parallel 1. It is not a throughput figure.
• The two prefill numbers are both true. 174 t/s at a 420-token prompt is a latency measurement with a 7,168-token ubatch about 6% occupied; ~1,305 t/s at 90K depth is the same configuration saturated. Quoting either one alone misrepresents the model.
• 126.7 W is not a typo and not an efficiency claim. Generation is bound by CPU expert GEMV over the memory fabric; the GPU is mostly idle during decode. Your CPU and memory bandwidth, not your GPU, set the generation speed of this file.
• No quality benchmark is published here. Fidelity is argued structurally (bit-exact experts, measured Q8_0-vs-BF16 KLD on the 6% that is quantized), not from a perplexity or eval sweep against other people's quants. Treat any comparison to other DeepSeek-V4-Flash GGUFs as unmeasured.

Chat template

0731 ships no Jinja chat template — the release uses a Python encoder (encoding_dsv4.py) instead. Serving it with a Preview-era template is a real quality loss unrelated to quantization, so a template is bundled here.

It is tarruda's llama.cpp PR #26398 template — an independent implementation validated by its author against the release's own encoder — with one term removed (or ns.has_tool_calls in the keep_reasoning expression). The reference encoder keys reasoning retention on a message carrying a tools schema; the upstream template also kept it when any message had role == "tool", a tool result, which diverges on conversations that replay tool results without resending schemas.

Cross-checked against the reference encoder: 18/19 shared cases byte-identical, 4/4 on the gap cases the shared suite does not reach (zero-argument tool calls, ## Response Format: blocks, drop_thinking as a real parameter, and the tool-result case above).

reasoning_effort is a real per-request parameter with the ladder low | high | max, passed via --chat-template-kwargs or the chat kwarg. Silent failure mode: the low prompt is the empty string, so a malformed value looks completely normal while your requested effort is quietly not applied. Verify via /apply-template and confirm the expected reasoning-effort prefix is present.

Files

File	What it is
dsv4f-0731-trunk-q8rest-mxfp4moe-0000N-of-00004.gguf	the model in 4 shards (llama-gguf-split; 145.63 GiB, 4.400 BPW, 284.3 B trunk params). Point -m at shard 00001 — llama.cpp loads the rest automatically
dsv4f-0731-chat-template.jinja	the chat template described above

Credits

This is a repack and a serving envelope. The hard parts are other people's work:

• DeepSeek — DeepSeek-V4-Flash-0731, the model and its weights, released under the MIT License. The routed experts in this file are their FP4 weights, bit for bit. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
• ggml-org / llama.cpp — the GGUF format, convert_hf_to_gguf.py's DeepseekV4Model and the MXFP4 expert-repack path that produced this file. https://github.com/ggml-org/llama.cpp
• Iwan Kawrakow (ikawrakow) and the ik_llama.cpp contributors — the fork this configuration requires, including its DeepSeek-V4 CUDA path and MXFP4 support. https://github.com/ikawrakow/ik_llama.cpp
• joelfarthing — --swa-compress (ik_llama.cpp PR #2266, "deepseek4: compacted sliding-window KV cache"), the single flag that makes 192K context fit on a 32 GB card. https://github.com/ikawrakow/ik_llama.cpp/pull/2266
• tarruda — the DeepSeek-V4-Flash-0731 chat template (llama.cpp PR #26398), which the bundled template is derived from. https://github.com/ggml-org/llama.cpp/pull/26398
• Unsloth — their UD-Q8_K_XL GGUF served as the independent external reference that validated this conversion's tensor table byte-for-byte. https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF

Converted, tuned and measured on a single RTX 5090 with a Threadripper PRO 9965WX, Windows 11, CUDA 13.3.

License

MIT, inherited from deepseek-ai/DeepSeek-V4-Flash-0731.