WaveCut/DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-NInfer-v3

DeepSeek-V4.1-Flash-1017 GSQ-RCO IQ1_M, NInfer v3 artifact

A NInfer v3 artifact of ISTA-DASLab's DeepSeek-V4.1-Flash-1017 GSQ-RCO IQ1_M (1.75 bits per weight over the 551.9B backbone parameters) for NInfer-all, the master branch of iamwavecut/ninfer-all. DeepSeek-V4.1-Flash-1017 is a 763B-parameter mixture of experts: 40 blocks of 384 routed experts, 6 of them active per token, beside a shared expert, with four-stream hyper-connections, latent attention over a 128-position window plus indexer-selected compressed latents, and two Engram blocks that add rows of a 196.6B parameter n-gram table, and a vision tower that turns each image into a span of prompt positions. The GGUF assigns a ggml quantization type to every tensor; this artifact keeps each tensor's ggml blocks byte for byte, and the engine multiplies the blocks in place.

Stock NInfer builds refuse this file: the model family and the gguf_* formats exist only in that line. Since October 27 the file carries the release's DSpark drafter in STQ, from beamster's Sushi pack; build master from commit 673e3e618 on, which adds the drafter's STQ product and its uint16 and fp16 objects. NInfer serves it on one 24 GB GPU with its routed experts in pinned host memory or streamed from this file, with up to eight concurrent requests, prompt-prefix reuse, reasoning, DSML tool calls returned as OpenAI tool calls, structured output, images (--vision), DSpark drafts (--spec dspark) and context-lookup drafting. With host experts, CPU threads compute a share of each decode call's missing experts while the GPU copies the rest.

The Engram table is a separate download, shared by every NInfer conversion of DeepSeek-V4.1-Flash-1017: WaveCut/DeepSeek-V4.1-Flash-Engram-IQ4_NL-NInfer-v3.

What is inside

component	representation
routed experts, 40 blocks × 384	the GGUF's blocks, one type per block and projection: IQ1_S (55 of the 80 banks), IQ1_M (12), IQ2_XXS (11), IQ2_S (2); each expert's gate rows are followed by its up rows, so one expert is one contiguous range of bytes
shared experts	Q5_K, as in the GGUF
latent attention	query and latent down-projections Q8_0; query up-projection and the grouped output projections Q5_K
router, compressors, indexers, hyper-connection matrices	FP32, widened from the GGUF's BF16 and F16, which FP32 holds exactly
norms, attention sinks	FP32, as stored
Engram projections	Q5_K; each block's gate vector is the FP32 product of the GGUF's q and k gate weights
token table, output head	Q5_K
DSpark drafter (dspark component)	the release's drafter of three blocks (mtp.0..2) from beamster's Sushi STQ pack, its matrices as the pack stores them: STQ trellis words (uint16; 2 bits a weight in the 128 routed experts of each block, 4 bits in the other matrices) and FP16 Hadamard sign vectors; FP32 vectors and the BF16 Markov tables
vision tower (vision component)	the release's tower and aligner from the BF16 mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf of ISTA-DASLab's repository: BF16 matrices; the patch embedding, norms, biases and the learned start, newline and end rows of an image's span FP32 (the same BF16 values); the release's routing bias of image positions (FP32): blk.N.exp_probs_b_vl.bias of ISTA-DASLab's current GGUFs, the values of the release's layers.N.ffn.gate.bias_vl
Engram table	not in this file: the model's engram description names its table by the SHA-256 of its rows (36ee7d37…6b41a5), published once in the table repository
tokenizer, configuration	DeepSeek's files
chat template	NInfer's transcription of the release's encoding/encoding.py; the release ships no Jinja template

One file, DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-ninfer-v3.ninfer, of 125,824,667,392 bytes (117.18 GiB), next to its conversion report, SHA256SUMS, NOTICE and LICENSE. The routed experts are 106.69 GiB of it; the GPU keeps the other 6.0 GiB, 3.58 GiB more for the drafter when it runs and 0.90 GiB for the vision tower with --vision. Conversion command for ISTA-DASLab's current files, in llama.cpp's layout (master from commit ab84dcfa0 on):

hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 config.json tokenizer.json tokenizer_config.json \
  --local-dir models/deepseek-v4.1/hf
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --revision 65d53bc8be65123e255ff02f3ded958b6dbf54fd \
  --include 'DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-*.gguf' \
  --include 'DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf' \
  --include 'mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf' --local-dir models/deepseek-v4.1
hf download beamster/DeepSeek-V4.1-Flash-1017-Sushi-STQ-1.75bpw \
  --revision 07fec972538639726b868f81e89d9d4f5005fe10 --include config.json \
  --include model.safetensors.index.json --include 'model-mtp-*.safetensors' \
  --local-dir models/deepseek-v4.1/stq
python -m tools.convert --model models/deepseek-v4.1/hf --recipe deepseek_v41_gguf \
  --components text,dspark,vision \
  --source gguf=models/deepseek-v4.1/DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-00001-of-00006.gguf \
  --source engram=models/deepseek-v4.1/DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf \
  --source dspark=models/deepseek-v4.1/stq \
  --source vision=models/deepseek-v4.1/mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf \
  --name DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M \
  --out models/DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-ninfer-v3.ninfer --device cpu

This file was converted on October 27 from the GGUFs ISTA-DASLab published before it moved to llama.cpp's layout that day (revision 653a9874: four shards and no routing bias of image positions, which the converter then read from the release's safetensors), in 6 min 23 s on the 96 threads of a 48-core EPYC 7K62 with a download writing to the same disk. On November 4 the command above, run on the current files, wrote every object, configuration and resource of this file byte for byte; only the random id and the source paths it records differ. The earlier files still convert (command).

The converter reads the Engram GGUF only for the table's digest. --components text,dspark leaves the vision tower out and --components text the drafter too; the objects they keep are the same bytes either way. With Unsloth's GGUF as the drafter's source (--source dspark=dspark-DeepSeek-V4.1-Flash-1017-Q8_0.gguf) the converter writes the file this repository held until October 27 (129,946,453,248 bytes; revision 646ce82d), whose text and vision objects are the same bytes as this file's. On October 27 this file's text and vision objects, configurations, metadata and resources were compared with that file's and found equal byte for byte; served on an RTX 3090 with host experts, --spec dspark and --vision, it answered a text, a code and an image request, the drafter accepting 53 of 81, 202 of 237 and 7 of 15 drafts.

Running

Download the model and the Engram table, then serve them from a build of master (its README):

hf download WaveCut/DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-NInfer-v3 \
  DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-ninfer-v3.ninfer --local-dir models
hf download WaveCut/DeepSeek-V4.1-Flash-Engram-IQ4_NL-NInfer-v3 \
  DeepSeek-V4.1-Flash-Engram-IQ4_NL-ninfer-v3.ninfer --local-dir models
M=models/DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ1_M-ninfer-v3.ninfer
T=models/DeepSeek-V4.1-Flash-Engram-IQ4_NL-ninfer-v3.ninfer

# One 24 GB GPU and RAM for the experts: 106.7 GiB pinned, the most used experts cached on the GPU.
ninfer-serve $M --ngram-table $T --model-id deepseek-v4.1-flash \
  --expert-residency host --max-context 32768

# One 24 GB GPU and less RAM: the experts stay in the file and stream into a GPU cache.
ninfer-serve $M --ngram-table $T --model-id deepseek-v4.1-flash \
  --expert-residency disk --max-context 32768

# The same, reading images (OpenAI image_url parts, Anthropic image blocks).
ninfer-serve $M --ngram-table $T --model-id deepseek-v4.1-flash \
  --expert-residency host --max-context 32768 --vision

# DSpark drafts, up to five a round (where they help: "The drafter" below).
ninfer-serve $M --ngram-table $T --model-id deepseek-v4.1-flash \
  --expert-residency host --max-context 32768 --spec dspark

With host experts the GPU holds the 6.0 GiB of dense weights, two whole-layer expert banks (7.1 GiB), the request state and an expert cache that takes what is free after startup (7.9 GiB on an RTX 4090; --expert-cache-mib sets it), filled at startup from the built-in routing profile; the startup reads the experts with direct I/O on eight threads and pins them (the server was ready in 34 to 43 s on the host below). CPU threads, one per physical core and at most half the usable CPUs (61 on the host below), compute a share of each decode call's missing experts from the pinned copy while the GPU copies the rest, bit for bit the GPU's products; --expert-dma-share 1 copies every miss, and disk experts copy every miss by default. Prompts run block by block in spans of up to 16,384 tokens, so each block's missing experts cross the bus once per span; a span borrows up to 1.3 GiB of the expert cache while it runs. With disk experts the host keeps no copy of the banks: each layer's missing experts are read from the file through the OS page cache into bounded transfer buffers, never through a memory map. The Engram rows are read from the table's file the same way (48 rows a token) unless --ngram-residency ram loads the 103 GiB table into RAM. A model started without its table is refused; --no-ngram-table runs it without the Engram blocks, which degrades the output (see the table's card). The caches are BF16. DeepSeek-V4.1-Flash describes the execution.

Thinking is on by default; reasoning_effort low, high (the default) or max writes the release's effort line, and enable_thinking: false or reasoning_effort: "none" turns it off. --spec dspark drafts with the release's DSpark drafter while one request decodes: up to five tokens a round (--draft-tokens 1..5), kept while its confidence is at least --draft-confidence (0.5), verified in one pass of decode rows; the tokens and log probabilities are plain decoding's. Its 3.58 GiB of device memory come out of the expert cache. In the October 27 measurements below, an RTX PRO 6000 decoded 1.28 to 2.07 times as fast with it; an RTX 4090, which caches 618 experts beside it instead of 1,137, decoded code and reasoning 1.19 and 1.21 times as fast, the other three requests 0.85 to 1.02 times, and prefilled at plain speed. Without --spec, --lookup-ngram 4 drafts from earlier occurrences in the context and keeps the plain decode's output.

With --vision the server reads images as the release's image_processor.py prepares them: an image takes a span of up to 1,024 prompt positions (a 31 × 31 grid of aligner cells for a square picture; detail: "low" a quarter of that, --vision-max-merged fewer), and the router adds the release's routing bias of image positions there. The tower's 0.90 GiB of BF16 matrices stay on the GPU, or with --vision-residency overlay its 32 blocks stay in pinned host memory and pass through two device slots (210 MiB of device memory), with the same rows bit for bit; either comes out of the expert cache. Video is refused. On an RTX 4090 the tower's output for a 55 × 37-patch picture is within a cosine of 0.981 and a relative RMS error of 2.6% of an FP64 forward over the mmproj's weights, and the model answered the repository's image examples (a chart's text and shapes, a scene's details, a comparison of two pictures) correctly.

Quality

The build that added the model (October 25) was checked against an FP64 forward of this quantization (tools/reference/deepseek_v41.py: the GGUF's blocks decoded exactly, the Engram table's rows, the release's attention roundings) on a 1,112-token WikiText slice:

	perplexity	greedy next token = FP64 argmax
FP64 reference	3.046
NInfer, with the Engram table	3.067	21 of 22 positions spread over the slice; 6 of the 10 where the FP64 top two logits are closest (gaps of 0.01 to 0.04)
NInfer, --no-ngram-table	9.68

NInfer's matrix kernels take BF16 activations rounded to ggml's q8_1 blocks, as llama.cpp's do, and keep the caches in BF16. Since October 26 the GPU's IQ1_S, IQ1_M and IQ2_XXS products round step by step as the CPU share computes them, which changes the outputs; the FP64 check was not rerun. Host experts, disk experts, the CPU share, context lookup and DSpark drafts produce the same greedy tokens and log probabilities.

ISTA-DASLab's card evaluates this quantization at AIME25 100.00, GPQA-Diamond 88.38 and LiveCodeBench v6 88.57 (reasoning effort 50; the release: 100.00, 90.91, 89.71). Those benchmarks were not rerun in NInfer.

Speed

Until October 27 this file carried Unsloth's Q8_0 drafter (7.4 GiB on the card), and the --spec dspark rows of the first two tables ran with it; the drafter compares it with the STQ drafter the file carries now.

October 25 and 26, 2026, one rented host: an RTX 4090 (450 W, PCIe 4.0 x16), two 32-core AMD EPYC engineering samples and 409 GiB of RAM. One sequence, greedy decoding, thinking off, the same prompt tokens in both engines: a 14-token question and an 8,086-token summary request. NInfer served the model with ninfer-serve --max-context 16384 --max-concurrency 1 --expert-residency host (master at 110ab168a, whose 61 CPU threads compute a share of each decode call's missing experts, and the release build of b3402a6fd, which ran prompts chunk by chunk and copied every miss). Strata 0.2.6, its latest release (strata-dsv41 0.1.51), ran the same GGUF shards and Engram GGUF through its pack index with its DSpark drafter, all 6 experts, --max-context 16384 --eos 1 --temperature 0: first with the flags Strata's app starts the engine with, then with the settings its tuner tries (each beside --dspark-conf 0.65).

engine and settings	decode, short answer	decode after 8,086 tokens	prefill of 8,086 tokens
NInfer, experts in pinned RAM	38.9 tok/s	31.4 tok/s	938 tok/s
NInfer, experts in pinned RAM, --expert-dma-share 1 (every miss copied)	33.5 tok/s	26.6 tok/s	929 tok/s
NInfer, experts in pinned RAM, --spec dspark (Unsloth's drafter)	39.9 tok/s	33.2 tok/s	693 tok/s
NInfer b3402a6fd, experts in pinned RAM	25.8 tok/s	22.1 tok/s	498 tok/s
Strata, --dspark auto --dspark-conf 0.65	36.0 tok/s	38.4 tok/s	635 tok/s
Strata, --dspark on	34.3 tok/s	38.7 tok/s	690 tok/s
Strata, --dspark on --dspark-block 3	39.9 tok/s	37.1 tok/s	660 tok/s
Strata, --cpu-threads 64	38.0 tok/s	38.6 tok/s	659 tok/s
Strata, --mmap-experts off --ssd-cache 0	39.4 tok/s	38.3 tok/s	658 tok/s

Without drafts (its --dspark auto rows drafted nothing), Strata decodes 0.93 to 1.01 times as fast as NInfer after the short prompt and 1.22 to 1.23 times after the long one; with both engines' drafters, Strata's fastest setting for each prompt decodes 1.00 and 1.17 times as fast as NInfer's --spec dspark. Strata's CPU threads compute the experts its VRAM cache misses (53 to 61% of the routed experts here); NInfer's compute a share of its misses and the GPU copies the rest. Without drafts NInfer prefills 1.36 to 1.48 times as fast as Strata, whose prompt windows are 16,384 tokens too; with the drafter its spans shorten to 8,192 tokens and it prefills at 693 tok/s. Strata's greedy answer to the short prompt ran 74 to 119 tokens over its runs; NInfer's tokens were the same in every run. With --experts 5 (5 of the 6 experts, so another model's output) Strata decodes 44.8 and 42.1 tok/s. NInfer's DSpark row accepted 43 of 68 and 16 of 31 drafts: on this card Unsloth's drafter leaves the expert cache 73 experts instead of 1,137, and it decodes 1.03 and 1.06 times as fast as plain decoding. One run per row: no spread is measured.

On an RTX PRO 6000 (600 W, PCIe 5.0 x16) with a 24-core Threadripper PRO 7965WX, October 26, the same prompts and host experts:

NInfer 436d3aaf6	decode, short answer	decode after 8,086 tokens	prefill of 8,086 tokens
experts in pinned RAM	50.8 tok/s	48.3 tok/s	1,210 tok/s
experts in pinned RAM, --spec dspark (Unsloth's drafter)	83.1 tok/s	75.0 tok/s	1,290 tok/s
experts in pinned RAM, --spec dspark --draft-tokens 3 (Unsloth's drafter)	83.3 tok/s	77.0 tok/s	1,290 tok/s

The card keeps 10,317 of the 15,360 experts with Unsloth's drafter, and 38 of 58 and 16 of 26 drafts were accepted. Strata 0.2.6 exited with router picked expert 2147483647 in all ten of its runs on that host (its app's flags, --dspark on, --dspark-block 3, --reserve-mib, --dspark off, --vram-experts 0.5), so that card has no Strata comparison.

On other cards, October 26, the native release build of ed8338c52 with the same two prompts, one host per card: host experts, then disk experts with the file in the page cache. Host and disk experts produced the same tokens on each card.

card and host	experts cached on the card	experts	decode, short answer	decode after 8,086 tokens	prefill of 8,086 tokens
RTX 5090 (600 W), Core Ultra 7 265K, PCIe 5.0 x16	2,236 (15%)	pinned RAM	51.5 tok/s	42.2 tok/s	912 tok/s
	2,266	disk	16.7 tok/s	12.0 tok/s	440 tok/s
RTX 4090 (450 W), EPYC 7K62, PCIe 4.0 x16	1,126 (7%)	pinned RAM	33.8 tok/s	27.5 tok/s	915 tok/s
	1,156	disk	9.5 tok/s	7.8 tok/s	341 tok/s
RTX 3090 (300 W), two Xeon E5-2690 v4, PCIe 3.0 x16	1,150 (7%)	pinned RAM	19.8 tok/s	15.2 tok/s	416 tok/s
	1,181	disk	6.0 tok/s	5.0 tok/s	157 tok/s

On 16 GB (an RTX 4090 held to 16 GB free) the span build of October 25 decodes 21.4 tok/s with host experts and prefills 8,086 tokens at 679 tok/s. The measurements add longer prompts and the hosts.

The drafter

On October 27 the STQ drafter this file carries and Unsloth's Q8_0 drafter it carried before ran beside the same backbone (the two artifacts' text objects equal byte for byte), with the build that added the STQ drafter, on two rented hosts: an RTX 4090 (450 W, PCIe 3.0 x16) with a 24-core EPYC 7352 and 251 GiB of RAM, and an RTX PRO 6000 (500 W, PCIe 5.0 x16) with a Core Ultra 9 285K and 249 GiB of RAM. One request, greedy, host experts with the CPU share, confidence floor 0.5: the two prompts above (the short answer, up to 128 tokens; 32 tokens after the 8,086-token prompt), a code request (384 tokens), a reasoning request with thinking on (384 tokens) and a summary of the 8,086-token prompt (up to 256 tokens). Decode is each request's second run, its prompt from the prefix cache; prefill is the 8,086-token prompt's first run. Tok/s:

card	drafter	experts on the card	short answer	after 8,086 tokens	code	reasoning	summary	prefill of 8,086 tokens
RTX 4090	none	1,137	28.1	28.7	22.8	30.3	20.5	620
	Unsloth's (7.4 GiB)	73	22.0	19.7	23.2	26.0	18.2	405
	STQ (3.58 GiB)	618	25.9	24.3	27.1	36.6	20.9	634
RTX PRO 6000	none	11,387	84.2	79.6	81.4	82.4	79.4	1,250
	Unsloth's	10,316	114.3	107.0	170.0	143.0	123.2	1,250
	STQ	10,868	119.2	102.2	168.8	136.5	115.9	1,260

Over the five requests' first runs the drafters accepted 724 of 1,035 (Unsloth's) and 728 of 987 (STQ) drafts on the RTX 4090, and 657 of 905 and 655 of 889 on the RTX PRO 6000; every arm gave plain decoding's tokens on its card. On the RTX 4090 the STQ drafter decodes 1.15 to 1.41 times as fast as Unsloth's, mostly from the larger expert cache it leaves, and keeps plain prefill: beside Unsloth's drafter the cache cannot lend a whole prompt's span, the prompt runs in pieces and each layer's experts cross the bus twice. On the RTX PRO 6000 it decodes 0.94 to 1.04 times as fast as Unsloth's, slower on four of the five requests (not attributed). One run per row. The comparison adds the floor 0, a 500 MiB cache and acceptance by position.

Credits and license

Quantized weights: ISTA-DASLab, DeepSeek-V4.1-Flash-1017 GSQ-RCO GGUFs, produced with GSQ and RCO from deepseek-ai/DeepSeek-V4.1-Flash-1017. DSpark drafter: beamster's STQ encoding of the release's drafter, mtp.0..2 of beamster/DeepSeek-V4.1-Flash-1017-Sushi-STQ-1.75bpw. All under the MIT License, reproduced in LICENSE. Attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.