Sharp-Spark-X2.6-4B-GGUF

Dynamic imatrix GGUF quants of XHToken/Spark-X2.6-4B, a 4B dense long-context model, carrying our cyber-and-coding-weighted importance matrix, a heuristic per-tensor bit allocation, and the Sharp-Spark chat template.

This is a small, fast, long-context coder that fits 6 GB-VRAM GPUs and still holds usable speed out to six-figure context. It is the successor to Sharp-Spark-X2.5-4B: same recipe, new weights.

---

What changed from 2.5

• MMLU-Pro: same accuracy, 39% fewer tokens. 71.7% against 2.5's 69.0% (a tie), with the average question costing 5,498 tokens instead of 9,007.
• SWE-bench-Live: 2.4× the solves of 2.5. SharpSpark 2.6 solves 8.7 of 17 problems per seed against 3.7 for SharpSpark 2.5 (+5.0 per seed, paired 95% CI [+2.0, +8.0] over the 17 problems).
• Against XHToken's own Q8_0 (4.4 GB) this 3.6 GB build ties on both benchmarks (SWE 8.7 vs 9.0, MMLU 71.7% vs 71.3%) and spends 53% fewer tokens per MMLU question at the median (26% at the mean, p < 0.001). SWE attempts take 15 min at the median for SharpSpark 2.6 and 15 for the stock Q8_0 (means 29 and 25); counting all work per solve, the stock Q8_0 is quicker (46 min against 57).

About the SWE numbers. Every SWE run on this card was made with the agent's container on a sealed network: it can reach the model server and nothing else. Without that, agents sometimes look up the upstream fix for the issue they are working on. The board on the SharpSpark 2.5 card predates the seal; the 2.5 bar here is a sealed rerun of the same 2.5 file.

---

Who is this for?

This is the coder model for people with a small-VRAM GPU and ≤ 16GB RAM.

If you have more regular RAM than 16GB, try a (Cyber)TielCoder with partial GPU offloading instead. Its MoE architecture makes it fast even when it's split onto system RAM, and it is a stronger coder.

---

Which quant should I download?

Grab the Q6_K_XL when you can: it is the tier both boards above measured. The Q4 and Q5 are fallbacks, baked with the same imatrix, recipe family and template, but not benchmarked.

tier	size	pick it for
Q4_K_XL	2.68 GB	4 GB VRAM GPU, or more context headroom
Q5_K_XL	3.25 GB	5 GB VRAM GPU, a step up from Q4
Q6_K_XL	3.61 GB	6+ GB VRAM GPU, validated SWE and MMLU-Pro performance

If you find yourself needing a smaller quant than Q4, consider a natively smaller model instead. Small models like this are sensitive to quantization damage below Q6.

---

Long context

Spark is a hybrid-attention model: of its 36 layers, only 9 are full-attention (every 4th layer); the other 27 use a 512-token sliding window. So the KV cache barely grows — only the 9 full layers scale with context, which is what makes a 4B usable at six-figure context. 2.6 keeps 2.5's architecture exactly (same layer layout, heads and window), so everything below carries over.

Short vs 131k context (measured on SharpSpark 2.5 Q6_K_XL, the same architecture at the same file size):

context	prefill tok/s	decode tok/s
short (512)	1684	111
131,072	557	42

KV cache cost. Only 9 of the 36 layers hold a full-length cache — the other 27 are windowed to 512 tokens and cost a flat ~57 MB, so KV grows at ≈ 4 KB / token / full layer, i.e. 36 KB/token at f16 and 19 KB/token at q8_0:

context	KV f16	KV q8_0
4K	0.21 GB	0.11 GB
8K	0.36 GB	0.19 GB
16K	0.66 GB	0.35 GB
32K	1.27 GB	0.67 GB
64K	2.47 GB	1.31 GB
128K	4.83 GB	2.60 GB
256K	9.49 GB	5.16 GB

So on a 6 GB card, Q6_K_XL (3.6 GB) leaves room for ~48k at f16 — or ~96k at q8_0. For more still, drop the model a tier rather than the cache: Q4_K_XL clears 128k on the same card.

---

The imatrix

Quantizing means deciding which weights can afford to lose precision. An importance matrix answers that empirically instead of by guesswork: run the full-precision model over a calibration corpus, record how much each weight actually moves the output, and spend bits where the measurement says they matter.

This one is weighted toward security and systems code — a public, redistributable corpus of the kind of text this model is built to work on — and was measured afresh on the 2.6 BF16 weights (the same corpus and procedure as 2.5, not 2.5's matrix reused). At 8.2 GB the full-precision model fits in memory, so no step in the measurement chain is itself an approximation of the model.

• 3641 chunks × 4096 tokens = 14.9 M tokens. The run consumed the corpus to exhaustion rather than stopping at a preset chunk budget, so the matrix reflects all of the calibration text, not a prefix of it.
• Every chunk is 4096 tokens, and that length is the whole point. Spark's attention is mostly windowed: 27 of its 36 layers only ever see 512 tokens at a time, and just 9 attend across the full sequence. Calibrate on chunks at or below that 512-token window and those 9 layers are never put into the long-range regime they exist for — their importance scores would describe work they aren't doing, and the quantizer would then under-protect exactly the layers that carry long-context behaviour. At 4096 they are measured doing their real job.

Both figures are recorded in the file itself as imatrix.chunk_count and imatrix.chunk_size, so you can check them rather than take our word for it.

The imatrix .gguf ships in this repo — point llama-quantize --imatrix ... at it and bake your own tier if none of ours is what you want.

---

The Sharp-Spark template

The embedded chat template is Spark 2.6's own, with its remaining defects fixed and a terseness instruction spliced in. It keeps Spark's native control tokens, DeepSeek-style tool-call format and 2.6's thinking efforts untouched: this is Spark's own template with targeted repairs on top, not a template borrowed from another model.

Fixes over the stock 2.6 template:

1. A served system prompt replaces the canned "you are a helpful assistant" default instead of trailing it.
2. The tools block is ordered after the system prompt (stock 2.6 still puts the tool list before a caller's prompt; 2.6 itself fixed the missing separator 2.5 had).
3. Reasoning history is read from reasoning / thinking as well as reasoning_content (vLLM / Responses-API compatibility).
4. Tool results render through the content-block path, so a list-of-blocks tool result no longer leaks a Python repr into the prompt.

Thinking efforts are Spark 2.6's own, rendered exactly as stock does. Thinking is on by default at effort high; reasoning_effort: "low" selects low, and "none" (or enable_thinking: false) turns thinking off. Stock attaches the effort lines to its default persona, so they vanish once fix 1 swaps in your system prompt; here they follow whichever system prompt is in play.

Clients that send reasoning_effort: "medium" get an error, from the stock template and this one alike: 2.6 accepts only low and high. Some coding agents send medium by default — switch that off (or send high / low).

Addition: a short terseness instruction is force-appended to the system prompt (opt out with chat_template_kwargs={"terse": false}). It also stands its tool block down if the runtime injects its own tool protocol (suppress_tool_instructions).

Terseness was active for both boards at the top of this card, at the default high effort. The comparison with the stock Q8_0 moves the quant recipe and the template together, so it does not credit terseness with anything on its own. Flip the flag and A/B it if you need to know.

Tool calling works through llama-server --jinja, which parses Spark's tool-call syntax into standard OpenAI tool_calls (verified).

The raw template is in this repo as chat_template.jinja — apply it to your own quants of Spark-X2.6-4B with llama-server --chat-template-file chat_template.jinja, or embed it at quant time.

---

Requirements

Spark uses the spark2_5 architecture (2.6 included), which requires a recent llama.cpp with Spark support (the arch landed via PR #27868). An older build will fail to load these files. There is no MTP / draft head on this model, and no vision component.

Example:

llama-server -m Sharp-Spark-X2.6-4B-Q6_K_XL.gguf -ngl 99 -c 32768 --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

(temp 0.6 is our recommended operating point for coding, and what the SWE board ran at.)

---

Provenance & license

• Base model: XHToken/Spark-X2.6-4B, Apache 2.0.
• BF16 GGUF converted from the base weights with the spark2_5-capable llama.cpp; each tier quantized from that BF16 with our imatrix under a heuristic per-tensor bit allocation.
• These quants inherit the base model's Apache 2.0 license.