bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF

GGUF / llama.cpp quantizations of bottlecapai/ThinkingCap-Qwen3.8-Flash-Next — the ThinkingCap finetune of Qwen3.8-Flash-Next that keeps the base model's answer quality while reasoning in far fewer tokens.

➡️ Full model description, evaluation results (multi-seed, statistically tested), the measured thinking-token reduction, recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.8-Flash-Next.

About GGUF and quantization

GGUF is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, …). The quantized variants below store weights at reduced precision — e.g. about 5 bits per weight for Q4_K_M instead of the unquantized source's 16 — cutting download size and memory severalfold at a small quality cost.

The low-bit files are built with an importance matrix (activation statistics from a calibration corpus, which steer where the low-bit error lands) and per-tensor precision overrides (chosen tensors quantized at a precision other than the quant type's default). Both are taken from unsloth/Qwen3.8-Flash-Next-GGUF.

Files

File	Quant	Size
ThinkingCap-Qwen3.8-Flash-Next-BF16-*.gguf (8 parts)	BF16	359.2 GB
ThinkingCap-Qwen3.8-Flash-Next-IQ4_XS-*.gguf (8 parts)	IQ4_XS	95.5 GB
ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-*.gguf (8 parts)	Q4_K_M	113.2 GB
ThinkingCap-Qwen3.8-Flash-Next-Q6_K-*.gguf (8 parts)	Q6_K	171.9 GB
ThinkingCap-Qwen3.8-Flash-Next-Q8_0-*.gguf (8 parts)	Q8_0	191.0 GB
mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf	mmproj (vision)	907.5 MB

BF16 is the unquantized source; Q8_0 is near-lossless; Q6_K is a smaller near-lossless option; Q4_K_M is the recommended size/quality balance for most local setups; IQ4_XS is a smaller 4-bit option. For a smaller file, ThinkingCap-Qwen3.8-Flash-Next-GGUF-RCO-3.8bpw holds an 85.6 GB mixed-precision build whose weights take 56.5 GB of GPU memory, against 66.1 GB for IQ4_XS.

Expected performance

Each quant against the BF16 model served by vLLM, with thinking on at the chat template's default effort (xhigh) and the same sampling settings (temperature 1.0, top-p 0.95, top-k 20, up to 253,952 new tokens). AIME 2026 has 30 problems, run 8 times per quant and 32 times for BF16; RealWorldQA has 765 image questions, run twice per quant and 8 times for BF16.

Quant	AIME 2026	change	thinking tokens	RealWorldQA	change	thinking tokens
BF16 (vLLM)	97.19%			84.67%
IQ4_XS	97.50%	+0.31 pp [−1.25, +1.77]	+9.7% [+1.6, +23.0]	84.51%	−0.16 pp [−1.23, +0.87]	−4.5% [−14.1, +5.6]
Q4_K_M	97.08%	−0.10 pp [−1.98, +1.87]	+9.6% [+1.3, +22.2]	85.69%	+1.01 pp [−0.13, +2.16]	+5.6% [−4.6, +16.3]
Q6_K	96.25%	−0.94 pp [−2.92, +0.83]	+10.2% [+0.5, +17.8]	84.84%	+0.16 pp [−0.95, +1.27]	+17.0% [+2.0, +33.8]
Q8_0	96.25%	−0.94 pp [−3.85, +1.56]	+6.6% [−4.3, +14.3]	85.23%	+0.56 pp [−0.54, +1.62]	+2.9% [−9.2, +15.7]

A change is the quant minus BF16, with a 95% interval from resampling the questions (each question's runs averaged first); an interval that contains 0 is no measurable difference. No quant's accuracy differs measurably from BF16 on either benchmark. On AIME the three quants below Q8_0 used about 10% more thinking tokens than BF16, and Q8_0's +6.6% cannot be told apart from either 0 or those 10%; whether the extra thinking comes from quantization or from llama.cpp itself was not tested. On RealWorldQA the median answer thinks for 113–126 tokens on every quant and 115 on BF16, so the means there move with a few long answers, Q6_K's +17% among them.

Served by llama-server at llama.cpp 4e910a8 on one H200 (two for Q8_0 on AIME), with the vision projector loaded, a q8_0 KV cache and no speculative decoding; RealWorldQA one request at a time (see Vision below). llama-server does not count thinking tokens, so they were counted by re-tokenizing each trace with its /tokenize.

Usage (llama.cpp)

# pull a specific quant straight from the Hub and chat (-hf also loads the vision projector; see Vision below)
llama-cli -hf bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF:Q4_K_M -p "Hi"

# or download one quant and run it
huggingface-cli download bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF --include "ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-*.gguf" --local-dir .
llama-cli -m ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-00001-of-00008.gguf -p "Hi"

Use the sampling settings from the main model card (the base model's recommended thinking-mode settings). Greedy decoding can loop; keep temperature at the recommended value.

Checked with llama.cpp at commit 5f5acfe (2026-10-22): Q4_K_M with the vision projector answers a math request and an image request correctly, on one H200 (CUDA build) and on CPU.

Speculative decoding (MTP)

These GGUFs carry the model's MTP (multi-token-prediction) head, so llama.cpp can run self-speculative decoding — no separate draft model needed. Add --spec-type draft-mtp when serving:

# -hf also loads the vision projector
llama-server -hf bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF:Q4_K_M -np 4 --no-kv-unified --spec-type draft-mtp --spec-draft-n-max 4

Requires a llama.cpp build with MTP support for this architecture; runtimes that predate it may refuse to load the file with a missing tensor error — update the runtime. In that check on the H200, with 3 draft tokens (--spec-draft-n-max 3), MTP accepted on average 3.75 tokens per decoding step on the math request and 2.72 on the image request.

-np 4 --no-kv-unified gives each of the 4 parallel requests its own KV cache. llama-server's slots share one cache by default, and with this model that costs accuracy when several image requests are served at once: on 100 RealWorldQA questions the default 4 shared slots scored 71.0% against 86.0% for a single slot (see Vision below). For text only, add --no-mmproj to skip the projector and drop -np 4 --no-kv-unified.

Vision (image input)

ThinkingCap is a vision-language model. Image input needs the multimodal projector mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf (in this repo) loaded alongside a text GGUF — the single f16 mmproj pairs with any of the quants above.

• LM Studio / Jan / Ollama, …: download the mmproj-*.gguf from this repo; LM Studio auto-detects it and enables the image (🖼️) button.
• llama.cpp CLI:

huggingface-cli download bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF \
  --include "ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-*.gguf" mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf --local-dir .
llama-mtmd-cli -m ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-00001-of-00008.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf --image photo.jpg -p "Describe this image."

• llama-server (OpenAI-compatible endpoint; -np 4 --no-kv-unified gives each of 4 parallel requests its own KV cache, see below):

llama-server -m ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-00001-of-00008.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf -np 4 --no-kv-unified

By default llama-server runs 4 slots that share one KV cache, and with this model that costs accuracy when several image requests are served at once. On 100 RealWorldQA questions with IQ4_XS on one H200 (llama.cpp 4e910a8, one run per setting), the default 4 slots on one shared cache scored 71.0% and 32 slots on one shared cache 59.0%; 32 slots with a separate KV cache each (-np 32 --no-kv-unified) scored 83.0% and a single slot 86.0%, against 87.1% for the BF16 model served by vLLM. --no-kv-unified takes effect only together with -np: without it, llama-server picks the slot count itself and switches the shared cache back on. With separate caches, a context set with -c is split evenly across the slots; left unset, each slot got the model's full 262,144 tokens on the H200 (llama.cpp shrinks the context when it does not fit in free GPU memory). llama-cli and llama-mtmd-cli serve one request at a time.

License

Released under the base model's own license, the Qwen Community License 1.0 (full text in LICENSE). Its conditions carry over to this model, including the separate license from Qwen that Model-as-a-Service and AI work-assistant businesses need for commercial use.