--- quantized_by: stornic56 pipeline_tag: text-generation language: - en - zh license: apache-2.0 base_model: XHToken/Spark-X2.5-4B base_model_relation: quantized tags: - spark2_5 - spark-x2.5 - reasoning - thinking - tool-calling - imatrix - intel-arc - vulkan --- ## Llamacpp imatrix Quantizations of Spark-X2.5-4B by XHToken Using XHToken/llama.cpp fork commit b10514-4a3635c32 (build tag `b10514-4a3635c32`) for quantization. Original model: https://huggingface.co/XHToken/Spark-X2.5-4B **Model details:** - Parameter count: 4.11B - Input support: text - imatrix: yes - [details](#imatrix) [How to run](#how-to-run) **⚠️ Important:** the `spark2_5` architecture is not supported by mainline llama.cpp. All files in this repo require the [XHToken/llama.cpp fork](https://github.com/XHToken/llama.cpp) at the commit linked above or newer - see [How to run](#how-to-run). ## Prompt format The chat template is embedded in every GGUF; run with `--jinja` and it is applied automatically. The rendered format is: ``` <｜start▁of▁sentence｜><|System|> {system_prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|User|> {prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|Bot|> ``` The model is a thinking model by default: reasoning is emitted inside `...` before the final answer. The embedded template also supports tool calling, matching the upstream `chat_template.jinja`. **Don't know which to choose?** Grab [Spark-X2.5-4B-Q4_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q4_K_M.gguf) (2.60GB) - usually a good mix of size and performance. Download instructions available [here](#downloading-using-the-hugging-face-cli). ## Available files: | Filename | Quant type | File Size | Description | | -------- | ---------- | --------- | ----------- | | [Spark-X2.5-4B-bf16.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-bf16.gguf) | bf16 | 8.82GB | Full BF16 weights, converted directly from upstream safetensors. | | [Spark-X2.5-4B-Q8_0.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q8_0.gguf) | Q8_0 | 4.37GB | Extremely high quality, generally unneeded but max available quant. | | [Spark-X2.5-4B-Q6_K.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q6_K.gguf) | Q6_K | 3.37GB | Very high quality, near perfect. | | [Spark-X2.5-4B-Q5_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q5_K_M.gguf) | Q5_K_M | 2.97GB | High quality. | | [Spark-X2.5-4B-Q4_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q4_K_M.gguf) | Q4_K_M | 2.60GB | Good quality, default size for most use cases. | | [Spark-X2.5-4B-IQ4_NL.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-IQ4_NL.gguf) | IQ4_NL | 2.47GB | Similar quality to Q4_K_M in a smaller file. | | [Spark-X2.5-4B-Q3_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q3_K_M.gguf) | Q3_K_M | 2.16GB | Lower quality but usable, good for low RAM availability. Fastest generation quant on Intel Arc (Vulkan) in our benchmarks. | | [Spark-X2.5-4B-IQ3_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-IQ3_M.gguf) | IQ3_M | 2.04GB | Medium-low quality, imatrix-guided, comparable to Q3_K_M. | | [Spark-X2.5-4B-IQ2_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-IQ2_M.gguf) | IQ2_M | 1.63GB | Relatively low quality, imatrix keeps it surprisingly usable; completed our extended coherence test with correct code. | Note: a Q2_K quant was also generated and tested, but **excluded from this release** after it produced repetition loops on the extended coherence test, while IQ2_M (a smaller file) completed the same test correctly. Evidence: [reproducibility/q2_k_repetition_loops.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/q2_k_repetition_loops.log). Quants below IQ2_M (IQ2_XS, IQ1_M, Q1_0 class) were not produced; at this parameter count the quality loss is severe. ## Downloading using the Hugging Face CLI
Click to view download instructions First, make sure you have the Hugging Face CLI installed: ``` pip install -U "huggingface_hub[cli]" ``` Download a specific file: ``` hf download stornic56/Spark-X2.5-4B-GGUF --include "Spark-X2.5-4B-Q4_K_M.gguf" --local-dir ./ ```
## How to run These quants require the XHToken llama.cpp fork. Mainline llama.cpp does not implement `spark2_5` and will refuse to load these files. ``` git clone https://github.com/XHToken/llama.cpp.git && cd llama.cpp # Intel/AMD GPU via Vulkan: add -DGGML_VULKAN=ON # NVIDIA GPU: add -DGGML_CUDA=ON # CPU only: plain build works cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON cmake --build build --config Release -j 8 ``` Interactive chat (template is embedded, `--jinja` applies it): ``` ./build/bin/llama-cli -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -cnv ``` OpenAI-compatible server: ``` ./build/bin/llama-server -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 16384 --jinja ``` Recommended sampling from the upstream model card: `temp 1.0, top_p 0.95, top_k -1`. Use `--temp 0` for deterministic output. **Other runtimes:** Ollama and LM Studio can run these files, but only with builds backed by the same fork - see the "Ollama" and "LM Studio" sections of the [official model card](https://huggingface.co/XHToken/Spark-X2.5-4B) for the build steps. ## Flash Attention The fork enables FlashAttention automatically for the sliding-window layers of `spark2_5`. No extra flag is needed. ## imatrix All quants below bf16 were made using the imatrix option, computed with `llama-imatrix` from this fork. The calibration corpus is the `wikitext-2-raw-v1` train split ([Salesforce/wikitext](https://huggingface.co/datasets/Salesforce/wikitext)), written to disk verbatim and in order by [reproducibility/calibration.py](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/calibration.py). The corpus file is included in this repo: [calibration_data.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/calibration_data.txt). The imatrix is available here: [Spark-X2.5-4B-imatrix.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-imatrix.gguf).
Calibration details ``` { "dataset": "Salesforce/wikitext", "config": "wikitext-2-raw-v1", "split": "train", "rendering": "raw text, no chat template, no special tokens", "chunk_size": 512, "chunks": 128, "tokens_consumed": 65536, "consumption_order": "sequential from file start (deterministic)", "threads": 8, "final_perplexity": "49.3694 +/- 1.09206", "imatrix_format": "GGUF (fork default)", "notes": "corpus is plain prose; a future revision may render tool-calling and reasoning conversations through the chat template, as done by other maintainers" } ``` Reproducibility check: two independent imatrix runs over the same file with the same parameters produced identical final perplexity (49.3694 ± 1.09206). Run log: [reproducibility/imatrix-run.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/imatrix-run.log).
## Which file should I choose?
Click here for details The first thing to figure out is how big a model you can run. If you want the model running as fast as possible, fit the whole thing in VRAM: aim for a file 1-2GB smaller than your GPU's total VRAM, leaving room for the KV cache (this model's hybrid attention keeps the cache small, which helps). If you want maximum quality and can tolerate partial CPU offload, add your system RAM and VRAM together and pick a file 1-2GB smaller than that total. Next, K-quant or I-quant? On NVIDIA (CUDA) and Apple silicon, I-quants (IQX_X) generally offer better quality per bit below Q4. On **Intel Arc via Vulkan (Mesa)**, our measurements show the opposite for generation speed: K-quants are faster at equal bit-width, while I-quants win on file size - see the benchmarks below. Either way, both families were validated for coherence in this release.
## Benchmarks Measured with `llama-bench` from the same fork commit, Intel Arc B580 (Battlemage, Mesa Vulkan driver), `-ngl 99 -t 4`, 3 runs. Raw log: [reproducibility/benchmarks_gpu.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/benchmarks_gpu.txt). | Quant | pp512 t/s | tg128 t/s | | ----- | -------: | -------: | | Q8_0 | 2039 | 69.5 | | Q6_K | 1844 | 70.5 | | Q5_K_M | 1949 | 79.8 | | Q4_K_M | 2004 | 93.3 | | IQ4_NL | 2079 | 53.9 | | Q3_K_M | 1894 | 95.3 | | IQ3_M | 1969 | 76.4 | | IQ2_M | 2048 | 46.4 | Prompt processing is roughly flat across quants (~2k t/s, compute-bound on the GPU's matrix cores). Long-context reading, Q4_K_M with the whole model in VRAM: `pp32768 = 309 t/s` (~6GB VRAM total including KV cache). The upstream 1M-token context was not validated in this release. CPU reference readings (i3-12100F, 8 threads, single runs, not formal benchmarks): BF16 ≈ 4.8 t/s, Q4_K_M ≈ 13.4 t/s, Q3_K_M ≈ 15.6 t/s, IQ2_M ≈ 10.9 t/s generation. ## Validation - `test-llama-archs -a spark2_5` from the fork: OK on CPU (NMSE 0.00e+00), Vulkan (8.58e-08) and meta buffers. `Roundtrip: SKIP` is expected for this architecture. - Greedy decoding (`--temp 0`) produced token-identical output on CPU and Vulkan GPU, verified on Q4_K_M. - Every published quant completed a 700-token coding coherence test at `--temp 0`; Q2_K failed with repetition loops and was excluded (log linked in the files table). ## Reproducibility Everything needed to rebuild these files bit-for-bit is in the repo: | File | Content | | ---- | ------- | | [SHA256SUMS.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/SHA256SUMS.txt) | Checksums of all GGUFs, imatrix and calibration corpus | | [Spark-X2.5-4B-imatrix.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-imatrix.gguf) | The importance matrix itself | | [calibration_data.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/calibration_data.txt) | Calibration corpus, verbatim | | [reproducibility/calibration.py](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/calibration.py) | Exact corpus generation script | | [reproducibility/fork_commit.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/fork_commit.txt) | Fork commit used for conversion, imatrix and quantization | | [reproducibility/base_model_revision.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/base_model_revision.txt) | Upstream safetensors revision | | [reproducibility/imatrix-run.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/imatrix-run.log) | Imatrix run output (tail of final clean run; identical PPL reproduced across runs) | | [reproducibility/benchmarks_gpu.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/benchmarks_gpu.txt) | Raw llama-bench output | | [reproducibility/q2_k_repetition_loops.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/q2_k_repetition_loops.log) | Evidence for the Q2_K exclusion | Exact commands: ``` python convert_hf_to_gguf.py Spark-X2.5-4B --outfile Spark-X2.5-4B-bf16.gguf --outtype bf16 ./build/bin/llama-imatrix \ -m Spark-X2.5-4B-bf16.gguf \ -f calibration_data.txt \ -o Spark-X2.5-4B-imatrix.gguf \ -c 512 --chunks 128 -t 8 ./build/bin/llama-quantize --imatrix Spark-X2.5-4B-imatrix.gguf \ Spark-X2.5-4B-bf16.gguf Spark-X2.5-4B-Q4_K_M.gguf Q4_K_M ``` ## ARM/AVX information llama.cpp automatically repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines, covering Q4_0, IQ4_NL and most K-quants. No special quant choice is needed for CPU inference. ## Credits Thanks to the XHToken/SparkLLM team for releasing the model and maintaining the llama.cpp fork, and to bartowski for the quantization card format this repo follows. ## License & attribution The upstream model is released under Apache License 2.0; these derivative quantizations inherit it. Preserve upstream attribution when redistributing. This is an unofficial community release, not endorsed by XHToken/SparkLLM.