Instructions to use stornic56/Spark-X2.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use stornic56/Spark-X2.5-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use stornic56/Spark-X2.5-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "stornic56/Spark-X2.5-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stornic56/Spark-X2.5-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Ollama
How to use stornic56/Spark-X2.5-4B-GGUF with Ollama:
```
ollama run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use stornic56/Spark-X2.5-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "stornic56/Spark-X2.5-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use stornic56/Spark-X2.5-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
```

Lemonade

How to use stornic56/Spark-X2.5-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Spark-X2.5-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use stornic56/Spark-X2.5-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use stornic56/Spark-X2.5-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "stornic56/Spark-X2.5-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Spark-X2.5-4B-GGUF / README.md

stornic56

Upload README.md with huggingface_hub

7ce72e5 verified about 1 hour ago

preview code

Raw

History Blame Contribute Delete

12.9 kB

	---
	quantized_by: stornic56
	pipeline_tag: text-generation
	language:
	- en
	- zh
	license: apache-2.0
	base_model: XHToken/Spark-X2.5-4B
	base_model_relation: quantized
	tags:
	- spark2_5
	- spark-x2.5
	- reasoning
	- thinking
	- tool-calling
	- imatrix
	- intel-arc
	- vulkan
	---

	## Llamacpp imatrix Quantizations of Spark-X2.5-4B by XHToken

	Using <a href="https://github.com/XHToken/llama.cpp">XHToken/llama.cpp</a> fork commit
	<a href="https://github.com/XHToken/llama.cpp/commit/4a3635c32fc9f044c2bde9ebeabf50c7e1ec5991">b10514-4a3635c32</a> (build tag
	`b10514-4a3635c32`) for quantization.

	Original model: https://huggingface.co/XHToken/Spark-X2.5-4B

	Companion repo: [stornic56/Spark-X2.5-1.7B-GGUF](https://huggingface.co/stornic56/Spark-X2.5-1.7B-GGUF) — same pipeline, same calibration corpus, same fork commit. The family quality floor study below compares both.

	Model details:
	- Parameter count: 4.11B
	- Input support: text
	- imatrix: yes - [details](#imatrix)

	[How to run](#how-to-run)

	Update: as of llama.cpp [b10828](https://github.com/ggml-org/llama.cpp/releases/tag/b10828),the spark2_5 architecture is supported in mainline llama.cpp [PR #27868](https://github.com/ggml-org/llama.cpp/pull/27868). These files load on anystock build b10828 or newer. The XHToken fork is only needed for older builds - [How to run](#how-to-run).


	## Prompt format

	The chat template is embedded in every GGUF; run with `--jinja` and it is applied
	automatically. The rendered format is:

	```
	<｜start▁of▁sentence｜><\|System\|>
	{system_prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><\|User\|>
	{prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><\|Bot\|>
	<think>
	```

	The model is a thinking model by default: reasoning is emitted inside `<think>...</think>`
	before the final answer. The embedded template also supports tool calling, matching the
	upstream `chat_template.jinja`.

	Don't know which to choose? Grab
	[Spark-X2.5-4B-Q4_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q4_K_M.gguf)
	(2.60GB) - usually a good mix of size and performance. Download instructions available
	[here](#downloading-using-the-hugging-face-cli).

	## Available files:

	\| Filename \| Quant type \| File Size \| Description \|
	\| -------- \| ---------- \| --------- \| ----------- \|
	\| [Spark-X2.5-4B-bf16.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-bf16.gguf) \| bf16 \| 8.82GB \| Full BF16 weights, converted directly from upstream safetensors. \|
	\| [Spark-X2.5-4B-Q8_0.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q8_0.gguf) \| Q8_0 \| 4.37GB \| Extremely high quality, generally unneeded but max available quant. \|
	\| [Spark-X2.5-4B-Q6_K.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q6_K.gguf) \| Q6_K \| 3.37GB \| Very high quality, near perfect. \|
	\| [Spark-X2.5-4B-Q5_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q5_K_M.gguf) \| Q5_K_M \| 2.97GB \| High quality. \|
	\| [Spark-X2.5-4B-Q4_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q4_K_M.gguf) \| Q4_K_M \| 2.60GB \| Good quality, default size for most use cases. \|
	\| [Spark-X2.5-4B-IQ4_NL.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-IQ4_NL.gguf) \| IQ4_NL \| 2.47GB \| Similar quality to Q4_K_M in a smaller file. \|
	\| [Spark-X2.5-4B-Q3_K_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-Q3_K_M.gguf) \| Q3_K_M \| 2.16GB \| Lower quality but usable, good for low RAM availability. Fastest generation quant on Intel Arc (Vulkan) in our benchmarks. \|
	\| [Spark-X2.5-4B-IQ3_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-IQ3_M.gguf) \| IQ3_M \| 2.04GB \| Medium-low quality, imatrix-guided, comparable to Q3_K_M. \|
	\| [Spark-X2.5-4B-IQ2_M.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-IQ2_M.gguf) \| IQ2_M \| 1.63GB \| Relatively low quality, imatrix keeps it surprisingly usable; completed our extended coherence test with correct code. \|

	Note: a Q2_K quant was also generated and tested, but excluded from this release after
	it produced repetition loops on the extended coherence test, while IQ2_M (a smaller file)
	completed the same test correctly. Evidence: [reproducibility/q2_k_repetition_loops.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/q2_k_repetition_loops.log).
	Quants below IQ2_M (IQ2_XS, IQ1_M, Q1_0 class) were not produced; at this parameter count
	the quality loss is severe.

	## Downloading using the Hugging Face CLI

	<details>
	<summary>Click to view download instructions</summary>

	First, make sure you have the Hugging Face CLI installed:

	```
	pip install -U "huggingface_hub[cli]"
	```

	Download a specific file:

	```
	hf download stornic56/Spark-X2.5-4B-GGUF --include "Spark-X2.5-4B-Q4_K_M.gguf" --local-dir ./
	```

	</details>

	## How to run

	These quants run with mainline llama.cpp release [b10828](https://github.com/ggml-org/llama.cpp/releases/tag/b10828) or newer, which added native spark2_5 support - installable in oneline via llama.app:

	```
	curl -LsSf https://llama.app/install.sh \| sh
	llama-server -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
	```

	Interactive chat (template is embedded, `--jinja` applies it):

	```
	./build/bin/llama-cli -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -cnv
	```

	OpenAI-compatible server:

	```
	./build/bin/llama-server -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 16384 --jinja
	```

	Recommended sampling from the upstream model card: `temp 1.0, top_p 0.95, top_k -1`.
	Use `--temp 0` for deterministic output.

	Other runtimes: Ollama and LM Studio can run these files, but only with builds backed
	by the same fork - see the "Ollama" and "LM Studio" sections of the
	[official model card](https://huggingface.co/XHToken/Spark-X2.5-4B) for the build steps.

	## Flash Attention

	llama.cpp enables enables FlashAttention automatically for the sliding-window layers of `spark2_5`.
	No extra flag is needed.

	## imatrix

	All quants below bf16 were made using the imatrix option, computed with `llama-imatrix`
	from this fork. The calibration corpus is the `wikitext-2-raw-v1` train split
	([Salesforce/wikitext](https://huggingface.co/datasets/Salesforce/wikitext)), written to
	disk verbatim and in order by [reproducibility/calibration.py](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/calibration.py).
	The corpus file is included in this repo:
	[calibration_data.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/calibration_data.txt).

	The imatrix is available here:
	[Spark-X2.5-4B-imatrix.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-imatrix.gguf).

	<details>
	<summary>Calibration details</summary>

	```
	{
	"dataset": "Salesforce/wikitext",
	"config": "wikitext-2-raw-v1",
	"split": "train",
	"rendering": "raw text, no chat template, no special tokens",
	"chunk_size": 512,
	"chunks": 128,
	"tokens_consumed": 65536,
	"consumption_order": "sequential from file start (deterministic)",
	"threads": 8,
	"final_perplexity": "49.3694 +/- 1.09206",
	"imatrix_format": "GGUF (fork default)",
	"notes": "corpus is plain prose; a future revision may render tool-calling and reasoning conversations through the chat template, as done by other maintainers"
	}
	```

	Reproducibility check: two independent imatrix runs over the same file with the same
	parameters produced identical final perplexity (49.3694 ± 1.09206). Run log:
	[reproducibility/imatrix-run.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/imatrix-run.log).

	</details>

	## Which file should I choose?

	<details>
	<summary>Click here for details</summary>

	The first thing to figure out is how big a model you can run. If you want the model
	running as fast as possible, fit the whole thing in VRAM: aim for a file 1-2GB smaller
	than your GPU's total VRAM, leaving room for the KV cache (this model's hybrid attention
	keeps the cache small, which helps). If you want maximum quality and can tolerate partial
	CPU offload, add your system RAM and VRAM together and pick a file 1-2GB smaller than
	that total.

	Next, K-quant or I-quant? On NVIDIA (CUDA) and Apple silicon, I-quants (IQX_X) generally
	offer better quality per bit below Q4. On Intel Arc via Vulkan (Mesa), our measurements
	show the opposite for generation speed: K-quants are faster at equal bit-width, while
	I-quants win on file size - see the benchmarks below. Either way, both families were
	validated for coherence in this release.

	</details>

	## Benchmarks

	Measured with `llama-bench` from the same fork commit, Intel Arc B580 (Battlemage,
	Mesa Vulkan driver), `-ngl 99 -t 4`, 3 runs. Raw log:
	[reproducibility/benchmarks_gpu.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/benchmarks_gpu.txt).

	\| Quant \| pp512 t/s \| tg128 t/s \|
	\| ----- \| -------: \| -------: \|
	\| Q8_0 \| 2039 \| 69.5 \|
	\| Q6_K \| 1844 \| 70.5 \|
	\| Q5_K_M \| 1949 \| 79.8 \|
	\| Q4_K_M \| 2004 \| 93.3 \|
	\| IQ4_NL \| 2079 \| 53.9 \|
	\| Q3_K_M \| 1894 \| 95.3 \|
	\| IQ3_M \| 1969 \| 76.4 \|
	\| IQ2_M \| 2048 \| 46.4 \|

	Prompt processing is roughly flat across quants (~2k t/s, compute-bound on the GPU's
	matrix cores). Long-context reading, Q4_K_M with the whole model in VRAM:
	`pp32768 = 309 t/s` (~6GB VRAM total including KV cache). The upstream 1M-token context
	was not validated in this release.

	CPU reference readings (i3-12100F, 8 threads, single runs, not formal benchmarks):
	BF16 ≈ 4.8 t/s, Q4_K_M ≈ 13.4 t/s, Q3_K_M ≈ 15.6 t/s, IQ2_M ≈ 10.9 t/s generation.

	## Validation

	- `test-llama-archs -a spark2_5` from the fork: OK on CPU (NMSE 0.00e+00), Vulkan
	(8.58e-08) and meta buffers. `Roundtrip: SKIP` is expected for this architecture.
	- Greedy decoding (`--temp 0`) produced token-identical output on CPU and Vulkan GPU,
	verified on Q4_K_M.
	- Every published quant completed a 700-token coding coherence test at `--temp 0`;
	Q2_K failed with repetition loops and was excluded (log linked in the files table).

	## Reproducibility

	Everything needed to rebuild these files bit-for-bit is in the repo:

	\| File \| Content \|
	\| ---- \| ------- \|
	\| [SHA256SUMS.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/SHA256SUMS.txt) \| Checksums of all GGUFs, imatrix and calibration corpus \|
	\| [Spark-X2.5-4B-imatrix.gguf](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/Spark-X2.5-4B-imatrix.gguf) \| The importance matrix itself \|
	\| [calibration_data.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/calibration_data.txt) \| Calibration corpus, verbatim \|
	\| [reproducibility/calibration.py](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/calibration.py) \| Exact corpus generation script \|
	\| [reproducibility/fork_commit.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/fork_commit.txt) \| Fork commit used for conversion, imatrix and quantization \|
	\| [reproducibility/base_model_revision.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/base_model_revision.txt) \| Upstream safetensors revision \|
	\| [reproducibility/imatrix-run.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/imatrix-run.log) \| Imatrix run output (tail of final clean run; identical PPL reproduced across runs) \|
	\| [reproducibility/benchmarks_gpu.txt](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/benchmarks_gpu.txt) \| Raw llama-bench output \|
	\| [reproducibility/q2_k_repetition_loops.log](https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF/blob/main/reproducibility/q2_k_repetition_loops.log) \| Evidence for the Q2_K exclusion \|

	Exact commands:

	```
	python convert_hf_to_gguf.py Spark-X2.5-4B --outfile Spark-X2.5-4B-bf16.gguf --outtype bf16

	./build/bin/llama-imatrix \
	-m Spark-X2.5-4B-bf16.gguf \
	-f calibration_data.txt \
	-o Spark-X2.5-4B-imatrix.gguf \
	-c 512 --chunks 128 -t 8

	./build/bin/llama-quantize --imatrix Spark-X2.5-4B-imatrix.gguf \
	Spark-X2.5-4B-bf16.gguf Spark-X2.5-4B-Q4_K_M.gguf Q4_K_M
	```

	## ARM/AVX information

	llama.cpp automatically repacks weights into an interleaved layout at load time for
	faster inference on ARM and AVX machines, covering Q4_0, IQ4_NL and most K-quants.
	No special quant choice is needed for CPU inference.

	## Credits

	Thanks to the XHToken/SparkLLM team for releasing the model and maintaining the llama.cpp
	fork, and to bartowski for the quantization card format this repo follows.

	## License & attribution

	The upstream model is released under Apache License 2.0; these derivative quantizations
	inherit it. Preserve upstream attribution when redistributing. This is an unofficial
	community release, not endorsed by XHToken/SparkLLM.