Instructions to use stornic56/Spark-X2.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use stornic56/Spark-X2.5-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use stornic56/Spark-X2.5-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "stornic56/Spark-X2.5-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stornic56/Spark-X2.5-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Ollama
How to use stornic56/Spark-X2.5-4B-GGUF with Ollama:
```
ollama run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use stornic56/Spark-X2.5-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "stornic56/Spark-X2.5-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use stornic56/Spark-X2.5-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/stornic56/Spark-X2.5-4B-GGUF:Q4_K_M
```

Lemonade

How to use stornic56/Spark-X2.5-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Spark-X2.5-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use stornic56/Spark-X2.5-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use stornic56/Spark-X2.5-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "stornic56/Spark-X2.5-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Llamacpp imatrix Quantizations of Spark-X2.5-4B by XHToken

Using XHToken/llama.cpp fork commit b10514-4a3635c32 (build tag b10514-4a3635c32) for quantization.

Original model: https://huggingface.co/XHToken/Spark-X2.5-4B

Model details:

Parameter count: 4.11B
Input support: text
imatrix: yes - details

How to run

⚠️ Important: the spark2_5 architecture is not supported by mainline llama.cpp. All files in this repo require the XHToken/llama.cpp fork at the commit linked above or newer - see How to run.

Prompt format

The chat template is embedded in every GGUF; run with --jinja and it is applied automatically. The rendered format is:

<｜start▁of▁sentence｜><|System|>
{system_prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|User|>
{prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|Bot|>
<think>

The model is a thinking model by default: reasoning is emitted inside <think>...</think> before the final answer. The embedded template also supports tool calling, matching the upstream chat_template.jinja.

Don't know which to choose? Grab Spark-X2.5-4B-Q4_K_M.gguf (2.60GB) - usually a good mix of size and performance. Download instructions available here.

Available files:

Filename	Quant type	File Size	Description
Spark-X2.5-4B-bf16.gguf	bf16	8.82GB	Full BF16 weights, converted directly from upstream safetensors.
Spark-X2.5-4B-Q8_0.gguf	Q8_0	4.37GB	Extremely high quality, generally unneeded but max available quant.
Spark-X2.5-4B-Q6_K.gguf	Q6_K	3.37GB	Very high quality, near perfect.
Spark-X2.5-4B-Q5_K_M.gguf	Q5_K_M	2.97GB	High quality.
Spark-X2.5-4B-Q4_K_M.gguf	Q4_K_M	2.60GB	Good quality, default size for most use cases.
Spark-X2.5-4B-IQ4_NL.gguf	IQ4_NL	2.47GB	Similar quality to Q4_K_M in a smaller file.
Spark-X2.5-4B-Q3_K_M.gguf	Q3_K_M	2.16GB	Lower quality but usable, good for low RAM availability. Fastest generation quant on Intel Arc (Vulkan) in our benchmarks.
Spark-X2.5-4B-IQ3_M.gguf	IQ3_M	2.04GB	Medium-low quality, imatrix-guided, comparable to Q3_K_M.
Spark-X2.5-4B-IQ2_M.gguf	IQ2_M	1.63GB	Relatively low quality, imatrix keeps it surprisingly usable; completed our extended coherence test with correct code.

Note: a Q2_K quant was also generated and tested, but excluded from this release after it produced repetition loops on the extended coherence test, while IQ2_M (a smaller file) completed the same test correctly. Evidence: reproducibility/q2_k_repetition_loops.log. Quants below IQ2_M (IQ2_XS, IQ1_M, Q1_0 class) were not produced; at this parameter count the quality loss is severe.

Downloading using the Hugging Face CLI

Click to view download instructions

First, make sure you have the Hugging Face CLI installed:

pip install -U "huggingface_hub[cli]"

Download a specific file:

hf download stornic56/Spark-X2.5-4B-GGUF --include "Spark-X2.5-4B-Q4_K_M.gguf" --local-dir ./

How to run

These quants require the XHToken llama.cpp fork. Mainline llama.cpp does not implement spark2_5 and will refuse to load these files.

git clone https://github.com/XHToken/llama.cpp.git && cd llama.cpp
# Intel/AMD GPU via Vulkan: add -DGGML_VULKAN=ON
# NVIDIA GPU:               add -DGGML_CUDA=ON
# CPU only:                 plain build works
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build --config Release -j 8

Interactive chat (template is embedded, --jinja applies it):

./build/bin/llama-cli -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -cnv

OpenAI-compatible server:

./build/bin/llama-server -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 16384 --jinja

Recommended sampling from the upstream model card: temp 1.0, top_p 0.95, top_k -1. Use --temp 0 for deterministic output.

Other runtimes: Ollama and LM Studio can run these files, but only with builds backed by the same fork - see the "Ollama" and "LM Studio" sections of the official model card for the build steps.

Flash Attention

The fork enables FlashAttention automatically for the sliding-window layers of spark2_5. No extra flag is needed.

imatrix

All quants below bf16 were made using the imatrix option, computed with llama-imatrix from this fork. The calibration corpus is the wikitext-2-raw-v1 train split (Salesforce/wikitext), written to disk verbatim and in order by reproducibility/calibration.py. The corpus file is included in this repo: calibration_data.txt.

The imatrix is available here: Spark-X2.5-4B-imatrix.gguf.

Calibration details

{
  "dataset": "Salesforce/wikitext",
  "config": "wikitext-2-raw-v1",
  "split": "train",
  "rendering": "raw text, no chat template, no special tokens",
  "chunk_size": 512,
  "chunks": 128,
  "tokens_consumed": 65536,
  "consumption_order": "sequential from file start (deterministic)",
  "threads": 8,
  "final_perplexity": "49.3694 +/- 1.09206",
  "imatrix_format": "GGUF (fork default)",
  "notes": "corpus is plain prose; a future revision may render tool-calling and reasoning conversations through the chat template, as done by other maintainers"
}

Reproducibility check: two independent imatrix runs over the same file with the same parameters produced identical final perplexity (49.3694 ± 1.09206). Run log: reproducibility/imatrix-run.log.

Which file should I choose?

Click here for details

The first thing to figure out is how big a model you can run. If you want the model running as fast as possible, fit the whole thing in VRAM: aim for a file 1-2GB smaller than your GPU's total VRAM, leaving room for the KV cache (this model's hybrid attention keeps the cache small, which helps). If you want maximum quality and can tolerate partial CPU offload, add your system RAM and VRAM together and pick a file 1-2GB smaller than that total.

Next, K-quant or I-quant? On NVIDIA (CUDA) and Apple silicon, I-quants (IQX_X) generally offer better quality per bit below Q4. On Intel Arc via Vulkan (Mesa), our measurements show the opposite for generation speed: K-quants are faster at equal bit-width, while I-quants win on file size - see the benchmarks below. Either way, both families were validated for coherence in this release.

Benchmarks

Measured with llama-bench from the same fork commit, Intel Arc B580 (Battlemage, Mesa Vulkan driver), -ngl 99 -t 4, 3 runs. Raw log: reproducibility/benchmarks_gpu.txt.

Quant	pp512 t/s	tg128 t/s
Q8_0	2039	69.5
Q6_K	1844	70.5
Q5_K_M	1949	79.8
Q4_K_M	2004	93.3
IQ4_NL	2079	53.9
Q3_K_M	1894	95.3
IQ3_M	1969	76.4
IQ2_M	2048	46.4

Prompt processing is roughly flat across quants (~~2k t/s, compute-bound on the GPU's matrix cores). Long-context reading, Q4_K_M with the whole model in VRAM: pp32768 = 309 t/s (~~6GB VRAM total including KV cache). The upstream 1M-token context was not validated in this release.

CPU reference readings (i3-12100F, 8 threads, single runs, not formal benchmarks): BF16 ≈ 4.8 t/s, Q4_K_M ≈ 13.4 t/s, Q3_K_M ≈ 15.6 t/s, IQ2_M ≈ 10.9 t/s generation.

Validation

test-llama-archs -a spark2_5 from the fork: OK on CPU (NMSE 0.00e+00), Vulkan (8.58e-08) and meta buffers. Roundtrip: SKIP is expected for this architecture.
Greedy decoding (--temp 0) produced token-identical output on CPU and Vulkan GPU, verified on Q4_K_M.
Every published quant completed a 700-token coding coherence test at --temp 0; Q2_K failed with repetition loops and was excluded (log linked in the files table).

Reproducibility

Everything needed to rebuild these files bit-for-bit is in the repo:

File	Content
SHA256SUMS.txt	Checksums of all GGUFs, imatrix and calibration corpus
Spark-X2.5-4B-imatrix.gguf	The importance matrix itself
calibration_data.txt	Calibration corpus, verbatim
reproducibility/calibration.py	Exact corpus generation script
reproducibility/fork_commit.txt	Fork commit used for conversion, imatrix and quantization
reproducibility/base_model_revision.txt	Upstream safetensors revision
reproducibility/imatrix-run.log	Imatrix run output (tail of final clean run; identical PPL reproduced across runs)
reproducibility/benchmarks_gpu.txt	Raw llama-bench output
reproducibility/q2_k_repetition_loops.log	Evidence for the Q2_K exclusion

Exact commands:

python convert_hf_to_gguf.py Spark-X2.5-4B --outfile Spark-X2.5-4B-bf16.gguf --outtype bf16

./build/bin/llama-imatrix \
  -m Spark-X2.5-4B-bf16.gguf \
  -f calibration_data.txt \
  -o Spark-X2.5-4B-imatrix.gguf \
  -c 512 --chunks 128 -t 8

./build/bin/llama-quantize --imatrix Spark-X2.5-4B-imatrix.gguf \
  Spark-X2.5-4B-bf16.gguf Spark-X2.5-4B-Q4_K_M.gguf Q4_K_M

ARM/AVX information

llama.cpp automatically repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines, covering Q4_0, IQ4_NL and most K-quants. No special quant choice is needed for CPU inference.

Credits

Thanks to the XHToken/SparkLLM team for releasing the model and maintaining the llama.cpp fork, and to bartowski for the quantization card format this repo follows.

License & attribution

The upstream model is released under Apache License 2.0; these derivative quantizations inherit it. Preserve upstream attribution when redistributing. This is an unofficial community release, not endorsed by XHToken/SparkLLM.

Downloads last month: 182

GGUF

Model size

4B params

Architecture

spark2_5

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

View +1 variant

Model tree for stornic56/Spark-X2.5-4B-GGUF

Base model

XHToken/Spark-X2.5-4B-Base

Finetuned

XHToken/Spark-X2.5-4B

Quantized

(18)

this model