basal-1.5-4.5B-GGUF

GGUF version of basal-1.5-4.5B (4.5B) for Ollama and llama.cpp: CPUs, Apple Silicon and consumer GPUs. See the main card for the description, benchmarks, limitations and license.

file	quantisation	size	agreement with bf16	accuracy change
basal-1.5-4.5B-Q8_0.gguf	Q8_0	5.06 GB	0.994	+0.002 (0.938 vs 0.936)
basal-1.5-4.5B-Q4_K_M.gguf	Q4_K_M (importance matrix; output head and embeddings Q8_0)	2.93 GB	0.980	±0.000 (0.936 vs 0.936)

Accuracy here is on a 500-decision sample, to check that each file matches its reference; to compare model sizes, see the benchmark table of the main card.

Latency: 427 ms with llama.cpp and 518 ms with Ollama (Q8_0); 446 / 542 ms (Q4_K_M) per decision (Apple M4 Pro, 24 GB, Metal; llama.cpp 0.5.0, Ollama 0.35.0).

Converted with llama.cpp convert_hf_to_gguf.py and llama-quantize, with two fixes that matter for decisions:

• Tokenizer fields. The stock conversion writes the same score (−1000) for every token of this tokenizer, so Ollama and other text clients would split prompts differently from the original tokenizer. This export sets the scores from the BPE merge ranks, sets add_space_prefix = false and matches every added token (<|im_start|>, <|im_end|>, …) whole, as the Hugging Face tokenizer does, so prompts are tokenized exactly as by the original tokenizer and Ollama and llama.cpp give the same answers.
• Q4_K_M is quantised with an importance matrix computed on decision prompts and keeps the output head and the token embeddings at Q8_0 (the decision is read from the output head's logits of the option letters).

Quick start (Ollama)

basal is a typed-decision model, not a chat model: the basal server runs in front of Ollama, renders the model's own prompt and answer prefix, sends it raw and reads the log-probabilities of the option letters of one generated token. Ollama 0.12.11 or newer.

hf download Remek/basal-1.5-4.5B-GGUF --local-dir basal-1.5-4.5B-GGUF
cd basal-1.5-4.5B-GGUF && ollama create basal-1.5-4.5b:q8_0 -f Modelfile.Q8_0 && cd ..
uv pip install "basal @ https://github.com/rkinas/basal/archive/refs/tags/v1.5.0.tar.gz"
basal-serve --model ./basal-1.5-4.5B-GGUF --mode ollama --ollama-model basal-1.5-4.5b:q8_0 --port 8000

--model gives the tokenizer, chat template and calibration file (this folder or Remek/basal-1.5-4.5B-GGUF); Ollama serves the weights. Then send requests to http://127.0.0.1:8000/v1/systemone as with the bf16 model (curl example on the main card).

llama.cpp (llama-server), with the engine's token ids sent directly:

llama-server -m basal-1.5-4.5B-GGUF/basal-1.5-4.5B-Q8_0.gguf -c 4096 --port 8080
basal-serve --model ./basal-1.5-4.5B-GGUF --mode llamacpp --llamacpp-url http://127.0.0.1:8080 --port 8000

Engines

engine	basal-serve mode	hardware	what it is for	quick start	status (basal-1.5, 4.5B)
basal engine (PyTorch)	fast (also fast-nocompile, fp8, nvfp4); eager --device mps|cpu	NVIDIA GPUs (sm80+); Apple Silicon (MPS) and CPU with eager	the primary server: packed requests (SOAM), evidence spans, fp8 on Hopper and Blackwell	basal-serve --model Remek/basal-1.5-4.5B --mode fast	verified: served check on one H100, 13.7 ms p50 per question over HTTP, 64.4 decisions/s with 32 clients
vLLM	vllm	NVIDIA GPUs	an alternative CUDA server	basal-serve --model Remek/basal-1.5-4.5B --mode vllm (extra basal[vllm])	verified offline (one H100, basal-bench, 1,000 development decisions): agreement 0.994 with the fp32 engine, accuracy 0.928 vs 0.930, 147 decisions/s
SGLang	sglang	NVIDIA GPUs	an alternative CUDA server (RadixAttention prefix sharing)	basal-serve --model Remek/basal-1.5-4.5B --mode sglang (own environment: sglang[srt]==0.5.21, then basal with --no-deps)	verified offline (one H100, basal-bench, 1,000 development decisions): agreement 0.995 with the fp32 engine, accuracy 0.925 vs 0.930, 104.6 decisions/s; on Werdykt the same answer as the basal engine on 98.3% of items (97.0% for states over 8k tokens)
MLX	mlx	Apple Silicon	Macs: -MLX-8bit (recommended) or -MLX-fp4 (half the memory)	basal-serve --model Remek/basal-1.5-4.5B-MLX-8bit --mode mlx (extra basal[mlx])	verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.992 (-MLX-8bit) / 0.970 (-MLX-fp4), 587 / 581 ms per question
Ollama	ollama	CPU, Apple Silicon, consumer GPUs	local and desktop use with Ollama (-GGUF)	1. hf download Remek/basal-1.5-4.5B-GGUF --local-dir basal-1.5-4.5B-GGUF 2. in that folder: ollama create basal-1.5-4.5b:q8_0 -f Modelfile.Q8_0 3. basal-serve --model ./basal-1.5-4.5B-GGUF --mode ollama --ollama-model basal-1.5-4.5b:q8_0	verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.994 (Q8_0) / 0.980 (Q4_K_M), 518 / 542 ms per question
llama.cpp	llamacpp	CPU, Apple Silicon, consumer GPUs	llama-server with the -GGUF files; the engine's token ids are sent as is	1. llama-server -m basal-1.5-4.5B-GGUF/basal-1.5-4.5B-Q8_0.gguf -c 4096 --port 8080 2. basal-serve --model ./basal-1.5-4.5B-GGUF --mode llamacpp	verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.994 (Q8_0) / 0.980 (Q4_K_M), 427 / 446 ms per question

basal-serve is the decision layer in every row. vLLM, SGLang, MLX, Ollama and llama.cpp run the weights; the typed answers (a calibrated probability for each of your options, both option orders, the state shared by several questions, multi, act and facts) come from basal-serve --mode <engine> in front of them, with the same HTTP API in every mode. A plain ollama run or llama-cli gives free text only. Evidence spans need the PyTorch engine (fast, eager); vLLM and SGLang start with a 4,096-token context (longer states are refused); raise it with --max-len (e.g. 32768 for long documents). Install and first request for each engine: quick start.

What differs from the bf16 model in the basal engine

• Same prompt, both option orders, multi, act and "facts": "auto"; the state of a request is computed once by the runtime's prompt cache (requests of one state are sent one after another; --http-parallel N with OLLAMA_NUM_PARALLEL / llama-server -np for concurrent clients).
• The letters are read from the top-20 log-probabilities at the answer position (a letter outside them counts as probability 0).
• CALIBRATION.json holds the bf16 model's temperatures and thresholds, not validated for these quantisations: refit the thresholds on your own labelled requests before automating decisions.
• No evidence spans; context 4,096 tokens by default (num_ctx in the Modelfile, -c for llama-server).
• Ollama 0.12.11 or newer (measured with 0.35.0); llama.cpp measured with 0.5.0 (b11146).

License

Apache-2.0, like basal-1.5-4.5B.