Instructions to use Remek/basal-1.5-4.5B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Remek/basal-1.5-4.5B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Remek/basal-1.5-4.5B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Remek/basal-1.5-4.5B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Remek/basal-1.5-4.5B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Ollama
How to use Remek/basal-1.5-4.5B-GGUF with Ollama:
```
ollama run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Remek/basal-1.5-4.5B-GGUF with Docker Model Runner:
```
docker model run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M
```

Lemonade

How to use Remek/basal-1.5-4.5B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.basal-1.5-4.5B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

basal-1.5-4.5B-GGUF / README.md

Remek

Release card (final draft)

6943511 verified 13 days ago

preview code

Raw History Blame Contribute Delete

7.98 kB

metadata

license: apache-2.0
language:
  - pl
  - en
base_model: Remek/basal-1.5-4.5B
library_name: gguf
pipeline_tag: text-generation
tags:
  - typed-decisions
  - decision-making
  - classification
  - calibration
  - polish
  - gguf
  - ollama

basal-1.5-4.5B-GGUF

GGUF version of basal-1.5-4.5B (4.5B) for Ollama and llama.cpp: CPUs, Apple Silicon and consumer GPUs. See the main card for the description, benchmarks, limitations and license.

file	quantisation	size	agreement with bf16	accuracy change
`basal-1.5-4.5B-Q8_0.gguf`	Q8_0	5.06 GB	0.994	+0.002 (0.938 vs 0.936)
`basal-1.5-4.5B-Q4_K_M.gguf`	Q4_K_M (importance matrix; output head and embeddings Q8_0)	2.93 GB	0.980	±0.000 (0.936 vs 0.936)

Accuracy here is on a 500-decision sample, to check that each file matches its reference; to compare model sizes, see the benchmark table of the main card.

Latency: 427 ms with llama.cpp and 518 ms with Ollama (Q8_0); 446 / 542 ms (Q4_K_M) per decision (Apple M4 Pro, 24 GB, Metal; llama.cpp 0.5.0, Ollama 0.35.0).

Converted with llama.cpp convert_hf_to_gguf.py and llama-quantize, with two fixes that matter for decisions:

Tokenizer fields. The stock conversion writes the same score (−1000) for every token of this tokenizer, so Ollama and other text clients would split prompts differently from the original tokenizer. This export sets the scores from the BPE merge ranks, sets add_space_prefix = false and matches every added token (<|im_start|>, <|im_end|>, …) whole, as the Hugging Face tokenizer does, so prompts are tokenized exactly as by the original tokenizer and Ollama and llama.cpp give the same answers.
Q4_K_M is quantised with an importance matrix computed on decision prompts and keeps the output head and the token embeddings at Q8_0 (the decision is read from the output head's logits of the option letters).

Quick start (Ollama)

basal is a typed-decision model, not a chat model: the basal server runs in front of Ollama, renders the model's own prompt and answer prefix, sends it raw and reads the log-probabilities of the option letters of one generated token. Ollama 0.12.11 or newer.

hf download Remek/basal-1.5-4.5B-GGUF --local-dir basal-1.5-4.5B-GGUF
cd basal-1.5-4.5B-GGUF && ollama create basal-1.5-4.5b:q8_0 -f Modelfile.Q8_0 && cd ..
uv pip install "basal @ https://github.com/rkinas/basal/archive/refs/tags/v1.5.0.tar.gz"
basal-serve --model ./basal-1.5-4.5B-GGUF --mode ollama --ollama-model basal-1.5-4.5b:q8_0 --port 8000

--model gives the tokenizer, chat template and calibration file (this folder or Remek/basal-1.5-4.5B-GGUF); Ollama serves the weights. Then send requests to http://127.0.0.1:8000/v1/systemone as with the bf16 model (curl example on the main card).

llama.cpp (llama-server), with the engine's token ids sent directly:

llama-server -m basal-1.5-4.5B-GGUF/basal-1.5-4.5B-Q8_0.gguf -c 4096 --port 8080
basal-serve --model ./basal-1.5-4.5B-GGUF --mode llamacpp --llamacpp-url http://127.0.0.1:8080 --port 8000

Engines

engine	`basal-serve` mode	hardware	what it is for	quick start	status (basal-1.5, 4.5B)
basal engine (PyTorch)	`fast` (also `fast-nocompile`, `fp8`, `nvfp4`); `eager --device mps\|cpu`	NVIDIA GPUs (sm80+); Apple Silicon (MPS) and CPU with `eager`	the primary server: packed requests (SOAM), evidence spans, `fp8` on Hopper and Blackwell	`basal-serve --model Remek/basal-1.5-4.5B --mode fast`	verified: served check on one H100, 13.7 ms p50 per question over HTTP, 64.4 decisions/s with 32 clients
vLLM	`vllm`	NVIDIA GPUs	an alternative CUDA server	`basal-serve --model Remek/basal-1.5-4.5B --mode vllm` (extra `basal[vllm]`)	verified offline (one H100, `basal-bench`, 1,000 development decisions): agreement 0.994 with the fp32 engine, accuracy 0.928 vs 0.930, 147 decisions/s
SGLang	`sglang`	NVIDIA GPUs	an alternative CUDA server (RadixAttention prefix sharing)	`basal-serve --model Remek/basal-1.5-4.5B --mode sglang` (own environment: `sglang[srt]==0.5.21`, then basal with `--no-deps`)	verified offline (one H100, `basal-bench`, 1,000 development decisions): agreement 0.995 with the fp32 engine, accuracy 0.925 vs 0.930, 104.6 decisions/s; on Werdykt the same answer as the basal engine on 98.3% of items (97.0% for states over 8k tokens)
MLX	`mlx`	Apple Silicon	Macs: `-MLX-8bit` (recommended) or `-MLX-fp4` (half the memory)	`basal-serve --model Remek/basal-1.5-4.5B-MLX-8bit --mode mlx` (extra `basal[mlx]`)	verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.992 (`-MLX-8bit`) / 0.970 (`-MLX-fp4`), 587 / 581 ms per question
Ollama	`ollama`	CPU, Apple Silicon, consumer GPUs	local and desktop use with Ollama (`-GGUF`)	1. `hf download Remek/basal-1.5-4.5B-GGUF --local-dir basal-1.5-4.5B-GGUF` 2. in that folder: `ollama create basal-1.5-4.5b:q8_0 -f Modelfile.Q8_0` 3. `basal-serve --model ./basal-1.5-4.5B-GGUF --mode ollama --ollama-model basal-1.5-4.5b:q8_0`	verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.994 (`Q8_0`) / 0.980 (`Q4_K_M`), 518 / 542 ms per question
llama.cpp	`llamacpp`	CPU, Apple Silicon, consumer GPUs	`llama-server` with the `-GGUF` files; the engine's token ids are sent as is	1. `llama-server -m basal-1.5-4.5B-GGUF/basal-1.5-4.5B-Q8_0.gguf -c 4096 --port 8080` 2. `basal-serve --model ./basal-1.5-4.5B-GGUF --mode llamacpp`	verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.994 (`Q8_0`) / 0.980 (`Q4_K_M`), 427 / 446 ms per question

basal-serve is the decision layer in every row. vLLM, SGLang, MLX, Ollama and llama.cpp run the weights; the typed answers (a calibrated probability for each of your options, both option orders, the state shared by several questions, multi, act and facts) come from basal-serve --mode <engine> in front of them, with the same HTTP API in every mode. A plain ollama run or llama-cli gives free text only. Evidence spans need the PyTorch engine (fast, eager); vLLM and SGLang start with a 4,096-token context (longer states are refused); raise it with --max-len (e.g. 32768 for long documents). Install and first request for each engine: quick start.

What differs from the bf16 model in the basal engine

Same prompt, both option orders, multi, act and "facts": "auto"; the state of a request is computed once by the runtime's prompt cache (requests of one state are sent one after another; --http-parallel N with OLLAMA_NUM_PARALLEL / llama-server -np for concurrent clients).
The letters are read from the top-20 log-probabilities at the answer position (a letter outside them counts as probability 0).
CALIBRATION.json holds the bf16 model's temperatures and thresholds, not validated for these quantisations: refit the thresholds on your own labelled requests before automating decisions.
No evidence spans; context 4,096 tokens by default (num_ctx in the Modelfile, -c for llama-server).
Ollama 0.12.11 or newer (measured with 0.35.0); llama.cpp measured with 0.5.0 (b11146).

License

Apache-2.0, like basal-1.5-4.5B.