Instructions to use Remek/basal-1.5-4.5B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Remek/basal-1.5-4.5B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Remek/basal-1.5-4.5B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Remek/basal-1.5-4.5B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Remek/basal-1.5-4.5B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Ollama
How to use Remek/basal-1.5-4.5B-GGUF with Ollama:
```
ollama run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Remek/basal-1.5-4.5B-GGUF with Docker Model Runner:
```
docker model run hf.co/Remek/basal-1.5-4.5B-GGUF:Q4_K_M
```

Lemonade

How to use Remek/basal-1.5-4.5B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Remek/basal-1.5-4.5B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.basal-1.5-4.5B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Remek commited on 14 days ago

Commit

bc4f975

verified ·

1 Parent(s): 2764aee

README: measured quality and speed (draft)

Browse files

Files changed (1) hide show

README.md +12 -0

README.md CHANGED Viewed

@@ -46,6 +46,18 @@ basal-serve --model Remek/basal-1.5-4.5B-GGUF --mode llamacpp --llamacpp-url htt
 - `tokenizer_config.json`
 - `README.md`
 ## Notes
 - `--model` points at this folder (or repo) for the tokenizer, the chat template and `CALIBRATION.json`; the GGUF file itself is served by Ollama or llama.cpp.

 - `tokenizer_config.json`
 - `README.md`
+## Quality and speed of this port
+Measured on Apple M4 Pro (10+4 CPU, 20 GPU cores), 24 GB, macOS 15.7 against the PyTorch reference (`torch-bf16`) on 500 held-out development decisions (both option orders, calibrated): basal-1.5-4.5B (source tag M15C_45); reference torch 2.14 bf16 on MPS; mlx 0.32.3 / mlx-lm 0.32.0, Ollama 0.35.0, llama.cpp 0.5.0 (b11146). Shipped files: mlx-8bit-linux / mlx-fp4-linux are the uploaded -MLX-8bit / -MLX-fp4 exports (made on the Modal Linux CPU container; no latency run, same format as the Mac exports mlx-8bit / mlx-fp4, which carry the latency); gguf-* are the uploaded -GGUF files (Q4_K_M with importance matrix)..
+| engine | argmax agreement | accuracy | mean abs. prob. diff. | ECE | 1 question / state | 5 questions / state |
+|---|---|---|---|---|---|---|
+| torch-bf16 | 1.000 | 0.934 | 0.0000 | 0.033 | 823 ms | 4157 ms |
+| gguf-q8_0-llamacpp | 0.994 | 0.936 | 0.0024 | 0.035 | 432 ms | 1699 ms |
+| gguf-q8_0-ollama | 0.994 | 0.936 | 0.0024 | 0.035 | 453 ms | 1804 ms |
+| gguf-q4_k_m-llamacpp | 0.970 | 0.926 | 0.0137 | 0.031 | 458 ms | 1801 ms |
+| gguf-q4_k_m-ollama | 0.970 | 0.926 | 0.0137 | 0.031 | 504 ms | 1833 ms |
 ## Notes
 - `--model` points at this folder (or repo) for the tokenizer, the chat template and `CALIBRATION.json`; the GGUF file itself is served by Ollama or llama.cpp.