Instructions to use batiai/gemma-4-12B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use batiai/gemma-4-12B-it-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Use Docker

docker model run hf.co/batiai/gemma-4-12B-it-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use batiai/gemma-4-12B-it-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "batiai/gemma-4-12B-it-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "batiai/gemma-4-12B-it-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/batiai/gemma-4-12B-it-GGUF:Q4_K_M

Ollama
How to use batiai/gemma-4-12B-it-GGUF with Ollama:
```
ollama run hf.co/batiai/gemma-4-12B-it-GGUF:Q4_K_M
```
Unsloth Desktop

How to use batiai/gemma-4-12B-it-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "batiai/gemma-4-12B-it-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use batiai/gemma-4-12B-it-GGUF with Docker Model Runner:
```
docker model run hf.co/batiai/gemma-4-12B-it-GGUF:Q4_K_M
```

Lemonade

How to use batiai/gemma-4-12B-it-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull batiai/gemma-4-12B-it-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-12B-it-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use batiai/gemma-4-12B-it-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default batiai/gemma-4-12B-it-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use batiai/gemma-4-12B-it-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf batiai/gemma-4-12B-it-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "batiai/gemma-4-12B-it-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma 4 12B-it GGUF — Quantized by BatiAI

Optimized GGUF quantizations of google/gemma-4-12B-it — Google DeepMind's encoder-free multimodal model (text + image + audio + video) that runs on a 16 GB Mac. Built directly from official Google BF16 weights by BatiAI for BatiFlow.

Why Gemma 4 12B?

26B-MoE-class quality at <half the memory — Google's release notes put the 12B near the Gemma 4 26B MoE on standard benchmarks
Encoder-free multimodal — first mid-sized model with native audio. Raw image patches and audio waveforms project directly into the LLM via lightweight linear layers — no separate vision/audio encoder
Strong vision + reasoning: DocVQA 94.9 · InfoVQA 88.4 · MMMU-Pro 69.1 · AIME 2026 77.5 · MATH-Vision 79.7
256K context, 140+ languages
Apache 2.0 — commercial-friendly
Released June 3, 2026

Quick Start

ollama pull batiai/gemma4-12b:q4

Available Quantizations

Quant	Size	Recommended For
Q2_K_S	~4.2 GB	8GB Mac, ultra-compact (imatrix)
IQ3_XXS	~4.6 GB	imatrix, smallest
Q3_K_M	~5.7 GB	8GB+ Mac, balanced
IQ4_XS	~6.2 GB	imatrix, best size/quality
Q4_K_M	~6.9 GB	16GB Mac (recommended)
Q6_K	~9.2 GB	near-original quality

Mac note on Q3_K_M: in every model we've benchmarked on Apple Silicon, Q3_K_M generated slower than Q4_K_M despite the smaller file — Granite 4.1 (+27%), Gemma 4 26B (+12%), Qwen3.8‑27B (+18%), Qwen3.6‑27B (+8%), on both M4 Max and M4 mini. Metal's Q3_K path is limited by dequantization compute rather than bandwidth. We have not measured this particular model's Q3/Q4 pair yet, so treat it as a strong prior, not a measurement: if Q4_K_M fits, take it. On CUDA the two are effectively tied, so this applies to Macs only.

IQ variants use imatrix calibration. Q2/Q3 (8GB Macs) are not in Google's reference set — BatiAI adds them.

Two modes — text-only by default, multimodal opt-in

Upstream Gemma 4 12B-it is multimodal. In the GGUF ecosystem this is delivered as two files: a main model GGUF (the LLM) and a small mmproj GGUF holding the vision + audio projectors together (encoder-free → the projector is tiny, ~167 MB, and one BF16 mmproj works with every quant).

What actually works locally (verified on M4 Max 128GB):

Modality	Status	How
Text	✅	Ollama or llama.cpp. ~45 tok/s (q4, M4 Max)
Image (OCR, captioning, visual reasoning)	✅	`llama-server` + mmproj — verified (Korean screenshot OCR, ~13 s/image)
Audio (speech understanding, ≤30s)	✅ experimental	`llama-server` + mmproj — verified (Korean call STT). `init_audio` flags reduced quality
Video (≤60s)	❌	Model supports it, but llama.cpp has no gemma4 video projector yet — image+audio only for now

⚠️ Ollama can't do images/audio for Gemma 4 yet (0.20 doesn't know the gemma4uv/gemma4ua projectors). Multimodal needs llama-server built from a recent llama.cpp master that includes the Gemma 4 projectors (the gemma4v/gemma4uv/gemma4a/gemma4ua clip graphs). Older builds fail with unknown projector type: gemma4uv.

# Multimodal via llama-server (recent llama.cpp master + --jinja required)
llama-server -m gemma-4-12B-it-Q4_K_M.gguf \
  --mmproj mmproj-google-gemma-4-12B-it-BF16.gguf \
  --host 127.0.0.1 --port 8899 -ngl 99 -c 8192 --jinja
# → GET /props returns {"vision": true, "audio": true}; POST /v1/chat/completions with image_url / input_audio

Gemma 4 12B is a reasoning model — give it enough max_tokens (image descriptions emit 700+ tokens incl. a thinking block; the answer arrives in reasoning_content + content).

RAM Requirements

Your Mac RAM	Q2	IQ3	Q3	IQ4	Q4	Q6
8GB	✅	✅	✅ tight	⚠️	❌	❌
16GB	✅	✅	✅	✅	✅ Recommended	✅
24GB+	✅	✅	✅	✅	✅	✅

Why BatiAI Quantization?

	BatiAI	Third-party
Source	Official Google weights	Re-quantized
imatrix	✅ IQ variants calibrated	varies
Low quants	✅ Q2/Q3 for 8GB Macs	often Q4 floor
mmproj (vision+audio)	✅ included	often text-only
Tool calling	✅ Verified	often untested
BatiAI signed	✅ `general.author=BatiAI`	—

Technical Details

Original Model: google/gemma-4-12B-it
Architecture: gemma4_unified — encoder-free unified multimodal, 12B dense, 48 layers, 256K context
Modalities: text + image + audio (≤30s) + video (≤60s)
License: Apache 2.0
Quantized with: llama.cpp + imatrix
Quantized by: BatiAI

File checksums (SHA-256)

File	SHA-256
`google-gemma-4-12B-it-Q2_K_S.gguf`	`fd0c02b4d06272127e10eaedcdf240535317bb13fabbd4c63d52b8929f18f99c`
`google-gemma-4-12B-it-IQ3_XXS.gguf`	`21a4aaeadc6b17a4e9e168034f65f641f13f122f88b24b50776835dc02b84f7f`
`google-gemma-4-12B-it-Q3_K_M.gguf`	`f3f3549c7e1389452cd8df63db58156e3a66555fcd5047150c8d78ca48c459b6`
`google-gemma-4-12B-it-IQ4_XS.gguf`	`63d7370e4de1aba0cb7dd04382e4e52c6803eb8dd276f016367f144bae78eed8`
`google-gemma-4-12B-it-Q4_K_M.gguf`	`e6db34561a77012484641bf35cb14b09a8ef15dbeaecb29cf3939e66e3a6dbad`
`google-gemma-4-12B-it-Q6_K.gguf`	`36f399a3348a001ffc13b61ac6951a5028ca63f8889f99ad56f4d98210bc8a9f`
`mmproj-google-gemma-4-12B-it-BF16.gguf`	`4aa2bb56fc57d0b7d27d3ea77a0df1c424f7968011d9b4a581bf8ecda8857139`

shasum -a 256 google-gemma-4-12B-it-Q4_K_M.gguf   # verify after download

About BatiFlow

BatiFlow — free, on-device AI automation for Mac. 5MB app, 100% local, unlimited.

License

Quantized from google/gemma-4-12B-it. License: Apache 2.0.

Benchmarks

Text (ollama run --verbose, thinking ON default):

Machine	Quant	Load	Warm gen	Prompt eval	Long resp	Cold 1st gen	Ollama RAM	Korean	Tool-call
M4 Max 128 GB	Q4_K_M	2.8 s	45.1 t/s	536 t/s	22.7 t/s	54.5 t/s	16 GB	✅	✅
M4 Max 128 GB	Q6_K	7.1 s	38.9 t/s	506 t/s	16.2 t/s	43.2 t/s	18 GB	✅	✅

Multimodal (llama-server static commit 308f61c + Q4_K_M + mmproj, M4 Max 128GB — measured):

Input	Latency	Throughput	Note
Cold start (incl. mmproj)	2–6 s	—	mmproj worst-case ~359 MiB
Image (screenshot → full Korean analysis)	~22 s	42 t/s gen, 949 tok	OCR + visual reasoning, accurate
Audio (10 s clip → Korean STT)	~23 s	RTF ~2.3	speech transcription, experimental

/props → {"vision": true, "audio": true}. Gemma 4 is a reasoning model — responses include a <think> block, so give generous max_tokens. Video not supported (no gemma4 video projector in llama.cpp).