Instructions to use YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma-4-12B-it (LEAN QAT, uniform 4-bit)

What this is

A lean MLX build of Google's Gemma4 quantization-aware-trained (QAT) checkpoint, converted directly from the QAT bf16 source weights with mlx-lm's own model-default quantization predicate -- not a third-party mixed-precision recipe. The MLP projections that most published MLX Gemma4 QAT builds silently upgrade to 8-bit stay at 4-bit / group-size-64 here (see Quantization policy below for this model's exact predicate, including whether it has an MoE router exception). The result is QAT quality at post-training-quantization (PTQ) size: accuracy calibrated for 4-bit weights, at the size a plain 4-bit build should actually be.

This is a LEAN build: the point is that the weights are exactly as small as the declared quantization policy promises, verifiable from config.json alone (see Validation below).

Provenance

Source checkpoint: google/gemma-4-12B-it-qat-q4_0-unquantized, revision a89c069a80c767b0d378c4806b2953ae9d2c711d
Conversion tool: mlx-lm 0.31.3 (mlx_lm.convert, programmatic API)
Quantization policy: uniform 4-bit group-64, zero per-tensor exceptions
Conversion script: scripts/convert_qat_lean.py (infinite repo, Yooz Labs) -- QUANT_POLICY["12b"]

Size

Build	Source	bits/weight	Size (GB)
This build	QAT (uniform 4-bit)	4.50	6.70
Community MLX build	QAT (mixed-precision)	-	10.99 (qat-4bit, mixed precision)
Google GGUF Q4_0	QAT	~4.5	6.98

All sizes are on-disk decimal GB (safetensors shard bytes on disk); bits/weight is the effective average over all quantized tensors (packed weight bytes plus scale/bias bytes, divided by total parameter count), computed by scripts/convert_qat_lean.py verify.

Runtime memory (measured, not estimated)

Metric	Community build	This build
Peak memory footprint, 8K context (GB)	15.0	10.7
Peak memory footprint, 32K context (GB)	17.2	12.9
Peak RSS, 32K context (GB)	12.1	7.8
Decode speed, 32K context (tokens/s)	20.2	30.5
Prefill speed, 32K context (tokens/s)	234	238

Community build: mlx-community/gemma-4-12B-it-qat-4bit. Measured by the benchmark harness during real retrieval runs (peak of the serving process across all tasks per context length) on Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16. Weight quantization pays off at runtime, not just on disk: every tensor a mixed-precision build silently upgrades to 8-bit stays resident at 8-bit.

Validation

Long-context retrieval A/B against mlx-community/gemma-4-12B-it-qat-4bit on the same stack (Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16): needle-in-a-haystack (NIAH) and RULER-style tasks scored per-word, MRCR scored by sequence ratio.

Task	8K community	8K this build	32K community	32K this build
NIAH single	1.00	1.00	1.00	1.00
NIAH multikey	1.00	1.00	1.00	1.00
Variable tracking (VT)	0.35	0.42	0.45	0.55
Frequent words (FWE)	0.72*	0.81*	0.60*	0.56*
MRCR 2-needle	0.69	0.69	0.73	0.64

n=10 per cell; * = adjudicated at n=30. Quality parity with the community QAT build. The FWE cells were adjudicated at n=30 (0.81 vs 0.72 at 8K, 0.56 vs 0.60 at 32K); the remaining deltas are within the n=10 noise band established during validation.

scripts/convert_qat_lean.py verify --path <this build> --model 12b confirms the quantization block in config.json carries no per-tensor overrides beyond uniform 4-bit group-64, zero per-tensor exceptions.

Usage

Loading note (mlx-lm <= 0.31.3): this build keeps its true model_type: gemma4_unified, which stock mlx-lm does not register until ml-explore/mlx-lm#1386 merges. Until then, alias it before loading:
import mlx_lm.utils as u
u.MODEL_REMAPPING["gemma4_unified"] = "gemma4"

from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List the first five prime numbers."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True))

About Yooz Labs

Maintained by Yooz Labs -- sovereign, on-device AI for macOS. Everything we ship runs locally; nothing goes to a cloud. This build comes out of our long-context research line, where weight footprint is the budget that decides how much context fits on a real machine -- which is why the lean conversion exists.

Questions, issues, or corrections: dev@yooz.info

License and attribution

Apache 2.0, inherited from the source checkpoint google/gemma-4-12B-it-qat-q4_0-unquantized (Google's Gemma4 QAT release; license verified apache-2.0 on the google/ repos). This is an independent, unofficial conversion; it is not distributed, endorsed, or certified by Google.