Instructions to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8")
model = AutoModelForMultimodalLM.from_pretrained("JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8

SGLang

How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with Docker Model Runner:
```
docker model run hf.co/JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8
```

Clef (27B) — GPTQ W4A16 g64 with INT8 embeddings, served with vLLM (18.0 GB)

Install from this card, not from generic snippets. Clef is a decision model, not a chat model, and its token embeddings are quantized: follow Requirements & install and Quick start.

A 4-bit weight quantization of Cloudflare/clef, Cloudflare's decision model: a Qwen3.8-27B backbone plus a "joint schema head" that scores every option of every question in a schema (choice / score / true-false) in one forward pass. It does not generate text.

18.0 GB (16.8 GiB) (vs 54 GB BF16): fits one 24 GB GPU with room for the KV cache.
Same decisions as BF16 on our test: 65/68, and a mean KL divergence of the option probabilities of 0.0006 against the BF16 model.
Served with vLLM (pooling runner) + Cloudflare's head on top: 0.16 s per record on one L40S. A small wrapper is included (scripts/clef_vllm.py).

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration data include real sysadmin/agent conversations in Spanish, and our test for Clef is a set of Spanish sysadmin decisions (see below). We use Clef as a fast guard and triage component next to the agent: classify the risk of a command before running it, and triage log lines. Numbers for other domains or languages may differ.

What is in this repository

Branch	Size
`main`	18.0 GB of weights

Download a branch with hf download JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 --revision <branch> --local-dir <dir>.

Results

Test: 50 sysadmin records in Spanish, 68 questions with known answers: the risk of a shell command (read-only / reversible / destructive, 32 questions), and for log lines the affected component (GPU / disk / network / memory / access / service, 18) and whether someone must act (18). "KL" is the KL divergence of each question's option probabilities (BF16 → this checkpoint), averaged over the 68 questions.

Checkpoint	Size	Correct	Risk	Component	Act?	KL vs BF16 (mean / max)	Time per record
Cloudflare/clef BF16 (transformers, partly offloaded to RAM on 48 GB)	54 GB	65/68	32/32	18/18	15/18	—	1.51 s
This repo, transformers (dequantized copy)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.007	—
This repo, vLLM + head (`scripts/clef_vllm.py`)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.009	0.16 s
Vishva007/clef-W4A16-AutoRound (for comparison, same test and code)	19.4 GB	65/68	32/32	18/18	15/18	0.0011 / 0.013	0.16 s

68 questions is a small test: treat differences of ±0.001 in KL as indicative. All measured on one NVIDIA L40S (48 GB).

Sampling and repetition loops

Not applicable: Clef does not generate text (it scores the options of a schema in one forward pass).

Requirements & install (vLLM + vllm-quantpack)

Before you install (checked on a clean Ubuntu 24.04 host with an A40, 2026-10-28):

An NVIDIA driver that supports CUDA 13.2 (tested with 595.84) and uv.
The Python 3.12 headers and a C compiler. uv venv --python 3.12 uses the system Python 3.12 when there is one, and Triton compiles a small C helper every time vLLM starts; without the headers every vllm serve stops with fatal error: Python.h: No such file or directory. On Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y python3.12-dev build-essential.
No CUDA toolkit is needed.
Disk: about 20 GB for the venv, as much again for uv's cache (uv cache clean frees it), plus the model.

Stock vLLM (0.29 to 0.31) does not pass quant_config to the embedding layer of qwen3_5-family models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Our vllm-quantpack plugin (on PyPI) fixes that from outside vLLM, without patching it:

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=auto
uv pip install --python venv/bin/python vllm-quantpack
venv/bin/hf download JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 --local-dir clef-W4A16

Plugin (2026-10-28): the 4-line vLLM patch is replaced by vllm-quantpack (pip install vllm-quantpack), which loads the quantized embeddings on unpatched vLLM. The measurements in this card were made with the previous stack (vLLM 0.31.0 + the embedding patch); this checkpoint was then served again with vllm-quantpack 0.1.0 following this card on a clean install (its Quick start record scored as expected: destructivo 0.974). The previous files stay in patches/ for reference.

Quick start

cat > record.json <<'EOF'
{"state": "Comando propuesto por el agente: rm -rf /var/lib/docker/volumes/*",
 "questions": {"riesgo": {"type": "choice", "instructions": "¿Qué riesgo tiene ejecutar este comando?",
                          "criteria": {"lectura": "Solo lee", "reversible": "Cambia algo que se puede deshacer", "destructivo": "Borra o rompe algo sin vuelta atrás"}}}}
EOF
venv/bin/python clef-W4A16/scripts/clef_vllm.py clef-W4A16 record.json

scripts/clef_vllm.py loads the backbone in vLLM's pooling runner (runner="pooling", convert="embed", PoolerConfig(task="token_embed", tok_pooling_type="ALL", use_activation=False)), takes the final hidden states of every token and applies the joint schema head with the BF16 lm_head matrix, exactly as Cloudflare's ClefModel does. For a server, wrap decide() in your HTTP framework of choice. For the original API (/v1/systemone) and multimodal records, see Cloudflare's model card.

Validation of the vLLM path: on the BF16 9B sibling (clef-flash), vLLM's hidden states match transformers' with a per-token cosine of 0.9995; served this way, the BF16 model gives the same answers as Cloudflare's loader (KL 0.00003).

Using it with opencode

Not applicable: Clef is not a chat model. We call it from scripts next to the agent (see Quick start), for example to classify the risk of a command before opencode runs it.

Important notes

Nothing beyond Quick start and Limitations.

Quantization recipe

Part	Treatment
Linear layers of the language model (attention, MLP)	GPTQ W4A16, group size 64, symmetric (llm-compressor)
`embed_tokens`	INT8, group size 128 (saves ≈1.3 GB; needs the vllm-quantpack plugin below)
`lm_head`	BF16 (unquantized): Clef's head builds the option vectors from the output embedding matrix
Joint schema head (`joint_head.safetensors`)	BF16, unchanged
Vision tower, GatedDeltaNet `in_proj_a/b`, norms	BF16, unchanged
Calibration	512 × 2,048 tokens: our sysadmin/agent conversation trace (Spanish and English) mixed with UltraChat

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with sudo, and give the agent a non-privileged account and a test machine where you can.
Text records only: the vision tower is kept in BF16 but images and video were not tested.
Evaluated on our own small Spanish sysadmin test, not on Cloudflare's Decision Index.
The joint_schema_model.py file is Cloudflare's code, unchanged; it is imported by scripts/clef_vllm.py.

Feedback welcome

Reports are welcome in the Community tab: GPU, vLLM version, your use case, and whether the decisions matched the BF16 model.

Credits & license

Apache License 2.0, as Cloudflare/clef (and its base, Qwen3.8-27B). See NOTICE. Unofficial quantization by JRamirez-UAB; not affiliated with Cloudflare, Alibaba Cloud or the Qwen team.