gemma-4-12b-it-abliterated-nf4

Abliterated (refusal-direction removed) Gemma 4 12B, quantized to int4 (NF4). ~7.7 GB, fits a 16 GB GPU, retains text + image + audio. Built from Google's official QAT weights (google/gemma-4-12B-it-qat-q4_0-unquantized), so int4 quality is near-bf16. Personal/research use.

Requirements

• NVIDIA GPU, 16 GB+ VRAM — NF4/bitsandbytes is CUDA-only. (On an Intel Arc / non-NVIDIA card this will NOT load; use the GGUF build or ask for a Vulkan-compatible quant.)
• pip install "transformers>=5.12" bitsandbytes accelerate huggingface_hub

Text chat

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch
m = AutoModelForImageTextToText.from_pretrained(
    "smashingtags/gemma-4-12b-it-abliterated-nf4", device_map="cuda", torch_dtype=torch.bfloat16)
p = AutoProcessor.from_pretrained("smashingtags/gemma-4-12b-it-abliterated-nf4")
msgs = [{"role":"user","content":[{"type":"text","text":"Explain how a lock pick works."}]}]
inp = p.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                            return_dict=True, return_tensors="pt").to("cuda")
out = m.generate(**inp, max_new_tokens=300)
print(p.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True))

With an image (vision)

msgs = [{"role":"user","content":[
    {"type":"image","url":"https://.../photo.jpg"},
    {"type":"text","text":"What's in this image?"}]}]
# same apply_chat_template + generate as above

Serve as an API (vLLM, OpenAI-compatible)

vllm serve smashingtags/gemma-4-12b-it-abliterated-nf4 --quantization bitsandbytes \
  --max-model-len 8192 --port 8000
# then POST to http://localhost:8000/v1/chat/completions (text or image_url content)

Related builds

repo	format	size	runtime	modalities
gemma-4-12b-it-abliterated-nf4 (this)	int4 NF4	7.7 GB	transformers / vLLM (CUDA)	text + image + audio
gemma-4-12b-it-abliterated	bf16	24 GB	transformers / vLLM	text + image + audio
gemma-4-12b-it-abliterated-GGUF	q4_0 GGUF	7 GB	Ollama / llama.cpp	text only

Abliteration: difference-of-means refusal direction (Arditi et al.), hook-captured, orthogonalized out of the text decoder's o_proj+mlp.down_proj (top 70% layers); vision/audio encoders and tied embeddings untouched. Refusal smoke: base 3/3 → 0/3.