alexwkleung/Mellum2.1-12B-A2.5B-Thinking-Abliterated-GGUF

Mellum2.1-12B-A2.5B-Thinking-Abliterated — GGUF

An abliterated build of JetBrains/Mellum2.1-12B-A2.5B-Thinking (12B total / ~2.5B active MoE, 64 experts top-8, a code-focused thinking model). It was produced with Heretic and quantized with llama.cpp. Every claim below is measured against a stock baseline built through the same pipeline.

Modification notice. These weights are a modified derivative. Directional ablation altered 662 tensors to reduce refusals (see What changed). Nothing else was retrained. JetBrains did not produce, endorse, or review this build, and its behavior is not theirs.

Read this first

This is an escape hatch for over-refusal, not an uncensored model. Abliteration removes a refusal direction from the residual stream; it adds no knowledge. This build knows what stock Mellum knows and nothing more. What changed is its willingness to say it. If stock answers your prompt, use stock.

It is partial. Refusals dropped from ~all to roughly a third of the held-out set (below), and the drop is uneven by topic. Expect some requests to still be refused.

The hazard worth naming: refusals double as hedging. Remove them and you get a model answering fluently in domains where it is unreliable, with none of the cues that used to mark the boundary. Factual reliability is unchanged; only the signalling is gone. You are accountable for what you generate and for applicable law.

Measurements

All on Q4_K_M against a stock Q4_K_M produced by the same converter and quantizer (sha256 51e4c081…5fa8f; not byte-identical to JetBrains' own Q4_K_M, which is why it was built rather than reused). Refusals and GSM8K use temp 1.0 / top-p 0.95 / top-k 20, seeded per item so both builds see the same randomness. Refusals use mlabonne/harmful_behaviors test[60:], the 44 prompts Heretic's optimiser never saw (it scored test[:60]). They are classified by Heretic's marker list on the answer text, reasoning excluded.

Refusals (lower = complies more):

regime	stock	abliterated
enable_thinking: false	43/44	14/44
enable_thinking: true	44/44	17/44

• Non-degenerate. None of the compliances loops or is empty. The median answer is 2–3k characters. Reading the openers, a handful are benign substitutions or safety-framed rewrites of the request, so the marker count slightly overstates compliance.
• Uneven by topic. Misinformation and weapons/violence prompts mostly comply now, and cyber prompts about two-thirds of the time. Fraud/theft is mixed. Drugs stayed refused in both regimes (0/2), a small sample but a consistent one.
• Answers to these prompts are longer. With thinking off they run ~3x stock's length (1041 vs 336 tokens), because the model now writes the answer it used to decline.

GSM8K, paired, n=100, enable_thinking: false:

	correct	truncated (1536 tokens)
stock	90/100	10
abliterated	89/100	10

Discordant pairs split 4/3 (McNemar exact p = 1.000), so no math regression is detectable. Abliteration damage tends to land in math reasoning, which is why this is the check that matters. n=100 rules out a large regression, not a 2–3 pp one. The truncations are the base model being verbose, not the ablation: both builds hit the cap equally.

Coding (3 tasks, one sampled run each, thinking on, card sampling): no regression seen. Both builds made the same correct fix to an interval-merging bug and wrote a sound async httpx retry refactor. On an ISO-8601 duration parser, stock thought past an 8192-token cap without answering while the abliterated build answered. One run each, so treat that as anecdote, not a difference.

Unmeasured: MMLU, long context, non-English, agentic/tool-calling loops.

A note on "thinking off"

enable_thinking: false makes the template emit a closed <think>\n\n</think> block, but on non-trivial prompts Mellum opens a new think block itself and reasons anyway. Stock did this on 41/44 of the prompts above. That is base-model behavior and applies to this build equally. Read "thinking off" as "not prompted to think", and budget max_tokens accordingly.

Files

file	size
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.gguf	7.52 GiB
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q5_K_M.gguf	8.58 GiB
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q6_K.gguf	10.13 GiB
Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q8_0.gguf	12.04 GiB

All quantized from one bf16 GGUF, without an imatrix (Q4_K_M and up don't require one). build/ holds the search record, not model files: the log (run.log), the list of modified tensors (heretic_deltas.json), and the full Optuna study journal (heretic-study.jsonl, all 100 trials).

Usage

Needs a llama.cpp recent enough to know the mellum architecture (built and tested at d7bd3bfc). The sampling is JetBrains' card values:

llama-server -m Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.gguf \
  -ngl 99 -c 32768 -fa on --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

Thinking is on by default; pass "chat_template_kwargs": {"enable_thinking": false} per request to skip the prompted think block (see the note above). Send a large max_tokens, or none. This model reasons at length, and a small budget is consumed inside <think>, returning empty content with finish_reason: "length".

What changed

From the merge-time delta list, computed on the bf16 weights before quantization:

tensor type	changed	layers
routed-expert down_proj	640	12–21, all 64 experts in each
attention o_proj	22	6–27
everything else (norms, embeddings, q/k/v, router, experts' gate/up)	0	—

Layers 0–5 are untouched.

Reproducing

Stock Heretic silently abliterates attention only on this model. Transformers 5 loads Mellum's experts as one fused module with 3D gate_up_proj / down_proj parameters. Heretic 1.4 iterates layer.mlp.experts inside a suppress(Exception), which fails on a fused module, so the experts are skipped without a word. PEFT 0.21 separately strips per-expert targets from target_modules for models with a checkpoint conversion mapping. Either drop alone leaves a run that changes only the 28 o_proj modules while reporting success. On any MoE, check Heretic's "Abliterable components" line: mlp.down_proj should be in the thousands (1792 here), not missing.

This build split the fused experts back into per-expert Linears named like the checkpoint tensors. It verified fused vs. unfused logits on the real weights (KL 0.0009), bypassed the PEFT rewrite for those modules, and merged the LoRA deltas straight onto the original per-expert safetensors. The search itself is stock Heretic 1.4.0 (transformers 5.19.0, peft 0.21.2) with that patch loaded in-process. Run bare, the command below reproduces the attention-only failure, not this build:

heretic JetBrains/Mellum2.1-12B-A2.5B-Thinking \
  --response-prefix $'<think>\n\n</think>\n\n' \
  --seed 1234 --max-batch-size 128 --max-response-length 64 \
  --n-trials 100 --n-startup-trials 40 \
  --good-evaluation-prompts '{"dataset":"mlabonne/harmless_alpaca","split":"test[:60]","column":"text"}' \
  --bad-evaluation-prompts '{"dataset":"mlabonne/harmful_behaviors","split":"test[:60]","column":"text"}'

--response-prefix is byte-identical to the served enable_thinking: false prompt, so scoring happens in that regime rather than on reasoning text. Abliteration is not bit-reproducible (the TPE search lands on a different trial each run). The selected trial was 62: 7/60 refusals @ KL 0.1162, from a baseline of 53/60, per-layer directions. Its parameters, from the study journal (Heretic's trial 62 is Optuna trial number 61; min_weight is shown as Heretic reports it, i.e. the stored fraction × max_weight):

attn.o_proj    : max_weight 1.21318130  max_weight_position 19.66037967  min_weight 0.93951503  min_weight_distance 14.02125443
mlp.down_proj  : max_weight 1.35222106  max_weight_position 16.43450663  min_weight 0.69089782  min_weight_distance  5.35777097

Verifying

2f27b1e83390e6f259cebe1f8f0e343ef990b5ec938abd99bb4c33ba1c1fc7af  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q4_K_M.gguf
6167587537cabd009e3e97b59cca9c25292926c28027e4c47d1cba20f66c349d  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q5_K_M.gguf
bef15ed028f483e18cb58ecaf64672a73b52a9056d7edde797624cc1cbccfe9c  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q6_K.gguf
6e7d2cbb6f320006c5ce841834ab53cd9ac85dacab480fd358e14407257a340f  Mellum2.1-12B-A2.5B-Thinking-Abliterated-Q8_0.gguf

License

Apache-2.0, inherited from JetBrains/Mellum2.1-12B-A2.5B-Thinking, whose repo declares apache-2.0 in its metadata but ships no LICENSE file (nor does any JetBrains Mellum repo). The LICENSE here is the canonical Apache License 2.0 text from apache.org, unmodified. No NOTICE file exists upstream, so none is propagated. As required by §4(b), this card states that the files were modified (see the notice at the top). Abliteration and quantization change nothing about the license.

Credits

• JetBrains: the Mellum2.1 base model
• p-e-w/heretic: the abliteration method and tooling
• ggml-org/llama.cpp: conversion and quantization
• mlabonne: the prompt sets used for directions and scoring

Tested setup

I've run the Q4_K_M quant via llama.cpp in Reika on an M2 MacBook Air (16GB) at 24k context, with the default f16 KV cache. This was general use and a few informal evals, not a benchmark.

Supporting my work

If these quants are useful to you, consider supporting them via Ko-fi. It goes toward local hardware for quantizing and testing larger models. I also build Reika, a coding agent CLI designed around small local models first.