Muse Glimmer 30B - Heretic Abliterated (BF16)

v2 Release - Significantly improved abliteration using 500 Optuna trials with Heretic.

Results

Version	Refusals	Compliance	KL Divergence	Trials
v2 (current)	6.5%	93.5%	0.076	500
v1	29%	71%	0.027	50

The v2 release achieves an 88% refusal reduction over v1 while maintaining strong model quality (KL=0.076).

Methodology

This model was abliterated using Heretic, a state-of-the-art refusal vector removal tool that uses LoRA adapters and Optuna hyperparameter optimization.

Abliteration Pipeline

Refusal Direction Computation: Refusal directions were computed across all 52 transformer layers by comparing residual stream activations between harmful and harmless prompts from the mlabonne/harmful_behaviors and mlabonne/harmless_alpaca datasets.
Optuna Optimization: 500 trials were run optimizing for minimum refusal rate while preserving model quality (measured via KL divergence). Each trial configures weight parameters for the attn.o_proj and mlp.down_proj components across layers.
LoRA Abliteration: The best trial parameters are applied as LoRA adapters to the target weight matrices, projecting out the refusal direction from the model's representations.
Weight Merging: LoRA adapters are merged back into the base weights, producing a clean BF16 model with no adapter overhead.

Best Trial (Trial 445 of 500)

Refusal rate: 6.5% (93.5% compliance)
KL divergence: 0.076
direction_index: 40.73

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer

model = AutoModelForImageTextToText.from_pretrained(
    "mlasli/Muse-Glimmer-30B-Heretic-Abliterated-BF16",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mlasli/Muse-Glimmer-30B-Heretic-Abliterated-BF16")