Instructions to use gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16

SGLang

How to use gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 with Docker Model Runner:
```
docker model run hf.co/gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16
```

Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16

W4A16 (int4) quantization of gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic — the Heretic-abliterated build of DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1.

Runs the full vision-language model with MTP / DFlash2 speculative decoding on a single 24 GB GPU (e.g. RTX 4090). ~19 GB on disk. Credit for all capability goes to DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1; this repo only removes refusals and quantizes.

⚠️ Disclaimer — read before use

This model has had its safety alignment removed via abliteration. It will comply with harmful requests the base model would refuse. Released for research (interpretability, red-teaming, robustness). You own what you do with it and what it generates. Add your own moderation before any deployment. Use must comply with the inherited Apache 2.0 license and applicable law. No liability is accepted for misuse.

Quantization

Produced with AutoRound, served with compressed-tensors in vLLM.

Component	Precision
Language-decoder `Linear` layers	int4, group 128, symmetric (W4A16)
`lm_head`, `embed_tokens`, MTP module	int8, group 128
Vision tower (`visual.*`), Gated DeltaNet `in_proj_a`/`in_proj_b`	BF16 (untouched)

The vision tower and the recurrence controls are kept at BF16 so image understanding and the hybrid-attention dynamics are unaffected. Round-trip errors on the int8 steps are 0.6–1.5%.

Refusal / quality (served build)

Measured on this quantized build, served under vLLM with an opening-phrase refusal classifier:

Benchmark	Refusal
AdvBench	0.0%
JailbreakBench (harmful)	0.0%
MaliciousInstruct	0.0%

The underlying abliteration scored 8 / 100 on Heretic's own (stricter) keyword scorer at KL divergence 0.0315 vs the base — see the BF16 card for method details. Vision verified working on the quantized build.

Quantization loss vs BF16

Measured against the full-precision BF16 build with the same harness hitting both (perplexity via vLLM prompt_logprobs, so it is a true like-for-like measurement, not two different loaders):

Metric	BF16	W4A16	Cost of int4
WikiText-2 perplexity	7.537	7.660	+1.6%
GSM8K (100 problems, matched basis)	95%	93%	−2 pts
Tool calling (12 cases, `qwen3_xml` parser) — valid / correct tool / correct args	11 / 11 / 11	10 / 10 / 10	−1 (noise at n=12)

The int4 quantization is close to free: language modeling barely moves, tool-calling is preserved, and the only real cost is a ~2-point dip on GSM8K — expected, since multi-step arithmetic is the most quantization-sensitive thing to measure. Vision is verified working on the quantized build.

Serving (vLLM)

vllm serve gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 \
  --served-model-name qwen3.8-27b \
  --max-model-len 32768 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image":1}'

Measured ~142 tok/s single-stream on an RTX 4090 with DFlash2 speculative decoding (mean acceptance length ~3.1). MTP speculative decoding also works out of the box.

License

Apache 2.0, inherited. Abliteration and quantization do not change the license.

Downloads last month: 125

Safetensors

Model size

6B params

Tensor type

I32

BF16

Model tree for gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16

Base model

Qwen/Qwen3.8-27B

Finetuned

DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1

Finetuned

gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic

Quantized

(4)

this model