Instructions to use windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4

SGLang

How to use windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 with Docker Model Runner:
```
docker model run hf.co/windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4
```

Qwen3.8-27B-Abliterated — NVFP4A16

4-bit NVFP4 build of windowsxp811203/Qwen3.8-27B-Abliterated, an abliterated (refusal-removed) Qwen/Qwen3.8-27B.

55.6 GB → 28.6 GB, and the MTP draft head is intact and verified working at 76–78 % draft acceptance.

Correction (2026-08-23, numbers revised 08-24). This card previously said that most quantized derivatives of this architecture ship a dead MTP head. That was wrong, and I withdraw it. I surveyed every NVFP4 build of Qwen3.8-27B and its finetunes on the Hub (136 repos; 91 independent auditable artifacts after collapsing true re-uploads): 78 of 91 ship the complete 15-tensor head; 5 quantize it. I had generalized from my own first broken build without checking the population. The failure mode described below is real — I hit it, and about 22% of published artifacts show some form of head problem — but it is the exception, not the rule. Full method, revision history and raw data: nvfp4-mtp-survey.

Requires a Blackwell GPU (sm100+) and vLLM.

What is and isn't quantized

group	treatment	count
MLP `gate/up/down` (64 layers) + full-attention `q/k/v/o` (16 layers)	NVFP4 — 4-bit float, group size 16, `float8_e4m3` scales	256 Linears
`mtp.*` (draft head)	bf16, grafted back after quantization, in `ignore`	15 tensors
`model.visual.*` (vision tower)	bf16 — kept bit-identical (as in essentially every other NVFP4 build of this model)	167 weight tensors (333 incl. biases/norms)
`linear_attn.*` (Gated DeltaNet / SSM)	bf16, all of it. This is the one place these builds differ from most: 15 of 91 surveyed artifacts leave every projection in the path unquantized, 20 quantize all 240 projections, and the partial builds span 20–97 % coverage. (AMD's official Quark build of the 2.4T model excludes `linear_attn` outright too.)	336 weight tensors; 432 keys incl. `A_log`/`dt_bias`
`lm_head`, embeddings	bf16

The MTP tensors are both grafted back and listed in quantization_config.ignore. Both halves matter: without the graft there is no draft head at all, and without the ignore entry vLLM's compressed-tensors loader treats the bf16 head as a quantization target, finds no scales, and rejects every draft — 0 % acceptance while the logs look perfectly clean.

Verification

Measured on this exact checkpoint on an RTX PRO 6000 Blackwell.

MTP speculative decoding — {"method":"mtp","num_speculative_tokens":1}:

metric	value
Avg draft acceptance rate	76.4 % – 78.3 %
Mean acceptance length	1.76 – 1.78
Accepted / drafted	1615 / 2113 tokens

That number is the proof the graft worked; a broken MTP head reads 0 %.

Refusal — greedy, non-thinking, no prefill jailbreak. For scale: unmodified Qwen3.8-27B refuses 99.04 % of the full 520-prompt AdvBench (515/520) under these settings; this build was evaluated on an 80-prompt subset, so the two denominators differ:

benchmark	result
AdvBench (80-prompt subset)	0/80 · 0.00 %
HarmBench safety categories (119)	0/119 · 0.0 %
HarmBench copyright (41)	17/41 · 41.5 %

Safety categories = chemical/biological, cybercrime, harassment, harmful, illegal, misinformation — every one exactly zero. The copyright column is not a safety refusal and is mostly classifier false positives: the model delivers the lyrics or passage, but the text trips the keyword list (either the generated prose itself opens with "I cannot quite…", or a pedantic "I cannot generate a new passage … but here is a long excerpt" precedes the excerpt).

Capability — MMLU, 400 equidistant questions, identical prompting and parsing for both:

build	MMLU
GGUF Q8_0 (reference)	78.00 %
NVFP4A16 (this)	77.75 %

One question apart. (Do not compare these to the parent card's 82.35 %: that figure was measured by next-token logit comparison, a different and more forgiving method. Only same-method numbers are comparable.)

Usage

vllm serve windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 \
  --max-model-len 8192 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Thinking is on by default; disable per request with "chat_template_kwargs": {"enable_thinking": false}.

1M-token window (824K-token prompt verified end-to-end)

The model's declared native limit is 262,144. The 1M-window configuration from the official Qwen3.8-27B recipe loads and serves on this checkpoint; the longest prompt actually measured through it was 823,878 tokens:

vllm serve windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4 \
  --tensor-parallel-size 2 --max-model-len 1010000 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}'

Needle-in-a-haystack, passphrase buried at 50 % depth, greedy:

context	prompt tokens	retrieved	time
1M window / 824K prompt	823,878	✅	364 s on 2× RTX PRO 6000 Blackwell

Budget ≈ 61 GiB of KV at 1M (16 full-attention layers × 4 KV heads × 256 dim) on top of the 26.6 GiB of weights, so 1M needs two 96 GB cards; 256K fits comfortably on one.

If vLLM fails to start with a FlashInfer error

On hosts where the CUDA toolkit and FlashInfer's bundled CCCL headers disagree, FlashInfer's JIT fails to build its sampling kernels and vLLM aborts with FlashInfer requires GPUs with sm75 or higher (a misleading message — the real cause is that the capability probe itself failed). Working around it:

export VLLM_USE_FLASHINFER_SAMPLER=0
export VLLM_ATTENTION_BACKEND=TRITON_ATTN

This is a host toolchain issue, not a property of these weights.

Provenance

Quantized with llm-compressor 0.13.0 (NVFP4A16) from the bf16 parent, which was produced by orthogonalizing 131 residual-writing tensors (including embed_tokens) against a refusal direction at λ=1.5, leaving the vision tower byte-identical. Full recipe and evaluation in the parent model card. A llama.cpp build is at Qwen3.8-27B-Abliterated-GGUF.

Support / 打賞

If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。

USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888

Disclaimer

This model will not refuse. It is published for alignment and safety research. You are responsible for your use of it and for complying with applicable law. Inherits the Apache-2.0 license of the base model.

Downloads last month: 1,951

Safetensors

Model size

19B params

Tensor type

BF16

Model tree for windowsxp811203/Qwen3.8-27B-Abliterated-NVFP4

Base model

Qwen/Qwen3.8-27B

Finetuned

windowsxp811203/Qwen3.8-27B-Abliterated

Quantized

(4)

this model