Instructions to use mconcat/Qwopus3.6-27B-v2-AWQ-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mconcat/Qwopus3.6-27B-v2-AWQ-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mconcat/Qwopus3.6-27B-v2-AWQ-4bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("mconcat/Qwopus3.6-27B-v2-AWQ-4bit")
model = AutoModelForMultimodalLM.from_pretrained("mconcat/Qwopus3.6-27B-v2-AWQ-4bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mconcat/Qwopus3.6-27B-v2-AWQ-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mconcat/Qwopus3.6-27B-v2-AWQ-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mconcat/Qwopus3.6-27B-v2-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mconcat/Qwopus3.6-27B-v2-AWQ-4bit

SGLang

How to use mconcat/Qwopus3.6-27B-v2-AWQ-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mconcat/Qwopus3.6-27B-v2-AWQ-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mconcat/Qwopus3.6-27B-v2-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mconcat/Qwopus3.6-27B-v2-AWQ-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mconcat/Qwopus3.6-27B-v2-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mconcat/Qwopus3.6-27B-v2-AWQ-4bit with Docker Model Runner:
```
docker model run hf.co/mconcat/Qwopus3.6-27B-v2-AWQ-4bit
```

Qwopus3.6-27B-v2-AWQ-4bit / README.md

mconcat

Fix: AWQ MTP 0% acceptance by adding mtp.* to modules_to_not_convert

baabcbb verified 6 months ago

preview code

Raw History Blame Contribute Delete

4.16 kB

---
language:
- en
- zh
- ko
- ja
license: apache-2.0
base_model: Jackrong/Qwopus3.6-27B-v2
tags:
- qwen3.5
- qwen3.6
- qwopus
- reasoning
- quantized
- awq
- int4
- multi-token-prediction
pipeline_tag: text-generation
library_name: transformers
quantized_by: mconcat
---
# Qwopus3.6-27B-v2-AWQ-4bit

AutoAWQ-format INT4 (W4A16) quantization of [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2), a Claude Opus reasoning-distilled fine-tune of Qwen 3.6 27B.

The hybrid DeltaNet + softmax attention architecture is preserved, the 1-layer MTP head is included for speculative decoding, and the multimodal processor metadata is kept intact. APEX-style edge protection keeps the first and last layers in BF16 for quality.

## Quick start

Requires **vLLM ≥ 0.21.0**:

```bash
vllm serve mconcat/Qwopus3.6-27B-v2-AWQ-4bit \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --trust-remote-code
```

## Benchmarks

Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on a single NVIDIA B300 SXM6, 100 samples per task, 0-shot CoT, `max_gen_toks=4096`:

| Task | Qwen 3.6 27B (base) | Qwopus 3.6 v2 (source BF16) | **This (AWQ-4bit)** |
|---|---:|---:|---:|
| GSM8K (flexible-extract) | 65.0% | 87.0% | **85.0%** |
| ARC-Challenge (acc_norm) | 46.0% | 45.0% | **47.0%** |
| TruthfulQA-MC2 | 55.1% | 59.3% | **59.3%** |
| IFEval (inst_level_strict) | 40.5% | 42.3% | **42.9%** |
Quantization preserves accuracy within standard error of the BF16 source on every task, and matches the source on TruthfulQA. The Claude Opus reasoning gain over the Qwen 3.6 base (+20 pp on GSM8K) is retained.
## Throughput
Measured on a single NVIDIA B300 SXM6 with vLLM 0.21.0 and `torch.compile` enabled:
| Setup | Throughput | Speedup |
|---|---:|---:|
| Batch = 1, no MTP | 115 tok/s | 1.00× |
| Batch = 1, MTP `num_speculative_tokens = 3` | **251 tok/s** | **2.19×** |
| Batch = 8 continuous batching, no MTP | 880 tok/s | — |
MTP speculative decoding hits an **Avg Draft acceptance rate of ~77 %** (per-position: 0.92 / 0.79 / 0.65) with a mean acceptance length of ~3.3, measured on a mixed reasoning + code prompt set at greedy decoding.
Self-test of tool calling with `--tool-call-parser qwen3_coder`: passes (model emits well-formed `<tool_call>...</tool_call>` syntax).

## Quantization

| Precision | Modules |
|---|---|
| **INT4 asymmetric, group_size = 128** | `q_proj`, `k_proj`, `v_proj`, MLP `gate_proj`, MLP `up_proj`, DeltaNet `in_proj_qkv`, `in_proj_z` |
| **BF16** | `o_proj`, MLP `down_proj`, `lm_head`, `embed_tokens`, norms, DeltaNet small projections (`in_proj_a`, `in_proj_b`), DeltaNet `out_proj`, vision tower, multimodal projector, 1-layer MTP head, **first 5 and last 5 transformer layers (APEX edge protection)** |

The AWQ skip list also names every `mtp.*` linear module explicitly so the MTP draft head stays unquantized — previous revisions of this checkpoint omitted those entries, which caused vLLM to build the MTP head with AWQ-packed parameters and produced 0 % draft acceptance.

Tuned with [AutoRound](https://github.com/intel/auto-round) on 1024 self-generated reasoning traces (200 iterations per block, batch_size=1).
Calibration data: 1024 self-generated traces from the BF16 source model (256 prompts × 4 generations) covering math, code, logic, analysis, creative writing, general knowledge, tool calling, and Korean.
## Files
| File | Size | Purpose |
|---|---:|---|
| `model-*.safetensors` (13 shards) | ~25 GB | Main quantized weights |
| `model_extra_tensors.safetensors` | ~1 GB | MTP head + edge-protected layers (BF16) |
| `quantization_config.json` | <1 KB | AWQ config (`quant_method=awq`, `bits=4`, `group_size=128`, `zero_point=true`) with BF16 MTP skip entries |
| `config.json` + tokenizer + processor configs | <100 MB | Standard metadata |
Total checkpoint size: **~26 GB** (down from ~54 GB BF16 source).
## License
Apache 2.0 (inherited from the base model).

	---
	language:
	- en
	- zh
	- ko
	- ja
	license: apache-2.0
	base_model: Jackrong/Qwopus3.6-27B-v2
	tags:
	- qwen3.5
	- qwen3.6
	- qwopus
	- reasoning
	- quantized
	- awq
	- int4
	- multi-token-prediction
	pipeline_tag: text-generation
	library_name: transformers
	quantized_by: mconcat
	---

	# Qwopus3.6-27B-v2-AWQ-4bit

	AutoAWQ-format INT4 (W4A16) quantization of [Jackrong/Qwopus3.6-27B-v2](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2), a Claude Opus reasoning-distilled fine-tune of Qwen 3.6 27B.

	The hybrid DeltaNet + softmax attention architecture is preserved, the 1-layer MTP head is included for speculative decoding, and the multimodal processor metadata is kept intact. APEX-style edge protection keeps the first and last layers in BF16 for quality.

	## Quick start

	Requires vLLM ≥ 0.21.0:

	```bash
	vllm serve mconcat/Qwopus3.6-27B-v2-AWQ-4bit \
	--tensor-parallel-size 1 \
	--max-model-len 16384 \
	--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
	--tool-call-parser qwen3_coder \
	--reasoning-parser qwen3 \
	--enable-auto-tool-choice \
	--trust-remote-code
	```

	## Benchmarks

	Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on a single NVIDIA B300 SXM6, 100 samples per task, 0-shot CoT, `max_gen_toks=4096`:

	\| Task \| Qwen 3.6 27B (base) \| Qwopus 3.6 v2 (source BF16) \| This (AWQ-4bit) \|
	\|---\|---:\|---:\|---:\|
	\| GSM8K (flexible-extract) \| 65.0% \| 87.0% \| 85.0% \|
	\| ARC-Challenge (acc_norm) \| 46.0% \| 45.0% \| 47.0% \|
	\| TruthfulQA-MC2 \| 55.1% \| 59.3% \| 59.3% \|
	\| IFEval (inst_level_strict) \| 40.5% \| 42.3% \| 42.9% \|

	Quantization preserves accuracy within standard error of the BF16 source on every task, and matches the source on TruthfulQA. The Claude Opus reasoning gain over the Qwen 3.6 base (+20 pp on GSM8K) is retained.

	## Throughput

	Measured on a single NVIDIA B300 SXM6 with vLLM 0.21.0 and `torch.compile` enabled:

	\| Setup \| Throughput \| Speedup \|
	\|---\|---:\|---:\|
	\| Batch = 1, no MTP \| 115 tok/s \| 1.00× \|
	\| Batch = 1, MTP `num_speculative_tokens = 3` \| 251 tok/s \| 2.19× \|
	\| Batch = 8 continuous batching, no MTP \| 880 tok/s \| — \|

	MTP speculative decoding hits an Avg Draft acceptance rate of ~77 % (per-position: 0.92 / 0.79 / 0.65) with a mean acceptance length of ~3.3, measured on a mixed reasoning + code prompt set at greedy decoding.

	Self-test of tool calling with `--tool-call-parser qwen3_coder`: passes (model emits well-formed `<tool_call>...</tool_call>` syntax).

	## Quantization

	\| Precision \| Modules \|
	\|---\|---\|
	\| INT4 asymmetric, group_size = 128 \| `q_proj`, `k_proj`, `v_proj`, MLP `gate_proj`, MLP `up_proj`, DeltaNet `in_proj_qkv`, `in_proj_z` \|
	\| BF16 \| `o_proj`, MLP `down_proj`, `lm_head`, `embed_tokens`, norms, DeltaNet small projections (`in_proj_a`, `in_proj_b`), DeltaNet `out_proj`, vision tower, multimodal projector, 1-layer MTP head, first 5 and last 5 transformer layers (APEX edge protection) \|

	The AWQ skip list also names every `mtp.*` linear module explicitly so the MTP draft head stays unquantized — previous revisions of this checkpoint omitted those entries, which caused vLLM to build the MTP head with AWQ-packed parameters and produced 0 % draft acceptance.

	Tuned with [AutoRound](https://github.com/intel/auto-round) on 1024 self-generated reasoning traces (200 iterations per block, batch_size=1).

	Calibration data: 1024 self-generated traces from the BF16 source model (256 prompts × 4 generations) covering math, code, logic, analysis, creative writing, general knowledge, tool calling, and Korean.

	## Files

	\| File \| Size \| Purpose \|
	\|---\|---:\|---\|
	\| `model-*.safetensors` (13 shards) \| ~25 GB \| Main quantized weights \|
	\| `model_extra_tensors.safetensors` \| ~1 GB \| MTP head + edge-protected layers (BF16) \|
	\| `quantization_config.json` \| <1 KB \| AWQ config (`quant_method=awq`, `bits=4`, `group_size=128`, `zero_point=true`) with BF16 MTP skip entries \|
	\| `config.json` + tokenizer + processor configs \| <100 MB \| Standard metadata \|

	Total checkpoint size: ~26 GB (down from ~54 GB BF16 source).

	## License

	Apache 2.0 (inherited from the base model).