Instructions to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WhiskyAKM/Qwen3.6-35B-A3B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WhiskyAKM/Qwen3.6-35B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Ollama
How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with Ollama:
```
ollama run hf.co/WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with Docker Model Runner:
```
docker model run hf.co/WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M
```

Lemonade

How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use WhiskyAKM/Qwen3.6-35B-A3B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "WhiskyAKM/Qwen3.6-35B-A3B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-35B-A3B - GGUF

GGUF quantizations of Qwen/Qwen3.6-35B-A3B, a Mixture-of-Experts multimodal model from the Qwen3.6 series with 35B total parameters and only 3B activated per token.

This repository contains GGUF conversions of the model, making it usable with llama.cpp and other GGUF-compatible inference engines.

Model Overview

Qwen3.6-35B-A3B is a multimodal model built by the Qwen Team that handles text, image, and video inputs and generates text output. It uses a hybrid linear attention + full attention architecture with Mixture-of-Experts, achieving strong agentic coding and reasoning performance while keeping the activated parameter count low for efficient inference.

Property	Value
Architecture	Qwen3_5MoeForConditionalGeneration
Parameters	35B total / 3B activated
Layers	40
Hidden Dimension	2048
Attention Layout	10 × (3 × Gated DeltaNet → 1 × Gated Attention)
Attention Heads	16 (Q), 2 (KV), head dim 256
Linear Attention Heads	32 (V), 16 (QK), head dim 128
Mixture of Experts	256 experts, 8 routed + 1 shared
Expert Intermediate Dim	512
Context Length	262,144 tokens (extensible to 1,010,000 via YaRN)
Vocabulary Size	248,320
Supported Modalities	Text, Image, Video
RoPE	Multimodal RoPE (interleaved), θ = 10,000,000
Multi-Token Prediction	Trained with MTP

GGUF Files

File	Format	Size	Description
`qwen3.6-35b-a3b-Q4_0.gguf`	Q4_0	19G	4-bit, fastest inference
`qwen3.6-35b-a3b-Q4_K_M.gguf`	Q4_K_M	21G	K-quant, medium
`qwen3.6-35b-a3b-Q4_K_S.gguf`	Q4_K_S	19G	K-quant, small
`qwen3.6-35b-a3b-Q5_K_M.gguf`	Q5_K_M	24G	K-quant, medium
`qwen3.6-35b-a3b-Q5_K_S.gguf`	Q5_K_S	23G	K-quant, small
`qwen3.6-35b-a3b-Q6_K.gguf`	Q6_K	28G	K-quant, higher precision
`qwen3.6-35b-a3b-Q8_0.gguf`	Q8_0	36G	8-bit, highest GGUF precision
`qwen3.6-35b-a3b-bf16.gguf`	bf16	67G	Full bfloat16 (unquantized)
`mmproj.gguf`	-	861M	Multimodal projector (vision)

A chat_template.jinja file is also provided for use with chat-based inference.

Note on mmproj: The mmproj.gguf file contains the vision projector needed for multimodal (image/video) inference. It is shared across all quantization variants.

Usage

llama.cpp (CLI)

# Run text-only inference
./llama-cli \
  -m qwen3.6-35b-a3b-Q4_K_M.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 1.0 --top-k 20 --top-p 0.95

llama-server (OpenAI-compatible API)

# Text-only
./llama-server \
  -m qwen3.6-35b-a3b-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080

# Multimodal (image + video)
./llama-server \
  -m qwen3.6-35b-a3b-Q4_K_M.gguf \
  --mmproj mmproj.gguf \
  --host 0.0.0.0 --port 8080

Multimodal (Image / Video)

For image and video inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing image/video data alongside text prompts.

Modality order tip: For best results, place image/video content before text in your prompt.

Generation Parameters

Recommended parameters from the model's generation_config.json:

Parameter	Value
Temperature	1.0
Top-K	20
Top-P	0.95
BOS Token ID	248044
EOS Token ID	248044, 248046
Pad Token ID	248044

Recommended Sampling by Mode

Mode	Temperature	Top-P	Top-K	Presence Penalty	Repetition Penalty
Thinking (general)	1.0	0.95	20	1.5	1.0
Thinking (coding/WebDev)	0.6	0.95	20	0.0	1.0
Instruct (non-thinking)	0.7	0.80	20	1.5	1.0

Thinking Mode

Qwen3.6 operates in thinking mode by default, generating reasoning content signified by <think>\n...\n</think>\n\n before producing the final response.

Enable (default): The model automatically generates thinking content before its final answer.
Disable: Set enable_thinking to false in the chat template kwargs (or equivalent in your inference engine) to obtain direct responses without thinking.

Qwen3.6 does not support the /think and /no_think soft switches from Qwen3. Thinking is controlled via API/chat template parameters.

Preserve Thinking

Qwen3.6 can optionally retain and leverage thinking traces from historical messages. Enable preserve_thinking in the chat template kwargs to keep full reasoning context across turns - particularly beneficial for agentic scenarios where it can improve decision consistency and reduce overall token consumption.

Key Features

Mixture of Experts: 35B total parameters with only 3B activated per token for efficient inference
Hybrid Attention: Gated DeltaNet (linear attention) with full attention every 4th layer
Multimodal: Text, image, and video understanding
Long Context: 262,144 token context window, extensible to 1,010,000 tokens via YaRN
Agentic Coding: Strong performance on SWE-bench, Terminal-Bench, and repository-level reasoning
Function Calling: Native support for structured tool use via <function=...> format
Multi-Token Prediction: Trained with MTP for speculative decoding acceleration
Multilingual: Support for 140+ languages

Processing Ultra-Long Texts

For contexts exceeding 262,144 tokens, YaRN RoPE scaling is recommended. The following RoPE parameters can be used to extend context up to ~1M tokens:

{
    "mrope_interleaved": true,
    "mrope_section": [11, 11, 10],
    "rope_type": "yarn",
    "rope_theta": 10000000,
    "partial_rotary_factor": 0.25,
    "factor": 4.0,
    "original_max_position_embeddings": 262144
}

Note: Static YaRN applies a constant scaling factor regardless of input length, which may impact performance on shorter texts. Only enable YaRN when long context is required, and adjust factor as needed (e.g., factor: 2.0 for ~524K context).

Acknowledgements

Original model: Qwen/Qwen3.6-35B-A3B
Blog post: Qwen3.6-35B-A3B

Citation

@misc{qwen36_35b_a3b,
    title = {{Qwen3.6-35B-A3B}: Agentic Coding Power, Now Open to All},
    url = {https://qwen.ai/blog?id=qwen3.6-35b-a3b},
    author = {{Qwen Team}},
    month = {April},
    year = {2026}
}