Instructions to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF
# Run inference directly in the terminal:
llama cli -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF
# Run inference directly in the terminal:
llama cli -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF
# Run inference directly in the terminal:
./llama-cli -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Use Docker

docker model run hf.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

LM Studio
Jan

vLLM

How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Ollama
How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with Ollama:
```
ollama run hf.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF
```
Unsloth Desktop

How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with Docker Model Runner:
```
docker model run hf.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF
```

Lemonade

How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-Escha-W2-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Escha Qwen3.6-35B-A3B W2 GGUF

I ported EschaLabs/Qwen3.6-35B-A3B-Escha-W2 to llama.cpp. Weights are theirs. This is not a requant.

I decode their native 2-bit eschamoe code in-kernel (GGML_OP_ESCHA_MOE). Expert payloads are the same bytes as the safetensors. Non-expert tensors are fp16.

Stock llama.cpp cannot load this file. You need my fork, branch escha-w2:

https://github.com/Ajay9o9/llama.cpp-escha/tree/escha-w2

If you only download the GGUF, it will not run.

File

	Source checkpoint	This GGUF
Size	12.30 GB	14.50 GB (+17.9%)
Expert code	native 2-bit / 3-bit	same bytes
Non-expert tensors	int8	fp16
Layers	40 + MTP	40, MTP dropped
Tensors		976
VRAM, 32k context		14.5 GB of 24

File: Escha-Qwen3.6-35B-W2.gguf. 256 experts, top-8. gate/up K=2, down K=3. Extra size is fp16 on the non-expert tensors, not the expert code.

Quality vs Escha SGLang

Same GPU. I sent token ids, not text. Tokenizers agreed on 196,608 tokens of wikitext. Perplexity uses the same chunking on both sides (n_ctx 2048, 96 chunks, 98,208 scored tokens).

Perplexity, wikitext-2
Escha SGLang	6.0988
This GGUF	6.0997 +/- 0.047
Difference	+0.0009 (+0.015%)

Gap is inside the llama.cpp run's own error bar, and smaller than SGLang's own rerun drift (+0.0016). One corpus. Not a task-accuracy claim.

Teacher-forced, 97 positions
Top-1 token agreement	93 / 97 = 95.9%
JS divergence over top-20	mean 1.7e-4, max 1.0e-3
\|delta logprob\| where they agree	mean 0.013, max 0.072
True-token NLL	SGLang 1.39704, this GGUF 1.38663

Same prefix at every position, so a miss at i-1 does not poison i. Two SGLang runs against each other were 96 / 97.

Greedy, 16 prompts, 64 tokens, temp 0
Token-identical	13 / 16
Diverged	3 / 16, at tokens 5, 26, 49

Two SGLang runs were also 13 / 16 identical. Greedy is a hard argmax. I would not hang a quality claim on this.

Op test vs a numpy dense-fold reference (CPU and CUDA scored separately): rel RMS 1.06e-6 at K=2, 9.26e-7 at K=3.

Speed vs Escha SGLang

Single stream, batch 1, one RTX 3090. Their runtime is still ~2.6x faster at generation. Context lengths were not matched, so the ratio is rough.

	Escha SGLang	This GGUF
Generation (tok/s)	110.4	42.6
Prefill, 4,566 tokens (tok/s)	2595	634

SGLang generation was 256 new tokens with ignore_eos. MTP is dropped here, so speculative decoding on their server is not in these numbers.

Build the fork

git clone -b escha-w2 https://github.com/Ajay9o9/llama.cpp-escha.git
cd llama.cpp-escha
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build -j 12

86 is an RTX 3090. Change that for your GPU. -j 12 is a 12-core CPU.

Download the GGUF

hf download aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF Escha-Qwen3.6-35B-W2.gguf

Direct file: https://huggingface.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF/resolve/main/Escha-Qwen3.6-35B-W2.gguf

Run it

./build/bin/llama-server \
  -m Escha-Qwen3.6-35B-W2.gguf \
  -ngl 99 --host 127.0.0.1 --port 8080 \
  -c 32768 -np 1 -t 12 --jinja

Or:

./build/bin/llama-cli \
  -m Escha-Qwen3.6-35B-W2.gguf \
  -ngl 99 --jinja -c 32768

It is a reasoning model. Give it room to think or you get an empty answer.

What this is not

Not a Q4_K / Q8_0 requant of a dense reconstruction
Not upstream llama.cpp
Not bit-identical to Escha SGLang on every greedy prompt
No MMLU / GSM8K / long-context eval

License

Apache-2.0, same as the Escha weights and the escha 1.0.2+qwen3moe wheel I used.

Source: EschaLabs/Qwen3.6-35B-A3B-Escha-W2
Runtime I compared against: escha 1.0.2+qwen3moe on SGLang

Downloads last month: 1,525

GGUF

Model size

7B params

Architecture

qwen35moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

EschaLabs/Qwen3.6-35B-A3B-Escha-W2

Quantized

(7)

this model

Collection including aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

Eschalab ggufs

Collection

My current and upcoming escha models converted ggufs for llama.cpp will be here. • 2 items • Updated 13 days ago