infosave/Qwen3.6-35B-A3B-Escha-W2cmf

Qwen3.6-35B-A3B-Escha-W2 — CMF q2tp (2-bit experts)

12.87 GB, one file, mmap-run — and 105 tok/s decode on a single card.

A transcode of EschaLabs/Qwen3.6-35B-A3B-Escha-W2 into the CMF container so it runs on the cortiq engine: one binary, no Python, no CUDA toolkit — Vulkan, Metal or plain CPU from the same bytes.

Credit where it is due: the 2-bit weights are Escha Labs' work. Their eschamoe codec (2-bit routed experts with per-expert scale ladders, int8 dense layers) is what makes a 35B MoE fit in 12 GB at this quality. This repo re-expresses those weights in CMF's layouts; it does not re-quantize from the original bf16 and claims no quality improvement over theirs.

How it was made (the honest chain)

1. Every routed expert was reconstructed to bf16 through Escha's own kernel (PackedScaledExpertLinear, identity probe) rather than a reimplementation of their bitpack — their codec stays the source of truth. The int8 dense layers were dequantized with their stored scales.
2. That bf16 checkpoint went through cortiq convert --quant q2tp, a mixed profile: routed and shared expert gate/up at 2 bits (q2tp), expert down and the whole skeleton at 4 bits (q4tp), mirroring Escha's own 2/3-bit split one bit up.
3. q2tp is a new CMF layout added for this model: 4 levels (±0.5, ±1.5)·s — the RMS-optimal symmetric grid for Gaussian-ish weights — on q4tp's predicted per-row scale ladder, 8 bytes per 32-weight group. Rung 0 of the ladder is an exact zero, because the 4-level grid cannot otherwise spell one and a pruned group would come back as noise.

Architecture: 40 layers (10 full attention + 30 GatedDeltaNet linear), 256 routed experts top-8 plus a shared expert, hidden 2048, vocab 248 320. Escha's build carries no MTP head, so neither does this file.

Requires cortiq 0.5.43+

q2tp is a new layout: earlier runtimes cannot read it.

cargo install cortiq-cli
hf download infosave/Qwen3.6-35B-A3B-Escha-W2cmf qwen36-35b-a3b-escha-w2-q2tp.cmf --local-dir .   # pip install -U huggingface_hub

Run it

# Since cortiq 0.5.48: no flags. The GPU, its VRAM budget and the
# device/host split are detected and sized automatically.
cortiq run qwen36-35b-a3b-escha-w2-q2tp.cmf

# OpenAI-compatible server (Cline / Roo-style block-array content accepted)
cortiq serve qwen36-35b-a3b-escha-w2-q2tp.cmf

Any VRAM size (cortiq 0.5.48+, dynamic expert loading)

As many leading layers as fit stay on the card, the host finishes the rest — one boundary crossing per token, no flags. RTX PRO 6000, budget capped to what each card size would auto-detect (CPU reference 15.6 tok/s, 48 cores):

VRAM	4 GB	6 GB	8 GB	12 GB	16 GB+
tok/s	20.5	24.5	32.3	88.3	136.2 (full graph)
layers on GPU	8/40	16/40	23/40	37/40	all

(0.5.49's pipelined submission: the full card went 110.7 → 136.2 with bit-identical greedy output.)

Perplexity is identical to the printed digit on every rung, on the CPU and on the full graph (4.971 over the reference text).

Measured

	hardware	decode
Vulkan	RTX PRO 6000 Blackwell	136.2 tok/s (0.5.49)
Metal (wgpu graph)	Apple M4, 24 GB	13.3 tok/s (CMF_GPU=wgpu, graph on by default since 0.5.48+)
CPU	Apple M4, 8 threads	7.3 tok/s

Same engine, same release, for scale: the 4-bit Qwen3.6-35B-A3B runs 99.6 tok/s and the dense 27B 39.1. The decode path is greedy-parity gated against the CPU at short context; on long prompts greedy decoding may pick a different-but-coherent continuation at probability ties (float ordering, as with any GPU offload).

Prompt ingest currently runs per position on the CPU for this layout — the batched GEMM twin for q2tp is not written yet. Long prompts are slow; decode is not.

File

• qwen36-35b-a3b-escha-w2-q2tp.cmf — 12 866 141 998 bytes, size-verified against the converter output after upload.

Weights derive from Qwen's release via Escha Labs' 2-bit build and remain under Apache-2.0. The CMF container and the cortiq runtime are Apache-2.0 as well (see the repository's LICENSE and PATENTS.md).