Qwen3.6-35B-A3B-oQ6e-fp16-mtp

An oQ mixed-precision MLX quantization of Qwen/Qwen3.6-35B-A3B, built for Apple Silicon.

Fastest of the three on this hardware, and the smallest on disk of the two 35B variants.

Quantization

Mixed precision, 6-bit affine base at group size 64, with 340 per-module overrides: 187 modules at 8-bit/g64, 123 at 8-bit/g128, 30 at 6-bit/g128. Embeddings are kept at 8 bits.

Bit-width assignment is imatrix-guided (oqe_code_multilingual, 128 samples x 512 tokens) — 481 of 523 collected entries applied. Layers left unquantized are kept at fp16, which is what the fp16 in the repo name refers to. The mtp suffix means the checkpoint retains its multi-token-prediction head (1 MTP layer, shared embeddings), so MTP speculative decoding works without a separate draft model.

• Format: MLX safetensors, 31.2 GB across 2,052 tensors
• Mode: affine
• Calibration report: oq_imatrix_report.json is included in this repo

Architecture

Base architecture	Qwen3_5MoeForConditionalGeneration
Parameters	35 B total
Active per token	~3 B active per token (8 of 256 experts)
Layers	40 (10 full-attention, 30 linear-attention)
Attention heads	16 query / 2 key-value, head dim 256
Experts	256 experts, 8 routed per token
Hidden size	2048
Vocabulary	248,320
Max position embeddings	262,144

This is a hybrid-attention model: only the 10 full-attention layers grow a KV cache with sequence length. The remaining 30 linear-attention layers hold a fixed-size recurrent state.

Measured performance

Single-stream, measured on an Apple M1 Max / 64 GB running oMLX. 300-token generations (median of 3) and a 7.5k-token uncached prefill.

Metric	Value
Decode	55.1 tok/s
Prefill	909 tok/s
Cold load	15.5 s

These are throughput figures on one machine, not a quality benchmark, and they will vary with hardware, context length and sampler settings. The first request after a model load reads low — that is warm-up, not steady state.

Memory planning

At fp16 the KV cache costs roughly 21 KB per token (10 full-attention layers x 2 KV heads x 256 head dim x 2 for K/V x 2 bytes).

Context	KV cache at fp16
32,768	0.66 GiB
65,536	1.31 GiB
131,072	2.62 GiB

Budget for weights plus KV cache plus runtime overhead. If the cache does not fit, quantizing it (for example TurboQuant 8-bit) roughly halves these figures at no measurable throughput cost.

Usage

Served through oMLX over an OpenAI-compatible API:

curl http://127.0.0.1:8055/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen3.6-35B-A3B-oQ6e-fp16-mtp", "messages": [{"role": "user", "content": "Hello"}]}'

Sampling follows Qwen guidance: thinking mode temperature 0.6, top_p 0.95; non-thinking temperature 0.7, top_p 0.8; top_k 20, min_p 0 throughout. Avoid greedy decoding with thinking enabled — Qwen documents that the combination can produce endless repetition.

Verified with oMLX. Loading outside oMLX (mlx-lm / mlx-vlm) requires a runtime that honours per-module quantization overrides in config.json; this has not been tested here.

License

Apache 2.0, inherited from Qwen/Qwen3.6-35B-A3B. A copy of the license is included as LICENSE. Quantization does not change the licensing terms — all credit for the underlying model belongs to the Qwen team.