YoozLabs/Qwen3.5-4B-qat-lean-4bit-mlx

Qwen3.5-4B (Yooz QAT, uniform 4-bit)

What this is

The first grid-aligned quantization-aware-trained (QAT) build of Qwen3.5-4B for MLX. The 4-bit Qwen3.5 builds available today are post-training quantized (PTQ), and at this model size PTQ onto the MLX 4-bit grid breaks the model's aggregation abilities outright: on variable-tracking, the community 4-bit build scores 0.00 where this build scores 0.97 (same grid, same harness, n=30 -- full table below). NIAH-style retrieval looks fine on both, which is exactly why this damage goes unnoticed: simple retrieval does not exercise it.

This build is quantization-aware trained by Yooz Labs: the model is trained to tolerate exactly the MLX affine 4-bit group-64 grid via knowledge distillation from the full-precision bf16 model, so the quantized weights reproduce the original model's behavior -- aggregation AND multi-turn recall -- instead of trading one for the other, at both 8K and 32K context. It is a general-purpose build, not a task specialist.

It is also text-only and LEAN: the base checkpoint is natively multimodal (it ships a vision tower for image/video input), and the community 4-bit conversion carries that tower along unquantized in BF16 even though a text-only long-context workload never touches it. This build drops it -- every remaining tensor is 4-bit group-64, verifiable from config.json (no vision weights, no BF16 stragglers) -- which is the actual source of the size delta below, not a different quantization grid.

Provenance

• Base checkpoint: Qwen/Qwen3.5-4B
• QAT training: Yooz Labs, 2026-07 -- knowledge distillation from the bf16 base model under fake-quantization matching the target grid (4-bit, group size 64, affine). Training recipes are not published.
• Conversion tool: mlx-lm 0.31.3 (mlx_lm.convert, 4-bit group-64, uniform -- no per-tensor overrides)
• Harness: benchmarks/ (infinite repo, Yooz Labs), all numbers n=30 per cell on the machine noted below.

Size

Build	Method	Size (GB)
This build	QAT, text-only (uniform 4-bit g64)	2.37
Community MLX 4-bit	PTQ (4-bit g64 + unquantized BF16 vision tower)	3.03
Community MLX 8-bit	PTQ (the working fallback until now)	5.14

Sizes are on-disk decimal GB (safetensors shard bytes). Until this build, getting working aggregation from Qwen3.5-4B on MLX meant the 8-bit build at 5.14 GB; this build delivers it at 2.37 GB.

Runtime memory (measured, not estimated)

Metric	Community 4-bit (as shipped)	This build
Peak memory footprint, 8K context (GB)	9.1	9.3
Peak memory footprint, 32K context (GB)	21.5	21.5
Peak RSS, 32K context (GB)	2.99	2.94
Decode speed, 32K context (tokens/s)	72.7	72.5
Prefill speed, 32K context (tokens/s)	759	759

Measured by the benchmark harness during real retrieval tasks on Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16. Decode/prefill speed is near-identical between builds, as expected: QAT changes weight values, not the compute graph.

Validation

Long-context A/B against qwen3_5-4b-stock-text (verified identical quantization: affine 4-bit group-64, zero per-tensor overrides -- the community 4-bit build for this size uses a clean chat template, so its numbers match this comparator directly) on the same stack: needle-in-a-haystack (NIAH) and RULER-style tasks scored per-word, MRCR scored by sequence ratio. n=30 per cell.

Task	Context	4-bit PTQ, same grid	Yooz QAT 4-bit
NIAH single	8K	1.00	1.00
NIAH multikey	8K	1.00	0.97
Variable tracking	8K	0.00	0.97
Frequent words	8K	0.12	0.83
MRCR 2-needle	8K	0.97	0.97
NIAH single	32K	0.97	1.00
NIAH multikey	32K	1.00	0.97
Variable tracking	32K	0.89	0.99
Frequent words	32K	0.22	0.77
MRCR 2-needle	32K	0.96	0.96

Clean sweep at both context lengths: no task regresses relative to the same-grid PTQ baseline, aggregation is rescued at both bins, and multi-turn recall (MRCR) holds at parity with stock throughout -- this build actually posts its best variable-tracking score at 32K (0.99).

Usage

from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/Qwen3.5-4B-qat-lean-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List the first five prime numbers."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True))

A GGUF (Q4_0) build of the same QAT weights is available at YoozLabs/Qwen3.5-4B-qat-GGUF for llama.cpp.

About Yooz Labs

Maintained by Yooz Labs -- sovereign, on-device AI for macOS. Everything we ship runs locally; nothing goes to a cloud. This build comes out of our long-context research line, where we found and quantified the PTQ aggregation crater -- and then trained it away.

Questions, issues, or corrections: dev@yooz.info

License and attribution

Apache 2.0, inherited from the base checkpoint Qwen/Qwen3.5-4B. Qwen is a model family by Alibaba Cloud; this is an independent, unofficial QAT build, not distributed, endorsed, or certified by Alibaba.