czl/CLM-v0.1-8B-MLX-4bit-outq2

CLM-v0.1-8B — MLX encoder (4bit, lm_head at 2-bit)

The same quant as czl/CLM-v0.1-8B-MLX-4bit, with one difference: lm_head is stored at 2-bit instead of the bulk width. Every other tensor is byte-identical to that repo. This is the MLX counterpart of the GGUF *-outq2.gguf line — see the section below for why that is safe and what it costs you.

What is in this repo

path	what
config.json, model*.safetensors, tokenizer*	the 4bit quantised encoder, mlx-community layout, weights at the repo root
clm_mlx/	the pooling pass and an OpenAI-compatible /v1/embeddings server
heads/	CLM_v0.1-8B.pt as safetensors, so no PyTorch is needed
clm_mlx.json	the pooling / embedding / scale contract
parity.json, quantization.json	measured agreement and the exact recipe

How to use

mlx_lm.load("czl/CLM-v0.1-8B-MLX-4bit-outq2")

# or serve it for CLM
python -m clm_mlx.server --model <dir> --port 8092
clm-serve --emb-url http://127.0.0.1:8092/v1/embeddings

Why the 2-bit head is free

lm_head is the output projection — 622,329,856 parameters, 7.6 % of the model. Under last-token pooling it is never evaluated: the pooling pass reads the last hidden state and never a logit, so the output projection is not in the graph at all. clm_mlx.Encoder replaces the head with an nn.Identity before the graph is realised.

This is measured, not assumed. Re-quantising the head from the bulk width down to 2-bit (a precision cut on 622,329,856 parameters) leaves the pooled embeddings effectively unchanged: No quality change, and not merely asserted. Across the full corpus (4,448 pooled vectors, 4096 dimensions, the same server path as the published numbers), swapping the head from 4-bit to 2-bit moves each vector by at most 6.09e-03 (cosine min 0.999753, 2 of 4,448 rows touched at all). That is exactly what re-running the unchanged model gives (null: 6.09e-03, cosine min 0.999753, 2 rows) — the two are indistinguishable. MLX is not run-to-run deterministic on the GPU (non-deterministic atomics in its reductions, ml-explore/mlx#878), and the residual lands entirely on the longest documents. The head is not in the forward pass: clm_mlx.Encoder replaces it with nn.Identity before the graph is realised. Were a 2-bit projection actually being evaluated, the embeddings would be wrecked (cosine O(1)), not moved by 1e-3.

So the size reduction below costs no measurable quality. There is no accuracy trade-off being made. The parent repo separately proved the head is dead by zeroing every lm_head tensor, and the GGUF line proved it independently with a wrecked q2_K head.

	bytes	size
czl/CLM-v0.1-8B-MLX-4bit	5,131,300,029	5.13 GB
this repo (4bit-outq2)	5,053,495,687	5.05 GB
saved	77,804,342	78 MB

⚠️ For pooling only — not for generation. A 2-bit lm_head is a deliberately wrecked output projection. It is never evaluated under pooling, so the embeddings and every quality number here are unaffected — but if you load this as a causal LM for a completion or a chat, it will emit nonsense logits. Use the non--outq2 repo for that, at 78 MB more. This is a deliberate size optimisation for the ranker use case, not a general-purpose quant.

Quality is the same as the parent quant

Because the head is not in the pooling forward pass, this repo inherits the parent's evaluation — same corpus, same head stack, same verdicts, and embeddings that are indistinguishable from re-running the parent. See czl/CLM-v0.1-8B-MLX-4bit for the full table: 23,926 scored questions, argmax(scale · cos) with scale = 100.0.

Also available: 6bit-outq2, 8bit-outq2.

Limitations

• This is a ranker, not a generator. Same as the parent: lm_head is never evaluated under pooling. The 2-bit head only makes that fact louder.
• The projection heads are encoder-locked to Qwen3-8B last-token-pooled 4096-d embeddings.
• The head stack amplifies cosine error 100×. See the parent repo's evaluation section.
• Probabilities are relative to the candidate set.

License

Apache-2.0, inherited from Qwen/Qwen3-8B and Contrastive-LM/CLM-v0.1-8B. Not gated.

Citation

@misc{kwok2026contrastivelanguagemodels,
  title={Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making},
  author={Jacky Kwok and Hangoo Kang and Tarun Suresh and Jon Saad-Falcon and Marco Pavone and Christopher Ré and Azalia Mirhoseini},
  year={2026},
  note={Notion Blog},
  url={https://contrastive-lm.notion.site}
}