wittjeff/unlimited-ocr-6bit-mlx

Unlimited-OCR — MLX conversion (6-bit, corrected tokenizer)

MLX conversion of baidu/Unlimited-OCR (3.34B MoE, SAM ViT-B + CLIP-L dual vision encoders, DeepSeek-V2 MoE decoder), converted with mlx-vlm 0.6.17 for Apple Silicon — including as a candidate MLX engine for the unlimited_ocr preset in docling (#3943).

Quantization: 6-bit affine on the language stack; vision towers (SAM + CLIP) kept in bf16. Effective 7.646 bits/weight overall.

Why another conversion

At the time of writing, 15 of the 18 MLX conversions of this model on the Hub ship a stripped tokenizer_config.json — and stock mlx_vlm.convert still produces one today. Upstream's is 165,938 bytes with 830 entries in added_tokens_decoder; the stripped version is ~545–567 bytes with zero.

Consequences:

• the grounding tokens (<|det|>, <|/det|>, <|ref|>, <|/ref|>, <|grounding|>) are no longer defined as special tokens, so they do not round-trip under skip_special_tokens=False — which breaks grounding consumers such as docling;
• the tokenizer falls back to a slow class without byte-level BPE decode, surfacing raw <0x..> byte markers in output.

Stock conversion also adds a chat_template.jinja that upstream deliberately does not have. It renders the prompt plus a trailing space, which matters for a model this prompt-sensitive: with no chat template present, mlx-vlm's fallback passes a single user message through verbatim, which is the behaviour you want.

This conversion restores the complete upstream tokenizer files (tokenizer.json, full tokenizer_config.json, special_tokens_map.json) and ships no chat template.

Verification

Converted and verified on Apple M4, 16 GB, macOS 26.6.2, with mlx-vlm 0.6.17 / mlx 0.32.2 / transformers 5.16.1:

Check	Result
tokenizer_config.json	165,938 bytes, 830 added tokens, loads as a fast tokenizer
Grounding tokens round-trip (skip_special_tokens=False)	5 / 5 exact
chat_template.jinja	absent (as upstream)
Quantization layout	quantized language stack only — vision towers unquantized
`<	det
`<	det
Raw <0x..> byte markers	0
Text similarity vs CUDA bf16 reference	1.000
Throughput	45.7 tok/s, peak memory 5.15 GB

The CUDA reference was produced with the upstream transformers implementation (transformers==4.57.1, infer(..., eval_mode=True)) on an RTX 3060.

Scope of that 1.000, stated plainly: the reference set is a single synthetic page (title, two paragraphs, a short list) at 200 dpi, grounding blocks stripped before diffing. It demonstrates the tokenizer and decode path are intact — it is not a document-understanding benchmark, and should not be read as one.

Usage

from mlx_vlm import load, generate

model, processor = load("wittjeff/unlimited-ocr-6bit-mlx", trust_remote_code=True)
out = generate(
    model, processor,
    "<image>document parsing.",   # keep the exact wording — the model is prompt-sensitive
    image=["page.png"],
    max_tokens=4096, temperature=0.0,
)
print(out.text)

Prompt notes:

• <image>document parsing. — structured parsing with grounding blocks. Recommended.
• <image>Free OCR. — plain-text mode. Community reports describe repetition loops under mlx-vlm (which lacks the reference no_repeat_ngram_size filter); we did not reproduce a loop on our sample page, but the filter is genuinely absent, so the risk is real on longer inputs.

Known differences vs the CUDA reference implementation

• mlx-vlm runs full attention in place of the model's reference sliding-window attention (R-SWA); equivalent on single pages, may differ on very long inputs.
• The reference no_repeat_ngram_size / ngram_window sampling filter is not implemented in mlx-vlm's generate loop.
• Gundam (cropped 640 px) dynamic-resolution mode is untested; verification used base mode.

License

MIT, inherited from upstream. Weights © Baidu, converted without modification beyond format and quantization.