jclyons52/Qwen3.8-27B-UD-Q3_K_XL-MLX-gptq-direct

Qwen3.8-27B UD-Q3_K_XL — MLX GPTQ-direct (native MLX, 3-bit dynamic)

Native-MLX port of unsloth's UD-Q3_K_XL dynamic quantization for Qwen/Qwen3.8-27B. Instead of converting GGUF weights, this build runs the quantizer search natively in MLX's affine format (group size 64) and writes final MLX weights directly — no GGUF round-trip, no mlx_lm.convert re-quantization step.

Perplexity (wikitext-2-raw first 32k tokens, 512-token windows, greedy NLL — identical math to llama-perplexity):

Build	PPL ↓
unsloth UD-Q3_K_XL GGUF in llama.cpp	6.896
this model (GPTQ-direct)	8.090
MLX imatrix-weighted (no GPTQ)	8.136
MLX plain RTN (same bit map)	8.169

Best published native-MLX Q3 for this model family to our knowledge. Full methodology and history: ud2mlx repo.

Method

1. Bit map taken from unsloth's Qwen3.8-27B-UD-Q3_K_XL.gguf (per-tensor bits, unsloth's dynamic layer selection).
2. Importance weighting from unsloth's published imatrix (per-column activation energies from their calibration set).
3. GPTQ with real activation Hessians captured on 128×512 = 65,536 calibration tokens (wikitext-2-train-raw) — full-rank for every input dim up to I=17408. Sequential error feedback through each block.
4. Per-group affine parameters are fixed before feedback (importance- weighted candidate search with margin headroom), so the emitted weights are exactly what the feedback pass produced — zero re-binning loss.
5. embed_tokens and lm_head have no block-input Hessian → RTN with the same affine search (uniform importance). lm_head at 5 bits.

14 GB, 498 quantized tensors, group size 64, avg ~4.4 bits on quantized modules (2-bit MLP-heavy early layers up to 8-bit norms/router).

Usage

pip install mlx-lm
mlx_lm.generate --model jclyons52/Qwen3.8-27B-UD-Q3_K_XL-MLX-gptq-direct \
    --prompt "Explain quantum tunneling" --max-tokens 256

Works with any MLX runtime that loads mlx-lm models (oMLX, etc.).

Limitations

• Affine vs IQ-codebook gap. MLX's quantized matmul only supports uniform affine grids; unsloth's ≤3-bit GGUFs use non-linear IQ codebooks (E8-lattice / super-block formats). At 3 bits this costs ~+1.2 PPL vs the GGUF; at 2 bits the gap widens to ~+1.95 (details). This is a runtime-kernel limitation of MLX, not a property of the weights.
• Embed/lm_head are RTN, not GPTQ (no activation Hessians available at those positions in the current capture harness).
• Text-only. The visual tower of the base checkpoint is not included.
• Eval scope: perplexity measured on wikitext-2 only, 32k tokens. Downstream task quality (coding, math, multilingual) is not yet characterized; low-bit dynamic quants typically degrade non-uniformly.
• Not for further training — quantized weights are frozen codes; use the fp16 base for fine-tuning.
• Quality can vary by prompt/domain; the late-window PPL drift visible in sequential evals suggests domain sensitivity typical of low-bit quants.

Calibration data

• GPTQ Hessians: wikitext-2-train-raw, first 65,536 tokens.
• Importance weighting: unsloth's published imatrix_unsloth.gguf (their calibration mixture).

Credits

• unsloth — dynamic bit maps + imatrix
• mlx-lm — MLX runtime
• Pipeline: jclyons52/ud2mlx

Built 2026-08-28. Base model license inherited from Qwen/Qwen3.8-27B.