Qwen3.8-27B-MoQ-GGUF

This repository contains imatrix-aware, layer-aware Mixture-of-Quantization (MoQ) GGUF files for the original Qwen/Qwen3.8-27B model. These files preserve the model's MTP tensors and are quantizations of the base Qwen model. In terms of overall quality, MoQ is comparable to Unsloth's Dynamic Quantization 3.0; however, our quantization series offers a more granular range of options (4.0 bpw – 4.75 bpw) than the Unsloth team's offerings, while maintaining a slight performance edge over Dynamic Quantization 3.0 within this range.

The complete series was evaluated locally alongside the Unsloth Dynamic GGUF series under identical conditions. Lower is better in all three charts.

The interactive comparison report supports pan, zoom, view reset, and per-series visibility controls. The companion CSV contains all measured values and tensor-composition summaries.

Comparison Summary

All 17 measured GGUF files contain 866 tensors and 27,320,697,856 parameters. They were evaluated against the same BF16 reference logits on WikiText-2 with a fixed context length of 512.

At the exact size of each of the 9 MoQ files, linearly interpolating the Unsloth curve favors this MoQ series on:

• PPL: 8 of 9 points
• p999 KLD: 7 of 9 points
• Mean KLD: 2 of 9 points

The results show the intended tradeoff clearly. The layer-aware MoQ recipes are especially effective on PPL and tail divergence, while the Unsloth Dynamic recipes remain very strong on Mean KLD, particularly at low and middle file sizes. The full table is included below so that users can choose based on the metric that matters for their workload.

Full Quality Results

Actual BPW is computed from the complete GGUF file size, including metadata and alignment. GB is decimal. PPL and all KLD values are lower-is-better; same top-p is higher-is-better.

Series	Recipe	Actual BPW	GB	PPL	Mean KLD	p999 KLD	p99 KLD	RMS delta-p	Same top-p
Jianqiao1 MoQ	3.2	3.0956	10.572	7.408463	0.111637	3.348125	1.084846	9.634%	85.679%
Jianqiao1 MoQ	3.6	3.4821	11.892	7.221080	0.072244	2.351531	0.717370	7.682%	88.722%
Jianqiao1 MoQ	3.8	3.7232	12.715	7.122706	0.048190	1.624749	0.437946	6.077%	90.109%
Jianqiao1 MoQ	4.1	4.1395	14.137	7.068801	0.031460	1.056327	0.285069	4.915%	91.872%
Jianqiao1 MoQ	4.3	4.3600	14.890	7.045628	0.024500	0.805452	0.216612	4.332%	92.648%
Jianqiao1 MoQ	4.6	4.4302	15.129	7.014436	0.020171	0.758995	0.200817	3.936%	93.809%
Jianqiao1 MoQ	4.8	4.6961	16.038	7.008832	0.017282	0.650152	0.169743	3.584%	94.339%
Jianqiao1 MoQ	4.9	4.8055	16.411	7.021034	0.016342	0.651529	0.162814	3.530%	94.423%
Jianqiao1 MoQ	5.1	5.0772	17.339	7.020549	0.012857	0.487299	0.123491	3.134%	94.968%
Unsloth	UD-IQ2_XXS	2.6383	9.010	7.957195	0.163689	4.714475	1.739405	11.822%	82.389%
Unsloth	UD-IQ2_M	3.0219	10.320	7.507406	0.103564	3.590751	1.076759	9.332%	85.496%
Unsloth	UD-Q2_K_XL	3.1263	10.676	7.403969	0.091579	3.107871	0.925859	8.708%	86.160%
Unsloth	UD-IQ3_XXS	3.4885	11.914	7.246240	0.054517	2.259040	0.627471	6.766%	90.359%
Unsloth	UD-Q3_K_XL	3.9358	13.441	7.111811	0.031355	1.248320	0.353498	5.129%	92.396%
Unsloth	UD-Q4_K_XL	5.2483	17.923	6.976017	0.008640	0.422880	0.090719	2.614%	96.073%
Unsloth	UD-Q5_K_XL	5.9203	20.218	6.967300	0.004526	0.200576	0.040594	1.849%	97.153%
Unsloth	UD-Q6_K_XL	7.5911	25.924	6.953912	0.001331	0.062800	0.011536	1.107%	98.493%

The Unsloth rows are local comparison measurements only. Unsloth GGUF files are not redistributed in this repository.

Available Models

The recipe number is a series label. Actual BPW below is calculated from the complete file and is the value to use for exact size comparisons.

File	Actual BPW	Size GB	Size GiB
Qwen3.8-27B-MTP-MoQ-3.2.gguf	3.0956	10.572	9.846
Qwen3.8-27B-MTP-MoQ-3.6.gguf	3.4821	11.892	11.075
Qwen3.8-27B-MTP-MoQ-3.8.gguf	3.7232	12.715	11.842
Qwen3.8-27B-MTP-MoQ-4.1.gguf	4.1395	14.137	13.166
Qwen3.8-27B-MTP-MoQ-4.3.gguf	4.3600	14.890	13.867
Qwen3.8-27B-MTP-MoQ-4.6.gguf	4.4302	15.129	14.090
Qwen3.8-27B-MTP-MoQ-4.8.gguf	4.6961	16.038	14.936
Qwen3.8-27B-MTP-MoQ-4.9.gguf	4.8055	16.411	15.284
Qwen3.8-27B-MTP-MoQ-5.1.gguf	5.0772	17.339	16.148

The repository also includes the locally generated calibration imatrix and a BF16 multimodal projector.

Quantization Approach

This series combines the advantages of the MoQ (Mixed-precision Quantization) scheme at the layer level with tensor-specific strategies derived from Unsloth Dynamic 3.0's low-bit GGUF implementations. Since Qwen3.8 and Qwen3.6 share a matching tensor-level architecture, we were able to transfer and fine-tune layer-specific quantization strategies without being forced to apply a uniform quantization type across all tensor families.

The specific process is as follows:

1. Generate a calibration imatrix specifically for Qwen3.8 using a calibration corpus.
2. Transfer the layer-wise MoQ allocation scheme from Qwen3.6 to the corresponding tensors in Qwen3.8.
3. Compare each tensor family and layer against the original weights.
4. Retain MoQ layer-wise precision adjustments (whether increasing or decreasing precision) if they improve perplexity (PPL) or KLD tail performance.
5. While maintaining the layer-wise structure, optimize average KLD performance for the low-BPW (bits per weight) schemes of versions 3.2 and 3.6 by referencing Unsloth's low-bit quantization strategies.

This is not a single, global quantization preset; different precision settings can be applied to specific tensor families and layers based on their measured or inferred sensitivity.

MTP Precision

Compared to the MTP layer in Qwen 3.6, the MTP layer in Qwen 3.8 employs lower-bit quantization; this is because our tests revealed that the prediction performance of Qwen 3.8's MTP layer—even at Q8 precision—falls short of that of Qwen 3.6, rendering the maintenance of high precision for the MTP layer less meaningful.

Evaluation Conditions

The comparison used llama.cpp build 10276, commit 6ea215d17, with CUDA on an NVIDIA GeForce RTX 5090. WikiText-2 wiki.test.raw was evaluated against BF16 logits from the original Qwen3.8-27B GGUF. Context length was fixed at 512; logical batch was 2048, ubatch was 8192, GPU layers were selected automatically with fit enabled, op offload and flash attention were enabled, and 16 CPU threads were used. GPU evaluations were run strictly one at a time.

The BF16 reference PPL reported by the evaluator was 6.950282.

Usage

Use a recent llama.cpp build with Qwen3.8 and MTP support. The files can be used for ordinary generation or MTP speculative decoding. Choose a file according to available memory and the quality curves above; the nominal recipe label is useful for navigating the series, while the actual file BPW and GB columns provide precise memory-planning values.

License and Acknowledgements

Released under the Apache License 2.0, following the base model license metadata.

Thanks to:

• the Qwen team for Qwen3.8-27B and its MTP architecture;
• the llama.cpp project and contributors for GGUF quantization, MTP support, and evaluation tooling;
• the Unsloth team for the Dynamic GGUF series.