vultr/Qwen3.5-397B-A17B-GGUF

Qwen3.5 397B A17B, GGUF

Qwen3.5 397B A17B is a multimodal Mixture-of-Experts (MoE) large language model optimized for long-context reasoning, vision-language understanding, and scalable agentic workflows. The model employs a hybrid architecture. It integrates a vision encoder for unified multimodal processing.

This repository holds Vultr's GGUF quantizations of it. They are hybrid quantizations: rather than applying one precision to the whole network, each layer's precision is chosen from a measured importance score, so the bits go where they change the output most.

Model Overview

Attribute	Value
Base model	Qwen/Qwen3.5-397B-A17B
Architecture	qwen35moe, Mixture-of-Experts
Parameters	397B total
Active parameters	17B
Layers	60, hybrid Gated DeltaNet and full attention
Vision encoder layers	27
Hidden size	4,096
Attention heads	32
Experts	512 total, 10 routed, 1 shared, per token
Context length	262K tokens, extendable to ~1M
Vocabulary	248,320
License	Apache 2.0

Attribute	Value
Base model	Qwen/Qwen3.5-397B-A17B
Architecture	qwen35moe, Mixture-of-Experts
Parameters	397B total
Active parameters	17B
Layers	60, hybrid Gated DeltaNet and full attention
Vision encoder layers	27
Hidden size	4,096
Attention heads	32
Experts	512 total, 10 routed, 1 shared, per token
Context length	262K tokens, extendable to ~1M
Vocabulary	248,320
License	Apache 2.0

Quantizations

File	Type	Size	KLD vs bf16
Qwen3.5-397B-A17B-Q5_K_M-*.gguf (3 parts)	Q5_K_M	309.83 GB	0.0045
Qwen3.5-397B-A17B-Q4_K_M-*.gguf (2 parts)	Q4_K_M	264.44 GB	0.0073
Qwen3.5-397B-A17B-IQ3_M-*.gguf (2 parts)	IQ3_M	190.82 GB	0.0242
Qwen3.5-397B-A17B-IQ3_S-*.gguf (2 parts)	IQ3_S	188.55 GB	0.0243

File	Type	Size	KLD vs bf16
Qwen3.5-397B-A17B-Q5_K_M-*.gguf (3 parts)	Q5_K_M	309.83 GB	0.0045
Qwen3.5-397B-A17B-Q4_K_M-*.gguf (2 parts)	Q4_K_M	264.44 GB	0.0073
Qwen3.5-397B-A17B-IQ3_M-*.gguf (2 parts)	IQ3_M	190.82 GB	0.0242
Qwen3.5-397B-A17B-IQ3_S-*.gguf (2 parts)	IQ3_S	188.55 GB	0.0243

The vision encoder is in Qwen3.5-397B-A17B-mmproj-f16.gguf (918 MB). llama.cpp downloads it automatically with the command under Run, so image input works without extra steps.

KL divergence measures how far a quantized model's next-token probabilities drift from those of the original bf16 model, averaged over every token of the evaluation text. The lower the value, the closer the quantized model is to the original; zero would mean identical output.

Evaluation

Every variant in this repository was evaluated against the original bf16 model for perplexity, KL divergence, top-token agreement, HellaSwag and WinoGrande, under identical conditions. The full results are maintained on the model's page in the Vultr Model Library:

https://docs.vultr.com/models/alibaba-qwen-35/qwen3.5-397b-a17b?tab=quantized

That page is the reference for these files. The summary above is an excerpt from it.

Run

llama-server -hf vultr/Qwen3.5-397B-A17B-GGUF:Q4_K_M

The model's chat template is embedded in every file, so no extra prompt formatting is needed.

The repository also includes mtp-Qwen3.5-397B-A17B-Q8_0.gguf, the model's multi-token prediction (MTP) head, for optional speculative decoding with any quant above:

llama-server -hf vultr/Qwen3.5-397B-A17B-GGUF:Q4_K_M --spec-type draft-mtp

License

These files are derived from Qwen/Qwen3.5-397B-A17B and are distributed under the same license, Apache 2.0. The model is the work of Qwen; Vultr produced the quantizations.