Hugging Face's logo

mistralrs-community
/
Qwen3.5-9B-UQFF

qwen3_5
uqff
mistral.rs
Model card Files
xet
 Community
Qwen3.5-9B-UQFF
File size: 2,185 Bytes
36831a8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
---
tags:
  - uqff
  - mistral.rs
base_model: Qwen/Qwen3.5-9B
base_model_relation: quantized
---

# `Qwen/Qwen3.5-9B`, UQFF quantization

Generated with [mistral.rs](https://github.com/EricLBuehler/mistral.rs) 0.9.0. Documentation: [UQFF docs](https://ericlbuehler.github.io/mistral.rs/guides/perf/use-uqff/).

1) **Flexible** 🌀: Multiple quantization formats in *one* file format with *one* framework to run them all.
2) **Reliable** 🔒: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1.
3) **Easy** 🤗: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file.
4) **Customizable** 🛠️: Make and publish your own UQFF files in minutes.

## Install

Install [mistral.rs](https://github.com/EricLBuehler/mistral.rs) ([full guide](https://ericlbuehler.github.io/mistral.rs/guides/install/)):

**Linux/macOS:**
```
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh
```

**Windows (PowerShell):**
```
irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex
```

## Examples

**Note:** AFQ variants are optimized for Apple Silicon / Metal.

|Quantization|Command|
|--|--|
|AFQ2|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff afq2-0.uqff`|
|AFQ3|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff afq3-0.uqff`|
|AFQ4|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff afq4-0.uqff`|
|AFQ6|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff afq6-0.uqff`|
|AFQ8|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff afq8-0.uqff`|
|Q2K|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff q2k-0.uqff`|
|Q3K|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff q3k-0.uqff`|
|Q4K|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff q4k-0.uqff`|
|Q5K|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff q5k-0.uqff`|
|Q6K|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff q6k-0.uqff`|
|Q8_0|`mistralrs run -m mistralrs-community/Qwen3.5-9B-UQFF --from-uqff q8_0-0.uqff`|