Hugging Face's logo

mistralrs-community
/
Qwen3-Embedding-4B-UQFF

qwen3
uqff
mistral.rs
Model card Files
xet
 Community
EricB HF Staff commited on
Commit
20dfeb7
·
0 Parent(s):

Upload uqff_report.json with huggingface_hub

Browse files
Files changed (19) hide show
  1. .gitattributes +47 -0
  2. 1_Pooling/config.json +10 -0
  3. README.md +48 -0
  4. afq2-0.uqff +3 -0
  5. afq3-0.uqff +3 -0
  6. afq4-0.uqff +3 -0
  7. afq6-0.uqff +3 -0
  8. afq8-0.uqff +3 -0
  9. config.json +30 -0
  10. modules.json +20 -0
  11. q2k-0.uqff +3 -0
  12. q3k-0.uqff +3 -0
  13. q4k-0.uqff +3 -0
  14. q5k-0.uqff +3 -0
  15. q6k-0.uqff +3 -0
  16. q8_0-0.uqff +3 -0
  17. residual.safetensors +3 -0
  18. tokenizer.json +3 -0
  19. uqff_report.json +0 -0
.gitattributes ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ afq2-0.uqff filter=lfs diff=lfs merge=lfs -text
37
+ afq3-0.uqff filter=lfs diff=lfs merge=lfs -text
38
+ afq4-0.uqff filter=lfs diff=lfs merge=lfs -text
39
+ afq6-0.uqff filter=lfs diff=lfs merge=lfs -text
40
+ afq8-0.uqff filter=lfs diff=lfs merge=lfs -text
41
+ q2k-0.uqff filter=lfs diff=lfs merge=lfs -text
42
+ q3k-0.uqff filter=lfs diff=lfs merge=lfs -text
43
+ q4k-0.uqff filter=lfs diff=lfs merge=lfs -text
44
+ q5k-0.uqff filter=lfs diff=lfs merge=lfs -text
45
+ q6k-0.uqff filter=lfs diff=lfs merge=lfs -text
46
+ q8_0-0.uqff filter=lfs diff=lfs merge=lfs -text
47
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
1_Pooling/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 2560,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": false,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false,
7
+ "pooling_mode_weightedmean_tokens": false,
8
+ "pooling_mode_lasttoken": true,
9
+ "include_prompt": true
10
+ }
README.md ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - uqff
4
+ - mistral.rs
5
+ base_model: Qwen/Qwen3-Embedding-4B
6
+ base_model_relation: quantized
7
+ ---
8
+
9
+ # `Qwen/Qwen3-Embedding-4B`, UQFF quantization
10
+
11
+ Generated with [mistral.rs](https://github.com/EricLBuehler/mistral.rs) 0.9.0. Documentation: [UQFF docs](https://ericlbuehler.github.io/mistral.rs/guides/perf/use-uqff/).
12
+
13
+ 1) **Flexible** 🌀: Multiple quantization formats in *one* file format with *one* framework to run them all.
14
+ 2) **Reliable** 🔒: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1.
15
+ 3) **Easy** 🤗: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file.
16
+ 4) **Customizable** 🛠️: Make and publish your own UQFF files in minutes.
17
+
18
+ ## Install
19
+
20
+ Install [mistral.rs](https://github.com/EricLBuehler/mistral.rs) ([full guide](https://ericlbuehler.github.io/mistral.rs/guides/install/)):
21
+
22
+ **Linux/macOS:**
23
+ ```
24
+ curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh
25
+ ```
26
+
27
+ **Windows (PowerShell):**
28
+ ```
29
+ irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex
30
+ ```
31
+
32
+ ## Examples
33
+
34
+ **Note:** AFQ variants are optimized for Apple Silicon / Metal.
35
+
36
+ |Quantization|Command|
37
+ |--|--|
38
+ |AFQ2|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff afq2-0.uqff`|
39
+ |AFQ3|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff afq3-0.uqff`|
40
+ |AFQ4|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff afq4-0.uqff`|
41
+ |AFQ6|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff afq6-0.uqff`|
42
+ |AFQ8|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff afq8-0.uqff`|
43
+ |Q2K|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff q2k-0.uqff`|
44
+ |Q3K|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff q3k-0.uqff`|
45
+ |Q4K|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff q4k-0.uqff`|
46
+ |Q5K|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff q5k-0.uqff`|
47
+ |Q6K|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff q6k-0.uqff`|
48
+ |Q8_0|`mistralrs run -m mistralrs-community/Qwen3-Embedding-4B-UQFF --from-uqff q8_0-0.uqff`|
afq2-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad8cd961a83b76fd2c954428b3d7c3b9661932a1ce8277b0f55bdb18dc192ff7
3
+ size 1135577544
afq3-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8791da6e3f1dbc742769a63ac270ec7b5156fba42bdf042fc9978d7f576a1fdc
3
+ size 1589742592
afq4-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:01dedbe915dfbba7cdd3f5ca37f00143a54b64fe1abf8723f8cbf77bcbd55064
3
+ size 2043907208
afq6-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7aef2eac91b0ab1e3de96d223c85e71d65f2bed44b2f1ac0d18b778178661814
3
+ size 2952236264
afq8-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5a524a46d5d3933cdc9e201757bec6a2c8b388c05264d132683b8f78d04c3792
3
+ size 3860565312
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 151643,
8
+ "eos_token_id": 151645,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 2560,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 9728,
14
+ "max_position_embeddings": 40960,
15
+ "max_window_layers": 36,
16
+ "model_type": "qwen3",
17
+ "num_attention_heads": 32,
18
+ "num_hidden_layers": 36,
19
+ "num_key_value_heads": 8,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_scaling": null,
22
+ "rope_theta": 1000000,
23
+ "sliding_window": null,
24
+ "tie_word_embeddings": true,
25
+ "torch_dtype": "bfloat16",
26
+ "transformers_version": "4.51.2",
27
+ "use_cache": true,
28
+ "use_sliding_window": false,
29
+ "vocab_size": 151665
30
+ }
modules.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ },
14
+ {
15
+ "idx": 2,
16
+ "name": "2",
17
+ "path": "2_Normalize",
18
+ "type": "sentence_transformers.models.Normalize"
19
+ }
20
+ ]
q2k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d2aacaa3f2ee2a0c424f50d09236b99542fdba99e40e1fd3d4240ba385846932
3
+ size 1192285272
q3k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7871a8441bdce790c835d0b32599c731f825b1faf9056fd18855ac3e82ecbd45
3
+ size 1561294320
q4k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ce9bb8b7426ca3a04851f149b61c7585ec87dd6489c23b143d3a38bf15bef315
3
+ size 2043844256
q5k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2da897db01b5357e2ab5020d9a667e569a922f9f01c03ada7abfba6811616feb
3
+ size 2498008832
q6k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9b7780761aa1d46733a75fb463546b1105886872741e6c01863713f327b3f06d
3
+ size 2980558656
q8_0-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e2d01fd32ba0a79f41fafafff3c2d1c597a53a5ea81f1ea307eb65e0d0d9181a
3
+ size 3860502496
residual.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a2a5e5aabe0721693d00101a20811e1a74652af3a4adf07ad8fe94aacc55c298
3
+ size 776932600
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15e1572e1b7425070bbf597336081bffbc12348828b670366746a42906ace655
3
+ size 11423637
uqff_report.json ADDED
The diff for this file is too large to render. See raw diff