Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,231 +1,7 @@
 ---
-base_model: deepreinforce-ai/Ornith-1.0-35B
-base_model_relation: quantized
-license: apache-2.0
-library_name: mlx
+language: en
 pipeline_tag: text-generation
+library_name: mlx
 tags:
 - mlx
-- mlx-optiq
-- quantization
-- moe
-- qwen3.5
-- qwen35moe
-- apple-silicon
-- speculative-decoding
-- multi-token-prediction
-- qwen3_5
 ---
-
-# Ornith-1.0-35B-oQ7-mtp
-
-MLX format quantization of [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B), produced with mlx-optiq and shipped with a grafted Multi-Token Prediction (MTP) head for speculative decoding on Apple Silicon.
-
-## Model Details
-
-### Model Description
-
-This repository contains a mixed precision MLX quantization of Ornith-1.0-35B, a Qwen3.5-35B-A3B based mixture of experts model. The quantization was produced with mlx-optiq, which performs a per-layer KL sensitivity analysis against the original BF16 checkpoint and assigns each layer either 6-bit or 8-bit precision so that the weighted average lands at a target of 7.5 bits per weight (BPW). Layers that are more sensitive to quantization error are kept at 8-bit, while more robust layers are reduced to 6-bit.
-
-An auxiliary MTP head, grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`, is included as `mtp.safetensors`. It is used as a self-speculative draft model during decoding and is architecturally compatible because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share the same hidden size, layer count, expert count, and block layout, and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.
-
-- **Developed by:** programmer-666
-- **Model type:** Causal decoder only mixture of experts language model (Qwen3.5-35B-A3B architecture, qwen35moe, 40 layers)
-- **Language(s):** Inherited from the base model
-- **License:** Apache 2.0 for this repository. See the Licenses section below for the licenses of the underlying components.
-- **Quantized from model:** [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)
-- **MTP head source:** [mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit](https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit)
-
-### Model Sources
-
-- **Base model:** https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B
-- **Quantization tool:** https://mlx-optiq.com
-- **Serving engine used for benchmarks:** https://github.com/jundot/omlx
-
-## Model Properties
-
-| Property | Value |
-|---|---|
-| Base model | deepreinforce-ai/Ornith-1.0-35B |
-| Architecture | Qwen3.5-35B-A3B (qwen35moe, 40 layers) |
-| Total parameters | 35B |
-| Active parameters per token | 3B |
-| Quantization | mlx-optiq mixed precision (6 bit / 8 bit) |
-| Target BPW | 7.5 |
-| Candidate bits | 6, 8 |
-| Group size | 32 |
-| Calibration sequences | 40 |
-| MTP head source | mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit |
-| Format | MLX safetensors |
-
-## Uses
-
-### Direct Use
-
-This model is intended for local text generation on Apple Silicon devices using MLX based inference engines such as `mlx-lm` or `optiq serve`. It is suited for users who want most of the quality of the BF16 checkpoint at roughly half the memory footprint, with an optional speculative decoding path for faster generation.
-
-### Out-of-Scope Use
-
-This is a quantized derivative of a third party base model. It has not been independently evaluated for safety, factuality, or fitness for any particular downstream task. It should not be used in high stakes settings (medical, legal, financial, or safety critical decisions) without additional evaluation. Refer to the base model card for its intended use cases and known limitations, since these are inherited by this quantization.
-
-## Bias, Risks, and Limitations
-
-Quantization can shift a model's behavior relative to the original weights, even when overall benchmark scores are similar. Mixed precision quantization at 7.5 BPW is expected to be close to BF16 quality, but no independent evaluation of downstream task accuracy, factuality, or bias has been performed for this specific quantized artifact. Users should treat outputs as they would from the base model and are encouraged to run their own evaluations for their use case before deploying it in production.
-
-## How to Get Started with the Model
-
-### With mlx-optiq serve (enables MTP speculative decoding)
-
-```bash
-optiq serve \
-  --model programmer-666/Ornith-1.0-35B-oQ7-mtp \
-  --mtp \
-  --port 8080
-```
-
-### With mlx-lm
-
-```bash
-mlx_lm.generate \
-  --model programmer-666/Ornith-1.0-35B-oQ7-mtp \
-  --prompt "Your prompt here"
-```
-
-Note: MTP is not available through `mlx_lm.generate`. Use `optiq serve` or `omlx` if you want speculative decoding.
-
-## Training Details
-
-This repository does not modify the base model's weights beyond quantization; no additional fine-tuning was performed. For training data and training procedure, refer to the [base model card](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).
-
-### Quantization Procedure
-
-Quantization was performed with mlx-optiq, which runs a per-layer KL sensitivity analysis to assign bit widths. Layers with higher sensitivity to quantization error retain 8-bit precision, while more robust layers are assigned 6-bit precision. The reference model used during sensitivity calibration was the original BF16 checkpoint.
-
-Conversion command:
-
-```bash
-optiq convert deepreinforce-ai/Ornith-1.0-35B \
-  --candidate-bits 6,8 \
-  --target-bpw 7.5 \
-  --reference bf16 \
-  --n-calibration 40 \
-  --group-size 32 \
-  --skip-baselines \
-  -o Ornith-1.0-35B-oQ7-mtp
-```
-
-## Multi-Token Prediction (MTP)
-
-The `mtp.safetensors` file contains an auxiliary prediction head grafted from `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit`. This works because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share an identical structure (hidden dimension 2048, 40 layers, 256 experts, same block layout), and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.
-
-MTP uses this auxiliary head as a draft model for speculative decoding, giving roughly a 1.3x to 1.4x decode speedup on Apple Silicon during greedy generation.
-
-## Hardware Requirements
-
-| Configuration | Notes |
-|---|---|
-| Recommended | Apple Silicon with 64GB or more of unified memory |
-| Tested on | M4 Max MacBook Pro, 128GB |
-| Disk space | Approximately 37GB |
-
-## Evaluation
-
-### Testing Setup
-
-Benchmarks were run with [oMLX](https://github.com/jundot/omlx), an LLM inference tool optimized for Apple Silicon Macs. Three models were compared: this repository (`Ornith-1.0-35B-oQ7-mtp`), the original `Ornith-1.0-35B-bf16` checkpoint, and the MTP head's source model, `Qwen3.6-35B-A3B-OptiQ-4bit`. All figures below come from single request runs unless noted as continuous batching.
-
-### Performance Benchmarks
-
-**Ornith-1.0-35B-oQ7-mtp (this repository)**
-
-| Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
-|---|---|---|---|---|---|---|---|
-| pp1024/tg128 | 787.3 | 10.70 | 1300.6 tok/s | 94.2 tok/s | 2.147 | 536.6 tok/s | 37.36 GB |
-| pp4096/tg128 | 2541.2 | 10.96 | 1611.9 tok/s | 91.9 tok/s | 3.934 | 1073.8 tok/s | 38.14 GB |
-| pp8192/tg128 | 5421.8 | 11.35 | 1510.9 tok/s | 88.8 tok/s | 6.863 | 1212.3 tok/s | 38.48 GB |
-| pp16384/tg128 | 12895.5 | 12.89 | 1270.5 tok/s | 78.2 tok/s | 14.533 | 1136.2 tok/s | 39.10 GB |
-| pp32768/tg128 | 29809.2 | 13.68 | 1099.3 tok/s | 73.7 tok/s | 31.546 | 1042.8 tok/s | 40.44 GB |
-| pp65536/tg128 | 94152.4 | 25.95 | 696.1 tok/s | 38.8 tok/s | 97.449 | 673.8 tok/s | 43.13 GB |
-| pp131072/tg128 | 348440.0 | 30.93 | 376.2 tok/s | 32.6 tok/s | 352.368 | 372.3 tok/s | 48.50 GB |
-| pp200000/tg128 | 726434.2 | 36.19 | 275.3 tok/s | 27.9 tok/s | 731.030 | 273.8 tok/s | 54.17 GB |
-
-Continuous batching, pp1024/tg128:
-
-| Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT (ms) | E2E (s) |
-|---|---|---|---|---|---|---|
-| 1x | 94.2 tok/s | 1.00x | 1300.6 tok/s | 1300.6 tok/s | 787.3 | 2.147 |
-| 2x | 110.1 tok/s | 1.17x | 549.5 tok/s | 274.8 tok/s | 3726.8 | 6.052 |
-| 4x | 125.5 tok/s | 1.33x | 826.0 tok/s | 206.5 tok/s | 4836.1 | 9.038 |
-
-**Ornith-1.0-35B-bf16 (reference)**
-
-| Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
-|---|---|---|---|---|---|---|---|
-| pp1024/tg128 | 834.6 | 16.17 | 1227.0 tok/s | 62.3 tok/s | 2.888 | 398.9 tok/s | 65.62 GB |
-| pp4096/tg128 | 2269.3 | 16.46 | 1805.0 tok/s | 61.2 tok/s | 4.359 | 969.0 tok/s | 66.40 GB |
-| pp8192/tg128 | 4984.5 | 15.94 | 1643.5 tok/s | 63.2 tok/s | 7.009 | 1187.1 tok/s | 66.73 GB |
-| pp16384/tg128 | 11321.8 | 16.73 | 1447.1 tok/s | 60.2 tok/s | 13.447 | 1228.0 tok/s | 67.39 GB |
-| pp32768/tg128 | 28415.1 | 18.08 | 1153.2 tok/s | 55.7 tok/s | 30.711 | 1071.1 tok/s | 68.70 GB |
-| pp65536/tg128 | 85199.7 | 30.94 | 769.2 tok/s | 32.6 tok/s | 89.130 | 736.7 tok/s | 71.35 GB |
-| pp131072/tg128 | 346498.2 | 36.68 | 378.3 tok/s | 27.5 tok/s | 351.157 | 373.6 tok/s | 76.79 GB |
-| pp200000/tg128 | 630686.4 | 40.49 | 317.1 tok/s | 24.9 tok/s | 635.829 | 314.8 tok/s | 82.47 GB |
-
-Continuous batching, pp1024/tg128:
-
-| Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT (ms) | E2E (s) |
-|---|---|---|---|---|---|---|
-| 1x | 62.3 tok/s | 1.00x | 1227.0 tok/s | 1227.0 tok/s | 834.6 | 2.888 |
-| 2x | 43.1 tok/s | 0.69x | 511.9 tok/s | 255.9 tok/s | 4000.4 | 9.941 |
-| 4x | 71.0 tok/s | 1.14x | 938.8 tok/s | 234.7 tok/s | 4212.9 | 11.578 |
-
-**Qwen3.6-35B-A3B-OptiQ-4bit (MTP head source, shown for reference)**
-
-| Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
-|---|---|---|---|---|---|---|---|
-| pp1024/tg128 | 765.7 | 9.03 | 1337.3 tok/s | 111.6 tok/s | 1.913 | 602.3 tok/s | 21.72 GB |
-| pp4096/tg128 | 2466.8 | 9.38 | 1660.5 tok/s | 107.5 tok/s | 3.658 | 1154.7 tok/s | 22.49 GB |
-| pp8192/tg128 | 5296.1 | 9.66 | 1546.8 tok/s | 104.4 tok/s | 6.523 | 1275.6 tok/s | 22.83 GB |
-| pp16384/tg128 | 12197.2 | 10.30 | 1343.3 tok/s | 97.9 tok/s | 13.505 | 1222.6 tok/s | 23.46 GB |
-| pp32768/tg128 | 30598.7 | 11.70 | 1070.9 tok/s | 86.2 tok/s | 32.084 | 1025.3 tok/s | 24.80 GB |
-| pp65536/tg128 | 87223.7 | 18.90 | 751.4 tok/s | 53.3 tok/s | 89.624 | 732.7 tok/s | 27.48 GB |
-| pp131072/tg128 | 343907.5 | 27.19 | 381.1 tok/s | 37.1 tok/s | 347.361 | 377.7 tok/s | 32.86 GB |
-| pp200000/tg128 | 675070.1 | 32.83 | 296.3 tok/s | 30.7 tok/s | 679.240 | 294.6 tok/s | 38.52 GB |
-
-Continuous batching, pp1024/tg128:
-
-| Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT (ms) | E2E (s) |
-|---|---|---|---|---|---|---|
-| 1x | 111.6 tok/s | 1.00x | 1337.3 tok/s | 1337.3 tok/s | 765.7 | 1.913 |
-| 2x | 148.6 tok/s | 1.33x | 569.3 tok/s | 284.6 tok/s | 3597.4 | 5.320 |
-| 4x | 161.0 tok/s | 1.44x | 873.2 tok/s | 218.3 tok/s | 4564.0 | 7.870 |
-
-### Summary
-
-Compared to the BF16 checkpoint, this 7.5 BPW quantization roughly halves peak memory usage (about 37 to 54 GB versus about 66 to 82 GB depending on context length) while matching or exceeding the BF16 model's token generation speed at every tested context length, and reduces time to first token at longer contexts. The 4-bit `Qwen3.6-35B-A3B-OptiQ-4bit` model used as the MTP draft is smaller and faster on its own, which is what allows it to accelerate this model's decoding through speculative decoding.
-
-## Licenses
-
-- Ornith-1.0-35B (base model): MIT License
-- Qwen3.5-35B-A3B (base architecture): Apache 2.0
-- Qwen3.6-35B-A3B (MTP head source): Apache 2.0
-- This repository (quantized weights and MTP head): Apache 2.0
-
-Users should review the license terms of each underlying component before use or redistribution.
-
-## Citation
-
-If you use this model, please cite the original Ornith-1.0-35B model and the mlx-optiq quantization tool.
-
-```
-@misc{ornith-1.0-35b-oq7-mtp,
-  title = {Ornith-1.0-35B-oQ7-mtp},
-  author = {programmer-666},
-  year = {2026},
-  note = {MLX mixed precision quantization of deepreinforce-ai/Ornith-1.0-35B with grafted MTP head},
-  howpublished = {\url{https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ7-mtp}}
-}
-```
-
-## Model Card Contact
-
-For questions about this quantization, open a discussion on this repository's Community tab. For questions about the base model, refer to [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).

chat_template.jinja

@@ -0,0 +1,150 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 248044,
+  "do_sample": true,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
+  "pad_token_id": 248044,
+  "temperature": 1.0,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.8.1"
+}

model-00001-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:344dc26e3839de9898ab3ca20c67d0224004dd8737dfd60f8a879ecba4f28405
+size 5335398706

model-00002-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8643290d679d28b18496c32879d55d73fcb51646dffe5cbb9baa515852feb39a
+size 5337074973

model-00003-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b63a3a0a4ee283f8bddcc0f0ab27b4f6cda7f1e2741359b52c341381fd18fec
+size 5363501976

model-00004-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8a1446beb3a6dd9a38c908209d7002d851394128acee63e9a331b4f3cb17127
+size 5368695667

model-00005-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e981ba9a8b3cdd946e9d8c7d6d7d7a2670a11e68b78f36c0095c62a44818de7
+size 5323230767

model-00006-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3bbaf4160c84b79d013d82c288bca74b03769a4942607dc4e7ca66e972586cd
+size 5366320059

model-00007-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9044631056d75846693b9bcfbeed8c099fff68d9fa61c7a115c66dffe8ebc1db
+size 5359012678

model-00008-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14cac094e8d7997255fa4a318969c4a4726246fc6ea9e658c9fcd1c63c7f309d
+size 1481647677

mtp.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab850c643a37cd0c0b88530a5d6251f72a26c3b1dbf56707b307f5454942b49f
+size 1644816498

optiq_metadata.json

@@ -0,0 +1,1576 @@
+{
+  "method": "optiq_mixed_precision",
+  "base_model": "/Users/suhaarslan/.omlx/models/mlx-community/Ornith-1.0-35B-bf16",
+  "reference": "bf16",
+  "target_bpw": 7.5,
+  "achieved_bpw": 7.754236213817874,
+  "n_high_bits": 362,
+  "n_low_bits": 29,
+  "threshold": 0.0,
+  "per_layer": {
+    "language_model.lm_head": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.39.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_a": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_a": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_qkv": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.self_attn.k_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.35.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_qkv": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_qkv": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.linear_attn.out_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_b": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.self_attn.k_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.31.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.27.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.self_attn.k_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.23.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.linear_attn.out_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.linear_attn.out_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.19.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.linear_attn.out_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.linear_attn.out_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.15.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_qkv": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.self_attn.o_proj": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.11.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.7.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_qkv": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.3.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_z": {
+      "bits": 6,
+      "group_size": 32
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.mlp.gate": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 32
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 32
+    }
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "tool_parser_type": "qwen3_coder",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}