Upload Qwen3.8 NVFP4 model with NVFP4 PLE

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,16 @@
+Qwen Community License 1.0
+
+Copyright (c) 2026 Qwen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software, including the model weights, parameters, configuration files, inference code and associated documentation files (collectively, the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, sell, deploy, host, fine-tune, and create derivative works from (collectively, "Use" or "Using") copies of the Software; and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+1. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. If the Software (or any derivative works thereof) is Used for any of the licensee's commercial products or services that have more than 100,000,000 monthly active users or US$ 20,000,000 (or equivalent in other currencies) monthly revenue, respective model name must be prominently displayed on the user interface of such product or service; and,
+
+2. If the licensee or any of its affiliates conducts a Model as a Service or AI Work Assistant business, the licensee shall obtain a separate license from Qwen before Using the Software or its derivative works for any commercial purpose. The foregoing requirement shall not apply to the licensee's internal Use of the Software, provided that such Use does not make the Software, its outputs, or its underlying model capabilities available to any third party.
+
+"Model as a Service" means giving a third party access to language model inference or fine-tuning (e.g., via API or a hosted endpoint) in a manner that allows such third parties to exercise meaningful control over the inputs, parameters, or training data. This does not include the mere relaying of requests to models hosted by other third parties. 
+“AI Work Assistant” means an independent AI-powered product primarily designed for AI-assisted coding or office productivity (e.g., Qoder and QwenWork). It does not include: (a) a single-purpose AI tool (such as an AI translation tool); (b) an AI assistant primarily designed for a domain other than coding or office productivity (such as Taobao AI Shopping Assistant or AMap AI Chat); or (c) an AI assistant that is a feature of a product whose primary purpose is not AI-assisted coding or office productivity.
+
+THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL QWEN, ITS AFFILIATES OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. THE USE OF THE SOFTWARE MUST COMPLY WITH APPLICABLE LAWS AND REGULATIONS, AND MUST NOT INFRINGE THE INTELLECTUAL PROPERTY RIGHTS OF ANY THIRD PARTY.
+
+For any questions regarding this license, please contact model-business@notice.qwencloud.com.

README.md

@@ -0,0 +1,133 @@
+---
+library_name: vllm
+license: other
+license_name: qwen-community-1.0
+license_link: LICENSE
+base_model: Inferact/Qwen3.8-Flash-Next-NVFP4
+base_model_relation: quantized
+pipeline_tag: image-text-to-text
+inference: false
+tags:
+  - qwen
+  - qwen3.8
+  - qwen4
+  - modelopt
+  - nvfp4
+  - vllm
+  - custom-code
+---
+
+# Qwen3.8-Flash-Next NVFP4 with NVFP4 PLE
+
+This repository is a serving derivative of
+[`Inferact/Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/Inferact/Qwen3.8-Flash-Next-NVFP4).
+It keeps the ModelOpt NVFP4 backbone unchanged and converts only the BF16
+position-learning enhancement (PLE) n-gram table to NVFP4.
+
+This repository contains model weights and Hugging Face metadata only. It does
+not include runtime code, shell scripts, Docker files, or result artifacts.
+
+The packed PLE format requires an external `qwen38-nvfp4-ple` runtime plugin.
+Stock vLLM and Transformers cannot load it. The checkpoint retains the original
+vision tower.
+
+## Storage
+
+| Artifact | Exact bytes | GiB |
+|---|---:|---:|
+| Source checkpoint payload | 182,738,792,440 | 170.189 |
+| Source BF16 PLE table | 102,400,491,520 | 95.368 |
+| Converted NVFP4 PLE table | 28,800,138,752 | 26.822 |
+| Complete converted checkpoint | 109,138,439,672 | 101.643 |
+
+The conversion saves 73,600,352,768 bytes, or 68.546 GiB. It does not use
+REAP or remove experts.
+
+## Why this fits on one DGX Spark
+
+This release targets one NVIDIA DGX Spark because its GB10 CPU and GPU share
+128 GB of LPDDR5x unified system memory. [NVIDIA documents the hardware and
+memory layout](https://docs.nvidia.com/dgx/dgx-spark/hardware.html). The GPU
+executes the NVFP4 backbone. The plugin keeps the 26.822 GiB packed PLE table
+CPU-offloaded and decodes only requested rows to BF16. Both allocations draw
+from the same physical memory pool.
+
+In a measured full multimodal TP1 run, vLLM reported 73.97 GiB during model
+loading and reserved a 2 GiB KV cache. The vision tower added 0.85 GiB over
+text-only serving. The PLE worker has been tested only with vLLM's `mp`
+executor at TP1.
+
+API startup took about 639 seconds. After image and text requests, the host had
+19.8 GiB available with 14.0 GiB of system-wide swap in use. The default
+4,096-token context, single sequence, and 2 GiB KV cache are conservative.
+This demonstrates fit; it is not a throughput claim.
+
+## External runtime contract
+
+| Component | Required or tested value |
+|---|---|
+| vLLM image | `vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8` |
+| vLLM version | `0.1.dev20073+g8e685d198` |
+| PLE mode | CPU offload with BF16 lookup output |
+| Serving model | Full `Qwen4ExpForConditionalGeneration` wrapper |
+| Multimodal limits | One image, no video, 65,536-to-262,144-pixel budget |
+| Executor | `mp`, including at TP1 |
+| Validated GPU | NVIDIA GB10, compute capability 12.1 |
+
+The pinned vLLM `uni` executor does not spawn the PLE CPU worker. The external
+plugin must select `mp` and register itself inside vLLM's special PLE
+subprocess.
+
+## PLE format
+
+Format `qwen38-nvfp4-ple-v1` stores each group of 16 values as packed E2M1
+values, one E4M3 block scale, and one FP32 global decode scale per source PLE
+shard. The runtime gathers only requested rows and decodes those rows to BF16.
+It never expands the complete table.
+
+## Evaluation
+
+A controlled teacher-forced study used 32 calibration prompts and eight held-
+out prompts, covering 236 next-token positions. The executable reference was
+the official FP8 backbone with normal PLE behavior.
+
+These measurements isolate changes to the PLE representation. They do not
+measure the separate quality difference between the Inferact NVFP4 backbone
+and the original model. This uploaded checkpoint uses the first row below;
+the REAP rows are comparison experiments only.
+
+| Perturbation | Top-1 agreement | Reference top-1 in candidate top-5 | KL, nats | NLL delta, nats | Logit cosine |
+|---|---:|---:|---:|---:|---:|
+| NVFP4 PLE only | 91.525% | 100.000% | 0.015591 | +0.019442 | 0.995399 |
+| REAP only, 25% experts removed | 87.712% | 100.000% | 0.042569 | +0.043668 | 0.991027 |
+| REAP and NVFP4 PLE | 91.102% | 99.576% | 0.040298 | +0.040545 | 0.990824 |
+
+The converted checkpoint passed text-only and full multimodal vLLM smoke
+tests. The multimodal run loaded the vision tower and served PNG inputs. With a
+neutral prompt, it correctly identified a generated Moon scene, Earth, casual
+clothing, and the absence of a spacesuit.
+
+
+## Limits
+
+- The alignment study is not a task-quality benchmark.
+- The runtime checks are smoke tests, not throughput or vision benchmarks.
+- Long-context, concurrent-request, tensor-parallel, and expert-parallel paths
+  have not been validated.
+- The validated image path accepts one image within the default pixel budget.
+  Video is disabled by default and has not been validated.
+- The source MTP weights remain in the checkpoint, but the validated runtime
+  did not enable MTP.
+- The tested host had system-wide swap in use after inference.
+- The PLE decoder uses ordinary PyTorch CPU operations. A fused decoder could
+  reduce lookup latency.
+
+## Provenance and license
+
+The backbone comes from
+[`Inferact/Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/Inferact/Qwen3.8-Flash-Next-NVFP4),
+which derives from
+[`Qwen/Qwen3.8-Flash-Next`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
+Only the PLE representation changed here.
+
+The model weights remain under the Qwen Community License 1.0 in `LICENSE`.

chat_template.jinja

@@ -0,0 +1,170 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set reasoning_instructions = '' %}
+{%- if enable_thinking is undefined or enable_thinking is true %}
+    {%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
+    {%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
+        {{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
+    {%- endif %}
+    {%- if resolved_reasoning_effort == 'xhigh' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
+    {%- elif resolved_reasoning_effort == 'low' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
+    {%- endif %}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {%- if reasoning_instructions %}
+        {{- reasoning_instructions + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '')  + content + '<|im_end|>\n' }}
+        {%- elif reasoning_instructions %}
+            {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+        {%- endif %}
+    {%- elif reasoning_instructions %}
+        {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined and tool_call.arguments != '' %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,1470 @@
+{
+  "architectures": [
+    "Qwen4ExpForConditionalGeneration"
+  ],
+  "image_token_id": 248056,
+  "language_model_only": false,
+  "model_type": "qwen4_exp",
+  "quantization_config": {
+    "activation_scale_search": false,
+    "actorder": false,
+    "batch_size": 1,
+    "calibration_applied": true,
+    "cold_expert_hessian_shrinkage": false,
+    "damp": 0.01,
+    "expert_block_reorder": false,
+    "gptq_applied": false,
+    "gptq_block_size": 128,
+    "gptq_reconstruction_rounds": 0,
+    "gptq_reconstruction_scope": null,
+    "group_size": 16,
+    "hessian_aware_codec_search": false,
+    "ignore": [
+      "lm_head",
+      "model.language_model.embed_tokens",
+      "model.language_model.hyper_connection_mixer.hc_norm",
+      "model.language_model.hyper_connection_mixer.input_mix_weight_down",
+      "model.language_model.hyper_connection_mixer.input_mix_weight_up",
+      "model.language_model.layers.0.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.0.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.0.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.0.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.0.linear_attn.A_log",
+      "model.language_model.layers.0.linear_attn.conv1d",
+      "model.language_model.layers.0.linear_attn.dt_bias",
+      "model.language_model.layers.0.linear_attn.in_proj_a",
+      "model.language_model.layers.0.linear_attn.in_proj_b",
+      "model.language_model.layers.0.linear_attn.in_proj_qkv",
+      "model.language_model.layers.0.linear_attn.in_proj_z",
+      "model.language_model.layers.0.linear_attn.norm",
+      "model.language_model.layers.0.linear_attn.out_proj",
+      "model.language_model.layers.0.mlp.gate",
+      "model.language_model.layers.0.mlp.shared_expert.down_proj",
+      "model.language_model.layers.0.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.0.mlp.shared_expert.up_proj",
+      "model.language_model.layers.0.mlp.shared_expert_gate",
+      "model.language_model.layers.0.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.0.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.0.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.0.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.1.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.1.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.1.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.1.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.1.linear_attn.A_log",
+      "model.language_model.layers.1.linear_attn.conv1d",
+      "model.language_model.layers.1.linear_attn.dt_bias",
+      "model.language_model.layers.1.linear_attn.in_proj_a",
+      "model.language_model.layers.1.linear_attn.in_proj_b",
+      "model.language_model.layers.1.linear_attn.in_proj_qkv",
+      "model.language_model.layers.1.linear_attn.in_proj_z",
+      "model.language_model.layers.1.linear_attn.norm",
+      "model.language_model.layers.1.linear_attn.out_proj",
+      "model.language_model.layers.1.mlp.gate",
+      "model.language_model.layers.1.mlp.shared_expert.down_proj",
+      "model.language_model.layers.1.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.1.mlp.shared_expert.up_proj",
+      "model.language_model.layers.1.mlp.shared_expert_gate",
+      "model.language_model.layers.1.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.1.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.1.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.1.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.1.ple.conv1d",
+      "model.language_model.layers.1.ple.key_proj",
+      "model.language_model.layers.1.ple.norm_conv",
+      "model.language_model.layers.1.ple.norm_key",
+      "model.language_model.layers.1.ple.norm_query",
+      "model.language_model.layers.1.ple.ple_embedding.layer_multipliers",
+      "model.language_model.layers.1.ple.ple_embedding.ngram_embedding",
+      "model.language_model.layers.1.ple.ple_embedding.ngram_heads_offsets",
+      "model.language_model.layers.1.ple.ple_embedding.ngram_heads_vocab_sizes",
+      "model.language_model.layers.1.ple.value_proj",
+      "model.language_model.layers.10.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.10.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.10.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.10.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.10.linear_attn.A_log",
+      "model.language_model.layers.10.linear_attn.conv1d",
+      "model.language_model.layers.10.linear_attn.dt_bias",
+      "model.language_model.layers.10.linear_attn.in_proj_a",
+      "model.language_model.layers.10.linear_attn.in_proj_b",
+      "model.language_model.layers.10.linear_attn.in_proj_qkv",
+      "model.language_model.layers.10.linear_attn.in_proj_z",
+      "model.language_model.layers.10.linear_attn.norm",
+      "model.language_model.layers.10.linear_attn.out_proj",
+      "model.language_model.layers.10.mlp.gate",
+      "model.language_model.layers.10.mlp.shared_expert.down_proj",
+      "model.language_model.layers.10.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.10.mlp.shared_expert.up_proj",
+      "model.language_model.layers.10.mlp.shared_expert_gate",
+      "model.language_model.layers.10.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.10.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.10.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.10.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.11.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.11.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.11.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.11.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.11.mlp.gate",
+      "model.language_model.layers.11.mlp.shared_expert.down_proj",
+      "model.language_model.layers.11.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.11.mlp.shared_expert.up_proj",
+      "model.language_model.layers.11.mlp.shared_expert_gate",
+      "model.language_model.layers.11.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.11.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.11.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.11.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.11.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.11.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.11.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.11.self_attn.k_norm",
+      "model.language_model.layers.11.self_attn.k_proj",
+      "model.language_model.layers.11.self_attn.o_proj",
+      "model.language_model.layers.11.self_attn.q_norm",
+      "model.language_model.layers.11.self_attn.q_proj",
+      "model.language_model.layers.11.self_attn.v_proj",
+      "model.language_model.layers.12.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.12.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.12.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.12.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.12.linear_attn.A_log",
+      "model.language_model.layers.12.linear_attn.conv1d",
+      "model.language_model.layers.12.linear_attn.dt_bias",
+      "model.language_model.layers.12.linear_attn.in_proj_a",
+      "model.language_model.layers.12.linear_attn.in_proj_b",
+      "model.language_model.layers.12.linear_attn.in_proj_qkv",
+      "model.language_model.layers.12.linear_attn.in_proj_z",
+      "model.language_model.layers.12.linear_attn.norm",
+      "model.language_model.layers.12.linear_attn.out_proj",
+      "model.language_model.layers.12.mlp.gate",
+      "model.language_model.layers.12.mlp.shared_expert.down_proj",
+      "model.language_model.layers.12.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.12.mlp.shared_expert.up_proj",
+      "model.language_model.layers.12.mlp.shared_expert_gate",
+      "model.language_model.layers.12.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.12.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.12.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.12.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.13.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.13.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.13.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.13.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.13.linear_attn.A_log",
+      "model.language_model.layers.13.linear_attn.conv1d",
+      "model.language_model.layers.13.linear_attn.dt_bias",
+      "model.language_model.layers.13.linear_attn.in_proj_a",
+      "model.language_model.layers.13.linear_attn.in_proj_b",
+      "model.language_model.layers.13.linear_attn.in_proj_qkv",
+      "model.language_model.layers.13.linear_attn.in_proj_z",
+      "model.language_model.layers.13.linear_attn.norm",
+      "model.language_model.layers.13.linear_attn.out_proj",
+      "model.language_model.layers.13.mlp.gate",
+      "model.language_model.layers.13.mlp.shared_expert.down_proj",
+      "model.language_model.layers.13.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.13.mlp.shared_expert.up_proj",
+      "model.language_model.layers.13.mlp.shared_expert_gate",
+      "model.language_model.layers.13.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.13.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.13.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.13.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.14.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.14.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.14.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.14.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.14.linear_attn.A_log",
+      "model.language_model.layers.14.linear_attn.conv1d",
+      "model.language_model.layers.14.linear_attn.dt_bias",
+      "model.language_model.layers.14.linear_attn.in_proj_a",
+      "model.language_model.layers.14.linear_attn.in_proj_b",
+      "model.language_model.layers.14.linear_attn.in_proj_qkv",
+      "model.language_model.layers.14.linear_attn.in_proj_z",
+      "model.language_model.layers.14.linear_attn.norm",
+      "model.language_model.layers.14.linear_attn.out_proj",
+      "model.language_model.layers.14.mlp.gate",
+      "model.language_model.layers.14.mlp.shared_expert.down_proj",
+      "model.language_model.layers.14.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.14.mlp.shared_expert.up_proj",
+      "model.language_model.layers.14.mlp.shared_expert_gate",
+      "model.language_model.layers.14.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.14.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.14.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.14.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.15.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.15.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.15.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.15.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.15.mlp.gate",
+      "model.language_model.layers.15.mlp.shared_expert.down_proj",
+      "model.language_model.layers.15.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.15.mlp.shared_expert.up_proj",
+      "model.language_model.layers.15.mlp.shared_expert_gate",
+      "model.language_model.layers.15.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.15.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.15.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.15.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.15.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.15.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.15.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.15.self_attn.k_norm",
+      "model.language_model.layers.15.self_attn.k_proj",
+      "model.language_model.layers.15.self_attn.o_proj",
+      "model.language_model.layers.15.self_attn.q_norm",
+      "model.language_model.layers.15.self_attn.q_proj",
+      "model.language_model.layers.15.self_attn.v_proj",
+      "model.language_model.layers.16.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.16.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.16.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.16.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.16.linear_attn.A_log",
+      "model.language_model.layers.16.linear_attn.conv1d",
+      "model.language_model.layers.16.linear_attn.dt_bias",
+      "model.language_model.layers.16.linear_attn.in_proj_a",
+      "model.language_model.layers.16.linear_attn.in_proj_b",
+      "model.language_model.layers.16.linear_attn.in_proj_qkv",
+      "model.language_model.layers.16.linear_attn.in_proj_z",
+      "model.language_model.layers.16.linear_attn.norm",
+      "model.language_model.layers.16.linear_attn.out_proj",
+      "model.language_model.layers.16.mlp.gate",
+      "model.language_model.layers.16.mlp.shared_expert.down_proj",
+      "model.language_model.layers.16.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.16.mlp.shared_expert.up_proj",
+      "model.language_model.layers.16.mlp.shared_expert_gate",
+      "model.language_model.layers.16.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.16.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.16.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.16.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.17.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.17.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.17.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.17.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.17.linear_attn.A_log",
+      "model.language_model.layers.17.linear_attn.conv1d",
+      "model.language_model.layers.17.linear_attn.dt_bias",
+      "model.language_model.layers.17.linear_attn.in_proj_a",
+      "model.language_model.layers.17.linear_attn.in_proj_b",
+      "model.language_model.layers.17.linear_attn.in_proj_qkv",
+      "model.language_model.layers.17.linear_attn.in_proj_z",
+      "model.language_model.layers.17.linear_attn.norm",
+      "model.language_model.layers.17.linear_attn.out_proj",
+      "model.language_model.layers.17.mlp.gate",
+      "model.language_model.layers.17.mlp.shared_expert.down_proj",
+      "model.language_model.layers.17.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.17.mlp.shared_expert.up_proj",
+      "model.language_model.layers.17.mlp.shared_expert_gate",
+      "model.language_model.layers.17.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.17.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.17.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.17.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.18.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.18.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.18.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.18.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.18.linear_attn.A_log",
+      "model.language_model.layers.18.linear_attn.conv1d",
+      "model.language_model.layers.18.linear_attn.dt_bias",
+      "model.language_model.layers.18.linear_attn.in_proj_a",
+      "model.language_model.layers.18.linear_attn.in_proj_b",
+      "model.language_model.layers.18.linear_attn.in_proj_qkv",
+      "model.language_model.layers.18.linear_attn.in_proj_z",
+      "model.language_model.layers.18.linear_attn.norm",
+      "model.language_model.layers.18.linear_attn.out_proj",
+      "model.language_model.layers.18.mlp.gate",
+      "model.language_model.layers.18.mlp.shared_expert.down_proj",
+      "model.language_model.layers.18.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.18.mlp.shared_expert.up_proj",
+      "model.language_model.layers.18.mlp.shared_expert_gate",
+      "model.language_model.layers.18.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.18.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.18.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.18.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.19.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.19.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.19.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.19.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.19.mlp.gate",
+      "model.language_model.layers.19.mlp.shared_expert.down_proj",
+      "model.language_model.layers.19.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.19.mlp.shared_expert.up_proj",
+      "model.language_model.layers.19.mlp.shared_expert_gate",
+      "model.language_model.layers.19.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.19.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.19.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.19.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.19.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.19.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.19.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.19.self_attn.k_norm",
+      "model.language_model.layers.19.self_attn.k_proj",
+      "model.language_model.layers.19.self_attn.o_proj",
+      "model.language_model.layers.19.self_attn.q_norm",
+      "model.language_model.layers.19.self_attn.q_proj",
+      "model.language_model.layers.19.self_attn.v_proj",
+      "model.language_model.layers.2.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.2.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.2.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.2.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.2.linear_attn.A_log",
+      "model.language_model.layers.2.linear_attn.conv1d",
+      "model.language_model.layers.2.linear_attn.dt_bias",
+      "model.language_model.layers.2.linear_attn.in_proj_a",
+      "model.language_model.layers.2.linear_attn.in_proj_b",
+      "model.language_model.layers.2.linear_attn.in_proj_qkv",
+      "model.language_model.layers.2.linear_attn.in_proj_z",
+      "model.language_model.layers.2.linear_attn.norm",
+      "model.language_model.layers.2.linear_attn.out_proj",
+      "model.language_model.layers.2.mlp.gate",
+      "model.language_model.layers.2.mlp.shared_expert.down_proj",
+      "model.language_model.layers.2.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.2.mlp.shared_expert.up_proj",
+      "model.language_model.layers.2.mlp.shared_expert_gate",
+      "model.language_model.layers.2.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.2.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.2.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.2.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.20.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.20.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.20.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.20.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.20.linear_attn.A_log",
+      "model.language_model.layers.20.linear_attn.conv1d",
+      "model.language_model.layers.20.linear_attn.dt_bias",
+      "model.language_model.layers.20.linear_attn.in_proj_a",
+      "model.language_model.layers.20.linear_attn.in_proj_b",
+      "model.language_model.layers.20.linear_attn.in_proj_qkv",
+      "model.language_model.layers.20.linear_attn.in_proj_z",
+      "model.language_model.layers.20.linear_attn.norm",
+      "model.language_model.layers.20.linear_attn.out_proj",
+      "model.language_model.layers.20.mlp.gate",
+      "model.language_model.layers.20.mlp.shared_expert.down_proj",
+      "model.language_model.layers.20.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.20.mlp.shared_expert.up_proj",
+      "model.language_model.layers.20.mlp.shared_expert_gate",
+      "model.language_model.layers.20.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.20.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.20.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.20.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.21.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.21.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.21.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.21.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.21.linear_attn.A_log",
+      "model.language_model.layers.21.linear_attn.conv1d",
+      "model.language_model.layers.21.linear_attn.dt_bias",
+      "model.language_model.layers.21.linear_attn.in_proj_a",
+      "model.language_model.layers.21.linear_attn.in_proj_b",
+      "model.language_model.layers.21.linear_attn.in_proj_qkv",
+      "model.language_model.layers.21.linear_attn.in_proj_z",
+      "model.language_model.layers.21.linear_attn.norm",
+      "model.language_model.layers.21.linear_attn.out_proj",
+      "model.language_model.layers.21.mlp.gate",
+      "model.language_model.layers.21.mlp.shared_expert.down_proj",
+      "model.language_model.layers.21.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.21.mlp.shared_expert.up_proj",
+      "model.language_model.layers.21.mlp.shared_expert_gate",
+      "model.language_model.layers.21.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.21.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.21.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.21.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.22.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.22.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.22.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.22.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.22.linear_attn.A_log",
+      "model.language_model.layers.22.linear_attn.conv1d",
+      "model.language_model.layers.22.linear_attn.dt_bias",
+      "model.language_model.layers.22.linear_attn.in_proj_a",
+      "model.language_model.layers.22.linear_attn.in_proj_b",
+      "model.language_model.layers.22.linear_attn.in_proj_qkv",
+      "model.language_model.layers.22.linear_attn.in_proj_z",
+      "model.language_model.layers.22.linear_attn.norm",
+      "model.language_model.layers.22.linear_attn.out_proj",
+      "model.language_model.layers.22.mlp.gate",
+      "model.language_model.layers.22.mlp.shared_expert.down_proj",
+      "model.language_model.layers.22.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.22.mlp.shared_expert.up_proj",
+      "model.language_model.layers.22.mlp.shared_expert_gate",
+      "model.language_model.layers.22.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.22.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.22.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.22.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.23.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.23.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.23.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.23.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.23.mlp.gate",
+      "model.language_model.layers.23.mlp.shared_expert.down_proj",
+      "model.language_model.layers.23.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.23.mlp.shared_expert.up_proj",
+      "model.language_model.layers.23.mlp.shared_expert_gate",
+      "model.language_model.layers.23.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.23.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.23.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.23.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.23.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.23.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.23.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.23.self_attn.k_norm",
+      "model.language_model.layers.23.self_attn.k_proj",
+      "model.language_model.layers.23.self_attn.o_proj",
+      "model.language_model.layers.23.self_attn.q_norm",
+      "model.language_model.layers.23.self_attn.q_proj",
+      "model.language_model.layers.23.self_attn.v_proj",
+      "model.language_model.layers.24.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.24.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.24.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.24.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.24.linear_attn.A_log",
+      "model.language_model.layers.24.linear_attn.conv1d",
+      "model.language_model.layers.24.linear_attn.dt_bias",
+      "model.language_model.layers.24.linear_attn.in_proj_a",
+      "model.language_model.layers.24.linear_attn.in_proj_b",
+      "model.language_model.layers.24.linear_attn.in_proj_qkv",
+      "model.language_model.layers.24.linear_attn.in_proj_z",
+      "model.language_model.layers.24.linear_attn.norm",
+      "model.language_model.layers.24.linear_attn.out_proj",
+      "model.language_model.layers.24.mlp.gate",
+      "model.language_model.layers.24.mlp.shared_expert.down_proj",
+      "model.language_model.layers.24.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.24.mlp.shared_expert.up_proj",
+      "model.language_model.layers.24.mlp.shared_expert_gate",
+      "model.language_model.layers.24.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.24.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.24.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.24.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.25.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.25.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.25.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.25.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.25.linear_attn.A_log",
+      "model.language_model.layers.25.linear_attn.conv1d",
+      "model.language_model.layers.25.linear_attn.dt_bias",
+      "model.language_model.layers.25.linear_attn.in_proj_a",
+      "model.language_model.layers.25.linear_attn.in_proj_b",
+      "model.language_model.layers.25.linear_attn.in_proj_qkv",
+      "model.language_model.layers.25.linear_attn.in_proj_z",
+      "model.language_model.layers.25.linear_attn.norm",
+      "model.language_model.layers.25.linear_attn.out_proj",
+      "model.language_model.layers.25.mlp.gate",
+      "model.language_model.layers.25.mlp.shared_expert.down_proj",
+      "model.language_model.layers.25.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.25.mlp.shared_expert.up_proj",
+      "model.language_model.layers.25.mlp.shared_expert_gate",
+      "model.language_model.layers.25.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.25.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.25.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.25.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.26.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.26.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.26.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.26.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.26.linear_attn.A_log",
+      "model.language_model.layers.26.linear_attn.conv1d",
+      "model.language_model.layers.26.linear_attn.dt_bias",
+      "model.language_model.layers.26.linear_attn.in_proj_a",
+      "model.language_model.layers.26.linear_attn.in_proj_b",
+      "model.language_model.layers.26.linear_attn.in_proj_qkv",
+      "model.language_model.layers.26.linear_attn.in_proj_z",
+      "model.language_model.layers.26.linear_attn.norm",
+      "model.language_model.layers.26.linear_attn.out_proj",
+      "model.language_model.layers.26.mlp.gate",
+      "model.language_model.layers.26.mlp.shared_expert.down_proj",
+      "model.language_model.layers.26.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.26.mlp.shared_expert.up_proj",
+      "model.language_model.layers.26.mlp.shared_expert_gate",
+      "model.language_model.layers.26.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.26.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.26.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.26.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.27.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.27.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.27.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.27.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.27.mlp.gate",
+      "model.language_model.layers.27.mlp.shared_expert.down_proj",
+      "model.language_model.layers.27.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.27.mlp.shared_expert.up_proj",
+      "model.language_model.layers.27.mlp.shared_expert_gate",
+      "model.language_model.layers.27.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.27.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.27.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.27.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.27.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.27.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.27.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.27.self_attn.k_norm",
+      "model.language_model.layers.27.self_attn.k_proj",
+      "model.language_model.layers.27.self_attn.o_proj",
+      "model.language_model.layers.27.self_attn.q_norm",
+      "model.language_model.layers.27.self_attn.q_proj",
+      "model.language_model.layers.27.self_attn.v_proj",
+      "model.language_model.layers.28.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.28.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.28.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.28.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.28.linear_attn.A_log",
+      "model.language_model.layers.28.linear_attn.conv1d",
+      "model.language_model.layers.28.linear_attn.dt_bias",
+      "model.language_model.layers.28.linear_attn.in_proj_a",
+      "model.language_model.layers.28.linear_attn.in_proj_b",
+      "model.language_model.layers.28.linear_attn.in_proj_qkv",
+      "model.language_model.layers.28.linear_attn.in_proj_z",
+      "model.language_model.layers.28.linear_attn.norm",
+      "model.language_model.layers.28.linear_attn.out_proj",
+      "model.language_model.layers.28.mlp.gate",
+      "model.language_model.layers.28.mlp.shared_expert.down_proj",
+      "model.language_model.layers.28.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.28.mlp.shared_expert.up_proj",
+      "model.language_model.layers.28.mlp.shared_expert_gate",
+      "model.language_model.layers.28.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.28.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.28.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.28.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.29.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.29.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.29.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.29.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.29.linear_attn.A_log",
+      "model.language_model.layers.29.linear_attn.conv1d",
+      "model.language_model.layers.29.linear_attn.dt_bias",
+      "model.language_model.layers.29.linear_attn.in_proj_a",
+      "model.language_model.layers.29.linear_attn.in_proj_b",
+      "model.language_model.layers.29.linear_attn.in_proj_qkv",
+      "model.language_model.layers.29.linear_attn.in_proj_z",
+      "model.language_model.layers.29.linear_attn.norm",
+      "model.language_model.layers.29.linear_attn.out_proj",
+      "model.language_model.layers.29.mlp.gate",
+      "model.language_model.layers.29.mlp.shared_expert.down_proj",
+      "model.language_model.layers.29.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.29.mlp.shared_expert.up_proj",
+      "model.language_model.layers.29.mlp.shared_expert_gate",
+      "model.language_model.layers.29.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.29.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.29.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.29.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.3.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.3.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.3.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.3.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.3.mlp.gate",
+      "model.language_model.layers.3.mlp.shared_expert.down_proj",
+      "model.language_model.layers.3.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.3.mlp.shared_expert.up_proj",
+      "model.language_model.layers.3.mlp.shared_expert_gate",
+      "model.language_model.layers.3.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.3.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.3.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.3.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.3.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.3.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.3.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.3.self_attn.k_norm",
+      "model.language_model.layers.3.self_attn.k_proj",
+      "model.language_model.layers.3.self_attn.o_proj",
+      "model.language_model.layers.3.self_attn.q_norm",
+      "model.language_model.layers.3.self_attn.q_proj",
+      "model.language_model.layers.3.self_attn.v_proj",
+      "model.language_model.layers.30.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.30.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.30.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.30.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.30.linear_attn.A_log",
+      "model.language_model.layers.30.linear_attn.conv1d",
+      "model.language_model.layers.30.linear_attn.dt_bias",
+      "model.language_model.layers.30.linear_attn.in_proj_a",
+      "model.language_model.layers.30.linear_attn.in_proj_b",
+      "model.language_model.layers.30.linear_attn.in_proj_qkv",
+      "model.language_model.layers.30.linear_attn.in_proj_z",
+      "model.language_model.layers.30.linear_attn.norm",
+      "model.language_model.layers.30.linear_attn.out_proj",
+      "model.language_model.layers.30.mlp.gate",
+      "model.language_model.layers.30.mlp.shared_expert.down_proj",
+      "model.language_model.layers.30.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.30.mlp.shared_expert.up_proj",
+      "model.language_model.layers.30.mlp.shared_expert_gate",
+      "model.language_model.layers.30.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.30.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.30.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.30.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.31.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.31.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.31.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.31.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.31.mlp.gate",
+      "model.language_model.layers.31.mlp.shared_expert.down_proj",
+      "model.language_model.layers.31.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.31.mlp.shared_expert.up_proj",
+      "model.language_model.layers.31.mlp.shared_expert_gate",
+      "model.language_model.layers.31.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.31.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.31.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.31.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.31.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.31.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.31.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.31.self_attn.k_norm",
+      "model.language_model.layers.31.self_attn.k_proj",
+      "model.language_model.layers.31.self_attn.o_proj",
+      "model.language_model.layers.31.self_attn.q_norm",
+      "model.language_model.layers.31.self_attn.q_proj",
+      "model.language_model.layers.31.self_attn.v_proj",
+      "model.language_model.layers.32.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.32.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.32.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.32.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.32.linear_attn.A_log",
+      "model.language_model.layers.32.linear_attn.conv1d",
+      "model.language_model.layers.32.linear_attn.dt_bias",
+      "model.language_model.layers.32.linear_attn.in_proj_a",
+      "model.language_model.layers.32.linear_attn.in_proj_b",
+      "model.language_model.layers.32.linear_attn.in_proj_qkv",
+      "model.language_model.layers.32.linear_attn.in_proj_z",
+      "model.language_model.layers.32.linear_attn.norm",
+      "model.language_model.layers.32.linear_attn.out_proj",
+      "model.language_model.layers.32.mlp.gate",
+      "model.language_model.layers.32.mlp.shared_expert.down_proj",
+      "model.language_model.layers.32.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.32.mlp.shared_expert.up_proj",
+      "model.language_model.layers.32.mlp.shared_expert_gate",
+      "model.language_model.layers.32.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.32.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.32.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.32.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.33.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.33.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.33.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.33.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.33.linear_attn.A_log",
+      "model.language_model.layers.33.linear_attn.conv1d",
+      "model.language_model.layers.33.linear_attn.dt_bias",
+      "model.language_model.layers.33.linear_attn.in_proj_a",
+      "model.language_model.layers.33.linear_attn.in_proj_b",
+      "model.language_model.layers.33.linear_attn.in_proj_qkv",
+      "model.language_model.layers.33.linear_attn.in_proj_z",
+      "model.language_model.layers.33.linear_attn.norm",
+      "model.language_model.layers.33.linear_attn.out_proj",
+      "model.language_model.layers.33.mlp.gate",
+      "model.language_model.layers.33.mlp.shared_expert.down_proj",
+      "model.language_model.layers.33.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.33.mlp.shared_expert.up_proj",
+      "model.language_model.layers.33.mlp.shared_expert_gate",
+      "model.language_model.layers.33.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.33.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.33.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.33.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.34.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.34.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.34.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.34.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.34.linear_attn.A_log",
+      "model.language_model.layers.34.linear_attn.conv1d",
+      "model.language_model.layers.34.linear_attn.dt_bias",
+      "model.language_model.layers.34.linear_attn.in_proj_a",
+      "model.language_model.layers.34.linear_attn.in_proj_b",
+      "model.language_model.layers.34.linear_attn.in_proj_qkv",
+      "model.language_model.layers.34.linear_attn.in_proj_z",
+      "model.language_model.layers.34.linear_attn.norm",
+      "model.language_model.layers.34.linear_attn.out_proj",
+      "model.language_model.layers.34.mlp.gate",
+      "model.language_model.layers.34.mlp.shared_expert.down_proj",
+      "model.language_model.layers.34.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.34.mlp.shared_expert.up_proj",
+      "model.language_model.layers.34.mlp.shared_expert_gate",
+      "model.language_model.layers.34.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.34.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.34.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.34.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.35.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.35.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.35.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.35.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.35.mlp.gate",
+      "model.language_model.layers.35.mlp.shared_expert.down_proj",
+      "model.language_model.layers.35.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.35.mlp.shared_expert.up_proj",
+      "model.language_model.layers.35.mlp.shared_expert_gate",
+      "model.language_model.layers.35.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.35.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.35.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.35.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.35.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.35.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.35.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.35.self_attn.k_norm",
+      "model.language_model.layers.35.self_attn.k_proj",
+      "model.language_model.layers.35.self_attn.o_proj",
+      "model.language_model.layers.35.self_attn.q_norm",
+      "model.language_model.layers.35.self_attn.q_proj",
+      "model.language_model.layers.35.self_attn.v_proj",
+      "model.language_model.layers.36.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.36.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.36.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.36.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.36.linear_attn.A_log",
+      "model.language_model.layers.36.linear_attn.conv1d",
+      "model.language_model.layers.36.linear_attn.dt_bias",
+      "model.language_model.layers.36.linear_attn.in_proj_a",
+      "model.language_model.layers.36.linear_attn.in_proj_b",
+      "model.language_model.layers.36.linear_attn.in_proj_qkv",
+      "model.language_model.layers.36.linear_attn.in_proj_z",
+      "model.language_model.layers.36.linear_attn.norm",
+      "model.language_model.layers.36.linear_attn.out_proj",
+      "model.language_model.layers.36.mlp.gate",
+      "model.language_model.layers.36.mlp.shared_expert.down_proj",
+      "model.language_model.layers.36.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.36.mlp.shared_expert.up_proj",
+      "model.language_model.layers.36.mlp.shared_expert_gate",
+      "model.language_model.layers.36.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.36.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.36.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.36.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.37.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.37.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.37.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.37.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.37.linear_attn.A_log",
+      "model.language_model.layers.37.linear_attn.conv1d",
+      "model.language_model.layers.37.linear_attn.dt_bias",
+      "model.language_model.layers.37.linear_attn.in_proj_a",
+      "model.language_model.layers.37.linear_attn.in_proj_b",
+      "model.language_model.layers.37.linear_attn.in_proj_qkv",
+      "model.language_model.layers.37.linear_attn.in_proj_z",
+      "model.language_model.layers.37.linear_attn.norm",
+      "model.language_model.layers.37.linear_attn.out_proj",
+      "model.language_model.layers.37.mlp.gate",
+      "model.language_model.layers.37.mlp.shared_expert.down_proj",
+      "model.language_model.layers.37.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.37.mlp.shared_expert.up_proj",
+      "model.language_model.layers.37.mlp.shared_expert_gate",
+      "model.language_model.layers.37.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.37.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.37.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.37.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.38.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.38.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.38.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.38.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.38.linear_attn.A_log",
+      "model.language_model.layers.38.linear_attn.conv1d",
+      "model.language_model.layers.38.linear_attn.dt_bias",
+      "model.language_model.layers.38.linear_attn.in_proj_a",
+      "model.language_model.layers.38.linear_attn.in_proj_b",
+      "model.language_model.layers.38.linear_attn.in_proj_qkv",
+      "model.language_model.layers.38.linear_attn.in_proj_z",
+      "model.language_model.layers.38.linear_attn.norm",
+      "model.language_model.layers.38.linear_attn.out_proj",
+      "model.language_model.layers.38.mlp.gate",
+      "model.language_model.layers.38.mlp.shared_expert.down_proj",
+      "model.language_model.layers.38.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.38.mlp.shared_expert.up_proj",
+      "model.language_model.layers.38.mlp.shared_expert_gate",
+      "model.language_model.layers.38.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.38.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.38.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.38.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.39.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.39.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.39.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.39.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.39.mlp.gate",
+      "model.language_model.layers.39.mlp.shared_expert.down_proj",
+      "model.language_model.layers.39.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.39.mlp.shared_expert.up_proj",
+      "model.language_model.layers.39.mlp.shared_expert_gate",
+      "model.language_model.layers.39.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.39.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.39.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.39.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.39.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.39.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.39.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.39.self_attn.k_norm",
+      "model.language_model.layers.39.self_attn.k_proj",
+      "model.language_model.layers.39.self_attn.o_proj",
+      "model.language_model.layers.39.self_attn.q_norm",
+      "model.language_model.layers.39.self_attn.q_proj",
+      "model.language_model.layers.39.self_attn.v_proj",
+      "model.language_model.layers.4.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.4.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.4.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.4.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.4.linear_attn.A_log",
+      "model.language_model.layers.4.linear_attn.conv1d",
+      "model.language_model.layers.4.linear_attn.dt_bias",
+      "model.language_model.layers.4.linear_attn.in_proj_a",
+      "model.language_model.layers.4.linear_attn.in_proj_b",
+      "model.language_model.layers.4.linear_attn.in_proj_qkv",
+      "model.language_model.layers.4.linear_attn.in_proj_z",
+      "model.language_model.layers.4.linear_attn.norm",
+      "model.language_model.layers.4.linear_attn.out_proj",
+      "model.language_model.layers.4.mlp.gate",
+      "model.language_model.layers.4.mlp.shared_expert.down_proj",
+      "model.language_model.layers.4.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.4.mlp.shared_expert.up_proj",
+      "model.language_model.layers.4.mlp.shared_expert_gate",
+      "model.language_model.layers.4.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.4.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.4.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.4.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.40.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.40.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.40.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.40.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.40.linear_attn.A_log",
+      "model.language_model.layers.40.linear_attn.conv1d",
+      "model.language_model.layers.40.linear_attn.dt_bias",
+      "model.language_model.layers.40.linear_attn.in_proj_a",
+      "model.language_model.layers.40.linear_attn.in_proj_b",
+      "model.language_model.layers.40.linear_attn.in_proj_qkv",
+      "model.language_model.layers.40.linear_attn.in_proj_z",
+      "model.language_model.layers.40.linear_attn.norm",
+      "model.language_model.layers.40.linear_attn.out_proj",
+      "model.language_model.layers.40.mlp.gate",
+      "model.language_model.layers.40.mlp.shared_expert.down_proj",
+      "model.language_model.layers.40.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.40.mlp.shared_expert.up_proj",
+      "model.language_model.layers.40.mlp.shared_expert_gate",
+      "model.language_model.layers.40.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.40.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.40.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.40.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.41.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.41.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.41.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.41.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.41.linear_attn.A_log",
+      "model.language_model.layers.41.linear_attn.conv1d",
+      "model.language_model.layers.41.linear_attn.dt_bias",
+      "model.language_model.layers.41.linear_attn.in_proj_a",
+      "model.language_model.layers.41.linear_attn.in_proj_b",
+      "model.language_model.layers.41.linear_attn.in_proj_qkv",
+      "model.language_model.layers.41.linear_attn.in_proj_z",
+      "model.language_model.layers.41.linear_attn.norm",
+      "model.language_model.layers.41.linear_attn.out_proj",
+      "model.language_model.layers.41.mlp.gate",
+      "model.language_model.layers.41.mlp.shared_expert.down_proj",
+      "model.language_model.layers.41.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.41.mlp.shared_expert.up_proj",
+      "model.language_model.layers.41.mlp.shared_expert_gate",
+      "model.language_model.layers.41.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.41.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.41.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.41.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.42.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.42.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.42.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.42.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.42.linear_attn.A_log",
+      "model.language_model.layers.42.linear_attn.conv1d",
+      "model.language_model.layers.42.linear_attn.dt_bias",
+      "model.language_model.layers.42.linear_attn.in_proj_a",
+      "model.language_model.layers.42.linear_attn.in_proj_b",
+      "model.language_model.layers.42.linear_attn.in_proj_qkv",
+      "model.language_model.layers.42.linear_attn.in_proj_z",
+      "model.language_model.layers.42.linear_attn.norm",
+      "model.language_model.layers.42.linear_attn.out_proj",
+      "model.language_model.layers.42.mlp.gate",
+      "model.language_model.layers.42.mlp.shared_expert.down_proj",
+      "model.language_model.layers.42.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.42.mlp.shared_expert.up_proj",
+      "model.language_model.layers.42.mlp.shared_expert_gate",
+      "model.language_model.layers.42.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.42.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.42.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.42.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.43.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.43.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.43.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.43.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.43.mlp.gate",
+      "model.language_model.layers.43.mlp.shared_expert.down_proj",
+      "model.language_model.layers.43.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.43.mlp.shared_expert.up_proj",
+      "model.language_model.layers.43.mlp.shared_expert_gate",
+      "model.language_model.layers.43.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.43.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.43.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.43.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.43.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.43.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.43.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.43.self_attn.k_norm",
+      "model.language_model.layers.43.self_attn.k_proj",
+      "model.language_model.layers.43.self_attn.o_proj",
+      "model.language_model.layers.43.self_attn.q_norm",
+      "model.language_model.layers.43.self_attn.q_proj",
+      "model.language_model.layers.43.self_attn.v_proj",
+      "model.language_model.layers.44.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.44.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.44.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.44.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.44.linear_attn.A_log",
+      "model.language_model.layers.44.linear_attn.conv1d",
+      "model.language_model.layers.44.linear_attn.dt_bias",
+      "model.language_model.layers.44.linear_attn.in_proj_a",
+      "model.language_model.layers.44.linear_attn.in_proj_b",
+      "model.language_model.layers.44.linear_attn.in_proj_qkv",
+      "model.language_model.layers.44.linear_attn.in_proj_z",
+      "model.language_model.layers.44.linear_attn.norm",
+      "model.language_model.layers.44.linear_attn.out_proj",
+      "model.language_model.layers.44.mlp.gate",
+      "model.language_model.layers.44.mlp.shared_expert.down_proj",
+      "model.language_model.layers.44.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.44.mlp.shared_expert.up_proj",
+      "model.language_model.layers.44.mlp.shared_expert_gate",
+      "model.language_model.layers.44.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.44.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.44.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.44.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.45.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.45.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.45.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.45.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.45.linear_attn.A_log",
+      "model.language_model.layers.45.linear_attn.conv1d",
+      "model.language_model.layers.45.linear_attn.dt_bias",
+      "model.language_model.layers.45.linear_attn.in_proj_a",
+      "model.language_model.layers.45.linear_attn.in_proj_b",
+      "model.language_model.layers.45.linear_attn.in_proj_qkv",
+      "model.language_model.layers.45.linear_attn.in_proj_z",
+      "model.language_model.layers.45.linear_attn.norm",
+      "model.language_model.layers.45.linear_attn.out_proj",
+      "model.language_model.layers.45.mlp.gate",
+      "model.language_model.layers.45.mlp.shared_expert.down_proj",
+      "model.language_model.layers.45.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.45.mlp.shared_expert.up_proj",
+      "model.language_model.layers.45.mlp.shared_expert_gate",
+      "model.language_model.layers.45.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.45.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.45.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.45.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.46.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.46.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.46.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.46.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.46.linear_attn.A_log",
+      "model.language_model.layers.46.linear_attn.conv1d",
+      "model.language_model.layers.46.linear_attn.dt_bias",
+      "model.language_model.layers.46.linear_attn.in_proj_a",
+      "model.language_model.layers.46.linear_attn.in_proj_b",
+      "model.language_model.layers.46.linear_attn.in_proj_qkv",
+      "model.language_model.layers.46.linear_attn.in_proj_z",
+      "model.language_model.layers.46.linear_attn.norm",
+      "model.language_model.layers.46.linear_attn.out_proj",
+      "model.language_model.layers.46.mlp.gate",
+      "model.language_model.layers.46.mlp.shared_expert.down_proj",
+      "model.language_model.layers.46.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.46.mlp.shared_expert.up_proj",
+      "model.language_model.layers.46.mlp.shared_expert_gate",
+      "model.language_model.layers.46.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.46.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.46.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.46.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.47.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.47.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.47.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.47.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.47.mlp.gate",
+      "model.language_model.layers.47.mlp.shared_expert.down_proj",
+      "model.language_model.layers.47.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.47.mlp.shared_expert.up_proj",
+      "model.language_model.layers.47.mlp.shared_expert_gate",
+      "model.language_model.layers.47.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.47.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.47.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.47.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.47.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.47.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.47.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.47.self_attn.k_norm",
+      "model.language_model.layers.47.self_attn.k_proj",
+      "model.language_model.layers.47.self_attn.o_proj",
+      "model.language_model.layers.47.self_attn.q_norm",
+      "model.language_model.layers.47.self_attn.q_proj",
+      "model.language_model.layers.47.self_attn.v_proj",
+      "model.language_model.layers.5.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.5.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.5.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.5.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.5.linear_attn.A_log",
+      "model.language_model.layers.5.linear_attn.conv1d",
+      "model.language_model.layers.5.linear_attn.dt_bias",
+      "model.language_model.layers.5.linear_attn.in_proj_a",
+      "model.language_model.layers.5.linear_attn.in_proj_b",
+      "model.language_model.layers.5.linear_attn.in_proj_qkv",
+      "model.language_model.layers.5.linear_attn.in_proj_z",
+      "model.language_model.layers.5.linear_attn.norm",
+      "model.language_model.layers.5.linear_attn.out_proj",
+      "model.language_model.layers.5.mlp.gate",
+      "model.language_model.layers.5.mlp.shared_expert.down_proj",
+      "model.language_model.layers.5.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.5.mlp.shared_expert.up_proj",
+      "model.language_model.layers.5.mlp.shared_expert_gate",
+      "model.language_model.layers.5.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.5.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.5.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.5.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.6.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.6.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.6.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.6.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.6.linear_attn.A_log",
+      "model.language_model.layers.6.linear_attn.conv1d",
+      "model.language_model.layers.6.linear_attn.dt_bias",
+      "model.language_model.layers.6.linear_attn.in_proj_a",
+      "model.language_model.layers.6.linear_attn.in_proj_b",
+      "model.language_model.layers.6.linear_attn.in_proj_qkv",
+      "model.language_model.layers.6.linear_attn.in_proj_z",
+      "model.language_model.layers.6.linear_attn.norm",
+      "model.language_model.layers.6.linear_attn.out_proj",
+      "model.language_model.layers.6.mlp.gate",
+      "model.language_model.layers.6.mlp.shared_expert.down_proj",
+      "model.language_model.layers.6.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.6.mlp.shared_expert.up_proj",
+      "model.language_model.layers.6.mlp.shared_expert_gate",
+      "model.language_model.layers.6.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.6.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.6.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.6.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.7.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.7.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.7.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.7.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.7.mlp.gate",
+      "model.language_model.layers.7.mlp.shared_expert.down_proj",
+      "model.language_model.layers.7.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.7.mlp.shared_expert.up_proj",
+      "model.language_model.layers.7.mlp.shared_expert_gate",
+      "model.language_model.layers.7.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.7.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.7.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.7.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.7.self_attn.indexer.index_qk_proj",
+      "model.language_model.layers.7.self_attn.indexer.k_layernorm",
+      "model.language_model.layers.7.self_attn.indexer.q_layernorm",
+      "model.language_model.layers.7.self_attn.k_norm",
+      "model.language_model.layers.7.self_attn.k_proj",
+      "model.language_model.layers.7.self_attn.o_proj",
+      "model.language_model.layers.7.self_attn.q_norm",
+      "model.language_model.layers.7.self_attn.q_proj",
+      "model.language_model.layers.7.self_attn.v_proj",
+      "model.language_model.layers.8.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.8.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.8.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.8.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.8.linear_attn.A_log",
+      "model.language_model.layers.8.linear_attn.conv1d",
+      "model.language_model.layers.8.linear_attn.dt_bias",
+      "model.language_model.layers.8.linear_attn.in_proj_a",
+      "model.language_model.layers.8.linear_attn.in_proj_b",
+      "model.language_model.layers.8.linear_attn.in_proj_qkv",
+      "model.language_model.layers.8.linear_attn.in_proj_z",
+      "model.language_model.layers.8.linear_attn.norm",
+      "model.language_model.layers.8.linear_attn.out_proj",
+      "model.language_model.layers.8.mlp.gate",
+      "model.language_model.layers.8.mlp.shared_expert.down_proj",
+      "model.language_model.layers.8.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.8.mlp.shared_expert.up_proj",
+      "model.language_model.layers.8.mlp.shared_expert_gate",
+      "model.language_model.layers.8.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.8.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.8.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.8.mlp_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.9.attn_hyper_connection.block_inject_weight",
+      "model.language_model.layers.9.attn_hyper_connection.hc_norm",
+      "model.language_model.layers.9.attn_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.9.attn_hyper_connection.input_mix_weight_up",
+      "model.language_model.layers.9.linear_attn.A_log",
+      "model.language_model.layers.9.linear_attn.conv1d",
+      "model.language_model.layers.9.linear_attn.dt_bias",
+      "model.language_model.layers.9.linear_attn.in_proj_a",
+      "model.language_model.layers.9.linear_attn.in_proj_b",
+      "model.language_model.layers.9.linear_attn.in_proj_qkv",
+      "model.language_model.layers.9.linear_attn.in_proj_z",
+      "model.language_model.layers.9.linear_attn.norm",
+      "model.language_model.layers.9.linear_attn.out_proj",
+      "model.language_model.layers.9.mlp.gate",
+      "model.language_model.layers.9.mlp.shared_expert.down_proj",
+      "model.language_model.layers.9.mlp.shared_expert.gate_proj",
+      "model.language_model.layers.9.mlp.shared_expert.up_proj",
+      "model.language_model.layers.9.mlp.shared_expert_gate",
+      "model.language_model.layers.9.mlp_hyper_connection.block_inject_weight",
+      "model.language_model.layers.9.mlp_hyper_connection.hc_norm",
+      "model.language_model.layers.9.mlp_hyper_connection.input_mix_weight_down",
+      "model.language_model.layers.9.mlp_hyper_connection.input_mix_weight_up",
+      "model.visual.blocks.0.attn.proj",
+      "model.visual.blocks.0.attn.qkv",
+      "model.visual.blocks.0.mlp.linear_fc1",
+      "model.visual.blocks.0.mlp.linear_fc2",
+      "model.visual.blocks.0.norm1",
+      "model.visual.blocks.0.norm2",
+      "model.visual.blocks.1.attn.proj",
+      "model.visual.blocks.1.attn.qkv",
+      "model.visual.blocks.1.mlp.linear_fc1",
+      "model.visual.blocks.1.mlp.linear_fc2",
+      "model.visual.blocks.1.norm1",
+      "model.visual.blocks.1.norm2",
+      "model.visual.blocks.10.attn.proj",
+      "model.visual.blocks.10.attn.qkv",
+      "model.visual.blocks.10.mlp.linear_fc1",
+      "model.visual.blocks.10.mlp.linear_fc2",
+      "model.visual.blocks.10.norm1",
+      "model.visual.blocks.10.norm2",
+      "model.visual.blocks.11.attn.proj",
+      "model.visual.blocks.11.attn.qkv",
+      "model.visual.blocks.11.mlp.linear_fc1",
+      "model.visual.blocks.11.mlp.linear_fc2",
+      "model.visual.blocks.11.norm1",
+      "model.visual.blocks.11.norm2",
+      "model.visual.blocks.12.attn.proj",
+      "model.visual.blocks.12.attn.qkv",
+      "model.visual.blocks.12.mlp.linear_fc1",
+      "model.visual.blocks.12.mlp.linear_fc2",
+      "model.visual.blocks.12.norm1",
+      "model.visual.blocks.12.norm2",
+      "model.visual.blocks.13.attn.proj",
+      "model.visual.blocks.13.attn.qkv",
+      "model.visual.blocks.13.mlp.linear_fc1",
+      "model.visual.blocks.13.mlp.linear_fc2",
+      "model.visual.blocks.13.norm1",
+      "model.visual.blocks.13.norm2",
+      "model.visual.blocks.14.attn.proj",
+      "model.visual.blocks.14.attn.qkv",
+      "model.visual.blocks.14.mlp.linear_fc1",
+      "model.visual.blocks.14.mlp.linear_fc2",
+      "model.visual.blocks.14.norm1",
+      "model.visual.blocks.14.norm2",
+      "model.visual.blocks.15.attn.proj",
+      "model.visual.blocks.15.attn.qkv",
+      "model.visual.blocks.15.mlp.linear_fc1",
+      "model.visual.blocks.15.mlp.linear_fc2",
+      "model.visual.blocks.15.norm1",
+      "model.visual.blocks.15.norm2",
+      "model.visual.blocks.16.attn.proj",
+      "model.visual.blocks.16.attn.qkv",
+      "model.visual.blocks.16.mlp.linear_fc1",
+      "model.visual.blocks.16.mlp.linear_fc2",
+      "model.visual.blocks.16.norm1",
+      "model.visual.blocks.16.norm2",
+      "model.visual.blocks.17.attn.proj",
+      "model.visual.blocks.17.attn.qkv",
+      "model.visual.blocks.17.mlp.linear_fc1",
+      "model.visual.blocks.17.mlp.linear_fc2",
+      "model.visual.blocks.17.norm1",
+      "model.visual.blocks.17.norm2",
+      "model.visual.blocks.18.attn.proj",
+      "model.visual.blocks.18.attn.qkv",
+      "model.visual.blocks.18.mlp.linear_fc1",
+      "model.visual.blocks.18.mlp.linear_fc2",
+      "model.visual.blocks.18.norm1",
+      "model.visual.blocks.18.norm2",
+      "model.visual.blocks.19.attn.proj",
+      "model.visual.blocks.19.attn.qkv",
+      "model.visual.blocks.19.mlp.linear_fc1",
+      "model.visual.blocks.19.mlp.linear_fc2",
+      "model.visual.blocks.19.norm1",
+      "model.visual.blocks.19.norm2",
+      "model.visual.blocks.2.attn.proj",
+      "model.visual.blocks.2.attn.qkv",
+      "model.visual.blocks.2.mlp.linear_fc1",
+      "model.visual.blocks.2.mlp.linear_fc2",
+      "model.visual.blocks.2.norm1",
+      "model.visual.blocks.2.norm2",
+      "model.visual.blocks.20.attn.proj",
+      "model.visual.blocks.20.attn.qkv",
+      "model.visual.blocks.20.mlp.linear_fc1",
+      "model.visual.blocks.20.mlp.linear_fc2",
+      "model.visual.blocks.20.norm1",
+      "model.visual.blocks.20.norm2",
+      "model.visual.blocks.21.attn.proj",
+      "model.visual.blocks.21.attn.qkv",
+      "model.visual.blocks.21.mlp.linear_fc1",
+      "model.visual.blocks.21.mlp.linear_fc2",
+      "model.visual.blocks.21.norm1",
+      "model.visual.blocks.21.norm2",
+      "model.visual.blocks.22.attn.proj",
+      "model.visual.blocks.22.attn.qkv",
+      "model.visual.blocks.22.mlp.linear_fc1",
+      "model.visual.blocks.22.mlp.linear_fc2",
+      "model.visual.blocks.22.norm1",
+      "model.visual.blocks.22.norm2",
+      "model.visual.blocks.23.attn.proj",
+      "model.visual.blocks.23.attn.qkv",
+      "model.visual.blocks.23.mlp.linear_fc1",
+      "model.visual.blocks.23.mlp.linear_fc2",
+      "model.visual.blocks.23.norm1",
+      "model.visual.blocks.23.norm2",
+      "model.visual.blocks.24.attn.proj",
+      "model.visual.blocks.24.attn.qkv",
+      "model.visual.blocks.24.mlp.linear_fc1",
+      "model.visual.blocks.24.mlp.linear_fc2",
+      "model.visual.blocks.24.norm1",
+      "model.visual.blocks.24.norm2",
+      "model.visual.blocks.25.attn.proj",
+      "model.visual.blocks.25.attn.qkv",
+      "model.visual.blocks.25.mlp.linear_fc1",
+      "model.visual.blocks.25.mlp.linear_fc2",
+      "model.visual.blocks.25.norm1",
+      "model.visual.blocks.25.norm2",
+      "model.visual.blocks.26.attn.proj",
+      "model.visual.blocks.26.attn.qkv",
+      "model.visual.blocks.26.mlp.linear_fc1",
+      "model.visual.blocks.26.mlp.linear_fc2",
+      "model.visual.blocks.26.norm1",
+      "model.visual.blocks.26.norm2",
+      "model.visual.blocks.3.attn.proj",
+      "model.visual.blocks.3.attn.qkv",
+      "model.visual.blocks.3.mlp.linear_fc1",
+      "model.visual.blocks.3.mlp.linear_fc2",
+      "model.visual.blocks.3.norm1",
+      "model.visual.blocks.3.norm2",
+      "model.visual.blocks.4.attn.proj",
+      "model.visual.blocks.4.attn.qkv",
+      "model.visual.blocks.4.mlp.linear_fc1",
+      "model.visual.blocks.4.mlp.linear_fc2",
+      "model.visual.blocks.4.norm1",
+      "model.visual.blocks.4.norm2",
+      "model.visual.blocks.5.attn.proj",
+      "model.visual.blocks.5.attn.qkv",
+      "model.visual.blocks.5.mlp.linear_fc1",
+      "model.visual.blocks.5.mlp.linear_fc2",
+      "model.visual.blocks.5.norm1",
+      "model.visual.blocks.5.norm2",
+      "model.visual.blocks.6.attn.proj",
+      "model.visual.blocks.6.attn.qkv",
+      "model.visual.blocks.6.mlp.linear_fc1",
+      "model.visual.blocks.6.mlp.linear_fc2",
+      "model.visual.blocks.6.norm1",
+      "model.visual.blocks.6.norm2",
+      "model.visual.blocks.7.attn.proj",
+      "model.visual.blocks.7.attn.qkv",
+      "model.visual.blocks.7.mlp.linear_fc1",
+      "model.visual.blocks.7.mlp.linear_fc2",
+      "model.visual.blocks.7.norm1",
+      "model.visual.blocks.7.norm2",
+      "model.visual.blocks.8.attn.proj",
+      "model.visual.blocks.8.attn.qkv",
+      "model.visual.blocks.8.mlp.linear_fc1",
+      "model.visual.blocks.8.mlp.linear_fc2",
+      "model.visual.blocks.8.norm1",
+      "model.visual.blocks.8.norm2",
+      "model.visual.blocks.9.attn.proj",
+      "model.visual.blocks.9.attn.qkv",
+      "model.visual.blocks.9.mlp.linear_fc1",
+      "model.visual.blocks.9.mlp.linear_fc2",
+      "model.visual.blocks.9.norm1",
+      "model.visual.blocks.9.norm2",
+      "model.visual.merger.linear_fc1",
+      "model.visual.merger.linear_fc2",
+      "model.visual.merger.norm",
+      "model.visual.patch_embed.proj",
+      "model.visual.pos_embed",
+      "mtp.fc_embedding",
+      "mtp.fc_hidden",
+      "mtp.hyper_connection_mixer.hc_norm",
+      "mtp.hyper_connection_mixer.input_mix_weight_down",
+      "mtp.hyper_connection_mixer.input_mix_weight_up",
+      "mtp.layers.0.attn_hyper_connection.block_inject_weight",
+      "mtp.layers.0.attn_hyper_connection.hc_norm",
+      "mtp.layers.0.attn_hyper_connection.input_mix_weight_down",
+      "mtp.layers.0.attn_hyper_connection.input_mix_weight_up",
+      "mtp.layers.0.mlp.gate",
+      "mtp.layers.0.mlp.shared_expert.down_proj",
+      "mtp.layers.0.mlp.shared_expert.gate_proj",
+      "mtp.layers.0.mlp.shared_expert.up_proj",
+      "mtp.layers.0.mlp.shared_expert_gate",
+      "mtp.layers.0.mlp_hyper_connection.block_inject_weight",
+      "mtp.layers.0.mlp_hyper_connection.hc_norm",
+      "mtp.layers.0.mlp_hyper_connection.input_mix_weight_down",
+      "mtp.layers.0.mlp_hyper_connection.input_mix_weight_up",
+      "mtp.layers.0.self_attn.indexer.index_qk_proj",
+      "mtp.layers.0.self_attn.indexer.k_layernorm",
+      "mtp.layers.0.self_attn.indexer.q_layernorm",
+      "mtp.layers.0.self_attn.k_norm",
+      "mtp.layers.0.self_attn.k_proj",
+      "mtp.layers.0.self_attn.o_proj",
+      "mtp.layers.0.self_attn.q_norm",
+      "mtp.layers.0.self_attn.q_proj",
+      "mtp.layers.0.self_attn.v_proj",
+      "mtp.pre_fc_norm_embedding",
+      "mtp.pre_fc_norm_hidden"
+    ],
+    "input_scale_calibration": "streaming_top_tail_4_5_6_proxy",
+    "input_scale_granularity": "per-tensor",
+    "input_scale_quantile": 0.9999,
+    "learned_rotation": false,
+    "learned_rotation_objective": null,
+    "mse_optimal_clipping": false,
+    "mtp_input_scale_k": 4,
+    "mtp_input_scale_quantile": 0.95,
+    "num_samples": 128,
+    "quant_algo": "NVFP4",
+    "quant_method": "modelopt",
+    "quarot_r1_fold": false,
+    "quarot_r1_fold_block_size": 32,
+    "rotate": false,
+    "scale_search": false,
+    "seq_len": 2048,
+    "smoothquant_alpha": null,
+    "smoothquant_dualscale": false,
+    "smoothquant_dualscale_round": 1,
+    "smoothquant_salient_ratio": 1.0,
+    "weight_scale_2_search_factors": null,
+    "with_input_scale": true
+  },
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 248044,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "hc_count": 4,
+    "hc_lowrank": 320,
+    "head_dim": 256,
+    "heads_per_ngram": 8,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "indexer_budget": 2048,
+    "indexer_compress_ratio": 4,
+    "indexer_head_dim": 128,
+    "indexer_kv_heads": 1,
+    "indexer_n_heads": 4,
+    "initializer_range": 0.02,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 48,
+    "linear_value_head_dim": 128,
+    "make_ngram_vocab_size_divisible_by": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "model_type": "qwen4_exp_text",
+    "moe_intermediate_size": 640,
+    "mtp": {
+      "hybrid": true,
+      "layer_types": [
+        "full_attention"
+      ],
+      "mtp_use_hidden_state_from_layer": null,
+      "num_hidden_layers": 1,
+      "rope_theta": 10000000
+    },
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "ngram_size": 3,
+    "ngram_vocab_size_base": 20000000,
+    "num_attention_heads": 24,
+    "num_experts": 512,
+    "num_experts_per_tok": 10,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 2,
+    "output_gate_type": "sigmoid",
+    "output_router_logits": false,
+    "pad_token_id": null,
+    "partial_rotary_factor": 0.25,
+    "ple_conv_kernel_size": 4,
+    "ple_embed_dim": 2560,
+    "ple_layer_ids": [
+      2
+    ],
+    "ple_quantization_config": {
+      "block_scale_dtype": "float8_e4m3fn",
+      "format": "qwen38-nvfp4-ple-v1",
+      "global_scale_dtype": "float32",
+      "group_size": 16,
+      "output_dtype": "bfloat16",
+      "packed_dtype": "uint8",
+      "packing": "low-nibble-first",
+      "value_dtype": "e2m1"
+    },
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "router_aux_loss_coef": 0.001,
+    "shared_expert_intermediate_size": 640,
+    "split_ngram_parts": 128,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "vocab_size": 248320
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.8.0.dev0",
+  "video_token_id": 248057,
+  "vision_config": {
+    "deepstack_visual_indexes": [],
+    "depth": 27,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen4_exp",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 2560,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 248054,
+  "vision_start_token_id": 248053
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+    "bos_token_id": 248044,
+    "do_sample": true,
+    "eos_token_id": [
+        248046,
+        248044
+    ],
+    "pad_token_id": 248044,
+    "temperature": 1.0,
+    "top_k": 20,
+    "top_p": 0.95
+}

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dea5f9ebc78eeac6f1c2751758e7365e3ba7fbeef39d723a9caa8206798767d0
+size 4980717000

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0883c2edbae8e2f0bf6240561536a9cb8059ba368e5209911deb1d704d3f24be
+size 4996346000

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:526f2738015ef06df7cd10f15cd811c7a49f6490b4da292f7820056173c252f9
+size 816384304

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f0003bf179800ac047e0247633a462626418ac186f8dda856dd4d732fad483a
+size 35742446

nvfp4_experts-00001-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f37b5ed37a7e598c3b125c5cf095f860896b0c3c070b8393487984c129fa0d52
+size 4249228128

nvfp4_experts-00002-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e89dace962c46e042f1bcd78889b1159762ac2c7d0ff7e179f68410826ac3796
+size 4249228128

nvfp4_experts-00003-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5043148ee50f5a04b3a8120f54c24ab67ae0863eeb047f6d2b3f30466bf4b5e9
+size 4249228128

nvfp4_experts-00004-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63523da74874b77aae413beea23741b0b974ad5f6b1b1e9e4d1fb2e8ed4363d5
+size 4249240416

nvfp4_experts-00005-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:963bfea776731902876d7fcb47d65fed7448404020f48c71d531e24138cece72
+size 4249246560

nvfp4_experts-00006-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecf4b4e6b50ed701bda0e2708bb983aa98fa98ee10196b0e4d6c3e79f700689f
+size 4249246560

nvfp4_experts-00007-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a06fc550d212d86e69846c9ed0fb92f0c927565d9ddd7bb669d8bb193d55bee1
+size 4249246560

nvfp4_experts-00008-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9f9de0f1de86159e9f1e5e4573e6375e6f786ec941d42d1533395071b24ff52
+size 4249246560

nvfp4_experts-00009-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:369c707829a36045fa6570b1a1e716d70fad2bfb3b3aac90adeccb980eeb470b
+size 4249246560

nvfp4_experts-00010-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dace02ef08e672d7646e95eee078b7268cd7644b849e28bfcdab3f66e6dec58a
+size 4249246560

nvfp4_experts-00011-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:351e24abcd554e790da4a523d73ca6fb0406e5af76a683de91989df4337a2685
+size 4249246560

nvfp4_experts-00012-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:459cc1a0e56b48f67aecd3d48039dd095e2a8805050b8bbc31cf44a9ca33f33c
+size 4249246560

nvfp4_experts-00013-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5af8d781f5077d6504ea58d3acb7ad1a14e8e8eae1bf77ab815cac3f44503701
+size 4249246560

nvfp4_experts-00014-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ec308f1e1bfc14f1759e6e059bb5a4876bbf4b7e0281ee9fc3f5b8ab135032b
+size 4249246560

nvfp4_experts-00015-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a33be5adf5d5a1ea7306ce47c00ebd60465b9e15afeeaee3f0bdd0b94fc080ab
+size 4249246560

nvfp4_experts-00016-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76fb6d93b3788ccd371a0b927c98e92e2fd8225e8ab3f356de1f60d7232c7249
+size 4249246560

nvfp4_experts_mtp.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d44e6d705d2313c713e60114e56874adf358ed5f646dc8704bb5be15f5ddbf7
+size 1597441088

ple-nvfp4-00001-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3637f1778cafefb2392dfbe824b8152ee08b5fd8e3f3a0006b3fb9d2bacbcd92
+size 225001588

ple-nvfp4-00002-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34c38184c7f0eae8e515d753f584e17e2fab70d63e630a9a999c72c75a97ed43
+size 225001588

ple-nvfp4-00003-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a3d2fc69b4a39157557531e25e8bf6d16c86e7c2ce448cc2be2d03b3b2d9b40
+size 225001588

ple-nvfp4-00004-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7e9478e689a6f867a0930986f89fbce50a2ff2ff519c34dff94d77a8d5a0602
+size 225001588

ple-nvfp4-00005-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a088caee78e57c8c6883b548a9dbd28a8d2c68d620b7a4eef6abeb988f995d4
+size 225001588

ple-nvfp4-00006-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68a251dcdbe0dc7e53b07fd1809e98c62d789514ab2a0b1b98fb06056ed9c157
+size 225001588

ple-nvfp4-00007-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f09a309d7bf8040bc7ae4593474fa95c4e456207971058473a4482ddd99d9b40
+size 225001588

ple-nvfp4-00008-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:653f1e2f375c6a694f37cca0bd951f47a008d6ef5cb432d547f821ebc29a4e35
+size 225001588

ple-nvfp4-00009-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc17299303dd93a37e98a546b48e09002184814254c6da1bbede3a1bd17ef728
+size 225001588

ple-nvfp4-00010-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f3a52379c164519928def1d163f89f4f4fb642dcd96a0f113e8a14077f56ade
+size 225001588

ple-nvfp4-00011-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e717777234b38a441b6e57f7a4b3285f7b0a1d7ed12346d9836d6ddecad6b530
+size 225001596

ple-nvfp4-00012-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efd2d836751b2b8b8c3e5462fcfc16758326522007893b729f8a126d8c872372
+size 225001596

ple-nvfp4-00013-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:734d1c62b47ebdffffa634ea3481f7fc7ae5becf889b1497669191f103ed7e44
+size 225001596

ple-nvfp4-00014-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aaf535afcd2aafc17676e0996d9d3fab3d0bc72731a8ad95d89a4a6bfecf25fa
+size 225001596

ple-nvfp4-00015-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5714fe9b0c7a82b8cd70f7af047c5eb7d5d1352b150645e07dd8693818437189
+size 225001596

ple-nvfp4-00016-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0c4857ab6c01f346614876847bb4415509d476831306a285e887e9fc7ab2135
+size 225001596

ple-nvfp4-00017-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:648af85aacced6f5aaf4001054e5bb00972da781202bd6a1134327b337a99a7d
+size 225001596

ple-nvfp4-00018-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48f611887c887eaef0b0cb230938869e6c5e9e860e099174dfb78fdd59dc7a81
+size 225001596

ple-nvfp4-00019-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e3ba8ae09cb8322a83a4dfccee18cd5639c74cd317875868d1ab186fa7b4caa
+size 225001596

ple-nvfp4-00020-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07ecf05fa9dfcad38e27e64c319ace82e4492ec93eaa4335cdc9944eaad2809e
+size 225001596

ple-nvfp4-00021-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71cd949bacc574838b27b19d9f212c2e9b3b25af6666bdaeacc4985edd607170
+size 225001596

ple-nvfp4-00022-of-00128.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c445dfb3a4717620df6b979aa6d47f3280ab0d02ae2ef5d21df512939d11973
+size 225001596