---
license: other
license_name: kimi-k3
license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
language:
- en
tags:
- vision
- feature-extraction
- moonvit
- kimi-k3
pipeline_tag: image-feature-extraction
library_name: transformers
base_model:
- moonshotai/Kimi-K3
---

# MoonViT-V2 (Standalone)

Standalone extraction of the **MoonViT-V2** vision encoder (and the Kimi K3 multimodal projector) from [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3), so the vision tower can be used **without downloading the full 2.8T-parameter model**.

## Why this exists

Kimi K3 ships as a single 96-shard, ~1.5 TB checkpoint whose vision subsystem coexists with the language model weights. For projects that only need the vision encoder — e.g. training a small **vision→LLM projector** on a frozen text backbone (adapter / vision-bridge / LLaVA-style grafting) — pulling the entire K3 MoE is wasteful and, on modest GPUs, impossible. The vision subsystem lives entirely in the **two tail shards** of the checkpoint, so it can be extracted losslessly.

## Contents

| File | Module | Tensors | Params | Dtype |
|------|--------|---------|--------|-------|
| `moonvit_v2.safetensors` | MoonViT-V2 encoder (`vision_tower.*`) | 165 | **401.2 M** | BF16 |
| `kimi_mm_projector.safetensors` | K3 multimodal projector (`mm_projector.*`) | 3 | 46.1 M | BF16 |
| `vision_config.json` | `vision_config` verbatim from K3 | — | — | — |

Keys are kept under their canonical Kimi prefixes (`vision_tower.*`, `mm_projector.*`) — byte-identical to the source shards, so state loaders bind with no rename logic.

## Architecture (from `vision_config.json`)

```
patch_size:            14
merge_kernel_size:     [2, 2]      # 2x2 spatial merge (PatchMergerV2)
merge_type:            sd2_tpool
vt_hidden_size:        1024        # per-patch output dim
qkv_hidden_size:       1536
intermediate_size:     4096
num_hidden_layers:     27
num_attention_heads:   12
activation:            gelu_pytorch_tanh
norm:                  rmsnorm
pos_emb:               divided_fixed (init 64x64 grid, bilinear interp)
attn_implementation:   flash_attention_2
text_hidden_size:      7168        # K3 LLM width (NOT this encoder's width)
mm_projector_type:     patchmergerv2
```

**Embedding contract (the part adapters depend on):**

- Per-patch encoder output = **1024**-dim.
- The 2×2 PatchMerger concatenates 4 neighbouring patches → a **4096**-dim visual token
  (`1024 * 2 * 2`). This 4096 vector is the input any downstream projector consumes.
- Image-token count after merge ≈ `ceil(H/28) * ceil(W/28)` (28 = patch 14 × merge 2).

The bundled `kimi_mm_projector` is K3's *own* 4096 → 7168 mapping into the K3 LLM
embedding space. When grafting onto a **different** LLM, train a **new** projector
sized to that LLM's hidden width (e.g. 4096 → 4096 for a 4096-hidden model) — do
not reuse `kimi_mm_projector`, which is K3-specific.

## Usage

```python
import json, torch
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

REPO = "keypa/MoonViT-V2-Standalone"
sd    = load_file(hf_hub_download(REPO, "moonvit_v2.safetensors"))   # encoder
cfg   = json.loads(hf_hub_download(REPO, "vision_config.json"))
proj  = load_file(hf_hub_download(REPO, "kimi_mm_projector.safetensors"))  # optional

# Build a MoonViT module matching cfg, then:
#   missing, unexpected = model.load_state_dict(sd, strict=True)
# Preprocess to patch-14 grid, forward, then apply the 2x2 PatchMerger reshape.
#   feats: [num_merged_tokens, 1024] -> merge -> [num_merged_tokens/4 * ... , 4096]
```

A reference loader module (`moonvit.py`) reproducing the encoder exactly is
recommended before training; verify forward parity against the source repo's
`trust_remote_code` implementation on identical inputs.

## Extraction method

1. Read `model.safetensors.index.json` from K3 (weight_map).
2. Vision tensors (`vision_tower.*`, `mm_projector.*`) map to shards
   `model-00095-of-000096.safetensors` and `model-00096-of-000096.safetensors` only (~895 MB).
3. Stream-download those two shards, carve the prefixes (names unchanged), save standalone.
No weight is modified, requantized, or rescaled — byte-identical to the upstream tensors.

## Intended use & caveats

- Vision **feature extraction** and as a frozen encoder for projector/adapter training.
- This is an encoder **only** — it has no text head and cannot generate or answer.
- Preprocessing parity (normalization, resolution tiling) lives in K3's
  `trust_remote_code` and must be reproduced for correct results.
- Not independently benchmarked here; treat as a research artifact.

## License

Upstream **`moonshotai/Kimi-K3` License** ("kimi-k3"). This is a derivative work of the
K3 weights; **all K3 license conditions apply**, including Moonshot AI's commercial
thresholds (e.g. Model-as-a-Service revenue / monthly-active-user limits). Read the
[full K3 license](https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE) before
commercial use. This model card **summarizes but does not replace** that license.

## Citation

```bibtex
@misc{moonshot2025kimik3,
  title  = {Kimi K3},
  author = {Moonshot AI},
  year   = {2026},
  url    = {https://huggingface.co/moonshotai/Kimi-K3}
}
```