MiniMax-H3 + LightVAE lite decoder (modular diffusers)

README.md

@@ -0,0 +1,101 @@
+---
+library_name: diffusers
+license: apache-2.0
+base_model:
+- MiniMaxAI/MiniMax-H3
+- lightx2v/LightVAE
+pipeline_tag: image-to-video
+tags:
+- minimax-h3
+- video
+- modular-diffusers
+- lightvae
+---
+
+# MiniMax-H3 + LightVAE (modular diffusers)
+
+A ready-to-load **modular diffusers** repo for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) in which the
+video VAE **decoder** is replaced by [lightx2v/LightVAE](https://huggingface.co/lightx2v/LightVAE)'s block-pruned Lite
+decoder (ViT decoder pruned from 36 to 12 transformer blocks). LightVAE preserves H3's latent interface
+(24-channel latents, same spatial/temporal compression, same `latents_mean/std`), so the transformer, the audio VAE,
+and the scheduler are untouched and the decoder is a drop-in replacement.
+
+Only the `vae` component points here; every other component (`transformer`, `text_encoder`, `audio_vae`, ...) is
+loaded from the H3 base and pruned repos through this repo's `modular_model_index.json`. The included VAE also keeps
+the **stock H3 encoder** (encode is cheap and leaves the latent interface untouched); only the heavy decode path is
+swapped.
+
+## Contents
+
+```
+modular_model_index.json     # vae -> this repo/vae ; everything else -> MiniMax-H3 / MiniMax-H3-Pruned
+vae/                         # AutoencoderKLMiniMaxH3, decoder_num_layers=12 (stock encoder + LightVAE Lite decoder)
+```
+
+## Usage
+
+Requires `diffusers` with the MiniMax-H3 integration.
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+from diffusers import ComponentsManager
+from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Blocks
+from diffusers.modular_pipelines.minimax_h3.references import MiniMaxH3VideoReference
+
+# ref2va workflow: loads only the reference transformer partition (not both)
+blocks = MiniMaxH3Blocks(workflow="ref2va")
+pipe = blocks.init_pipeline(
+    "diffusers-modular/MiniMax-H3-LightVAE",
+    components_manager=ComponentsManager(),
+    collection="h3",
+)
+pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
+
+# both autoencoders stay fp32 on the native attention backend
+pipe.vae.set_attention_backend("native")
+pipe.audio_vae.set_attention_backend("native")
+
+# optional: lightx2v 4-step Ref2V Turbo LoRA (on the transformer; independent of the VAE)
+pipe.load_lora_weights(
+    hf_hub_download("lightx2v/Minimax-h3-Turbo", "minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors"),
+    adapter_name="turbo",
+)
+pipe.set_adapters(["turbo"], [1.0])
+
+state = pipe(
+    prompt="a red fox walking through snow, cinematic",
+    references=[MiniMaxH3VideoReference.from_file("reference.mp4")],
+    height=448,
+    width=768,
+    num_frames=124,          # must be 17*n + 5
+    num_inference_steps=4,   # 4 with the Turbo LoRA; ~20 without
+    generator=torch.Generator("cpu").manual_seed(0),
+)
+
+frames = state.get("videos")[0]          # list of PIL frames
+audio = state.get("audio")               # waveform (or None)
+sampling_rate = state.get("sampling_rate")
+```
+
+Swapping back to the stock decoder is just pointing `vae` at `MiniMaxAI/MiniMax-H3` / `vae` in the index.
+
+## How it was built
+
+`vae/` is an `AutoencoderKLMiniMaxH3` instantiated with `decoder_num_layers=12`, loaded with:
+- `encoder.*` + `quant_conv.*` from `MiniMaxAI/MiniMax-H3/vae` (stock H3 encoder),
+- `decoder.*` + `post_quant_conv.*` from `lightx2v/LightVAE/lightvae-lite-h3-decoder.safetensors`.
+
+`latents_mean/std` are bit-identical between the stock VAE config and the LightVAE decoder checkpoint, confirming the
+shared latent interface. State dict loads strict (0 missing, 0 unexpected).
+
+## Roadmap
+
+- **Lite encoder.** LightVAE also ships a Lite *encoder* (`lightvae-lite-h3-encoder.safetensors`, non-uniform resnet
+  depth `[1,1,1,1,1,2]`). Loading it needs `AutoencoderKLMiniMaxH3` to accept a per-level `layers_per_block`; tracked
+  as a follow-up.
+- **int8 decoder.** `lightvae-lite-h3-decoder-int8-convrot.safetensors` for a cheaper decode once its kernel path is wired.
+
+## Credits
+
+LightVAE by [lightx2v](https://huggingface.co/lightx2v/LightVAE). MiniMax-H3 by [MiniMaxAI](https://huggingface.co/MiniMaxAI/MiniMax-H3).

modular_model_index.json

@@ -0,0 +1,131 @@
+{
+  "_class_name": "MiniMaxH3ModularPipeline",
+  "_diffusers_version": "0.40.0.dev0",
+  "_blocks_class_name": "MiniMaxH3Blocks",
+  "text_encoder": [
+    "transformers",
+    "Qwen3VLForConditionalGeneration",
+    {
+      "type_hint": [
+        "transformers",
+        "Qwen3VLForConditionalGeneration"
+      ],
+      "pretrained_model_name_or_path": "multimodalart/MiniMax-H3-Pruned",
+      "subfolder": "text_encoder",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "tokenizer": [
+    "transformers",
+    "Qwen2TokenizerFast",
+    {
+      "type_hint": [
+        "transformers",
+        "Qwen2TokenizerFast"
+      ],
+      "pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
+      "subfolder": "tokenizer",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "processor": [
+    "transformers",
+    "Qwen3VLProcessor",
+    {
+      "type_hint": [
+        "transformers",
+        "Qwen3VLProcessor"
+      ],
+      "pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
+      "subfolder": "processor",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKLMiniMaxH3",
+    {
+      "type_hint": [
+        "diffusers",
+        "AutoencoderKLMiniMaxH3"
+      ],
+      "pretrained_model_name_or_path": "diffusers-modular/MiniMax-H3-LightVAE",
+      "subfolder": "vae",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "audio_vae": [
+    "diffusers",
+    "AutoencoderKLMiniMaxH3Audio",
+    {
+      "type_hint": [
+        "diffusers",
+        "AutoencoderKLMiniMaxH3Audio"
+      ],
+      "pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
+      "subfolder": "audio_vae",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "transformer": [
+    "diffusers",
+    "AutoModel",
+    {
+      "type_hint": [
+        "diffusers",
+        "AutoModel"
+      ],
+      "pretrained_model_name_or_path": "multimodalart/MiniMax-H3-Pruned",
+      "subfolder": "transformer",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "transformer_ref": [
+    "diffusers",
+    "AutoModel",
+    {
+      "type_hint": [
+        "diffusers",
+        "AutoModel"
+      ],
+      "pretrained_model_name_or_path": "multimodalart/MiniMax-H3-Pruned",
+      "subfolder": "transformer_ref",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "scheduler": [
+    "diffusers",
+    "MiniMaxH3Scheduler",
+    {
+      "type_hint": [
+        "diffusers",
+        "MiniMaxH3Scheduler"
+      ],
+      "pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
+      "subfolder": "scheduler",
+      "variant": null,
+      "revision": null
+    }
+  ],
+  "audio_scheduler": [
+    "diffusers",
+    "MiniMaxH3Scheduler",
+    {
+      "type_hint": [
+        "diffusers",
+        "MiniMaxH3Scheduler"
+      ],
+      "pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
+      "subfolder": "audio_scheduler",
+      "variant": null,
+      "revision": null
+    }
+  ]
+}

vae/config.json

@@ -0,0 +1,97 @@
+{
+  "_class_name": "AutoencoderKLMiniMaxH3",
+  "_diffusers_version": "0.40.0.dev0",
+  "block_out_channels": [
+    128,
+    256,
+    256,
+    512,
+    512,
+    1024
+  ],
+  "clip_length": 17,
+  "decoder_attention_head_dim": 64,
+  "decoder_ffn_mult": 4,
+  "decoder_norm_eps": 1e-05,
+  "decoder_num_attention_heads": 32,
+  "decoder_num_layers": 12,
+  "decoder_num_register_tokens": 4,
+  "decoder_rope_dim_ratio": 0.75,
+  "decoder_rope_theta": 100.0,
+  "in_channels": 3,
+  "latent_channels": 24,
+  "latents_mean": [
+    0.858090341091156,
+    -0.9606591463088989,
+    1.0661640167236328,
+    -0.5090325474739075,
+    -0.2727581858634949,
+    -1.3675414323806763,
+    -0.2553254961967468,
+    -0.26907554268836975,
+    -0.5376840829849243,
+    -0.0464097298681736,
+    0.6657370328903198,
+    0.19690127670764923,
+    -0.5460608005523682,
+    -0.4035342037677765,
+    -0.23683024942874908,
+    0.25928452610969543,
+    -0.30133944749832153,
+    0.211341992020607,
+    -1.1206848621368408,
+    0.3581933379173279,
+    -0.04225143790245056,
+    0.2604829967021942,
+    0.22864092886447906,
+    0.7056031823158264
+  ],
+  "latents_std": [
+    1.2223774194717407,
+    1.2767263650894165,
+    1.6831774711608887,
+    1.7549455165863037,
+    1.5636216402053833,
+    2.194143533706665,
+    0.9653137922286987,
+    1.0569885969161987,
+    0.841948926448822,
+    0.7729952931404114,
+    1.8955937623977661,
+    0.946841835975647,
+    0.7996809482574463,
+    0.44988900423049927,
+    0.7197399735450745,
+    0.6936293244361877,
+    2.961095094680786,
+    2.7694199085235596,
+    3.0496184825897217,
+    2.1088054180145264,
+    3.276226282119751,
+    3.1627357006073,
+    2.2816812992095947,
+    2.6127843856811523
+  ],
+  "layers_per_block": 2,
+  "norm_eps": 1e-06,
+  "norm_num_groups": 32,
+  "out_channels": 3,
+  "spatial_downsample_factors": [
+    2,
+    2,
+    2,
+    2,
+    1,
+    1
+  ],
+  "spatial_padding_mode": "reflect",
+  "temporal_downsample_factors": [
+    1,
+    2,
+    2,
+    1,
+    1,
+    1
+  ],
+  "token_drop": 3
+}

vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1c69d47b24b0d5f8bbfba28bb04e93b4c16a3db1c4928e8b94dd7c4bfa3bf1d
+size 3969719952