File size: 5,595 Bytes
f66e612 51b541a f66e612 51b541a f66e612 51b541a f66e612 51b541a f66e612 51b541a f66e612 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | ---
library_name: diffusers
license: apache-2.0
base_model:
- MiniMaxAI/MiniMax-H3
- lightx2v/LightVAE
pipeline_tag: image-to-video
tags:
- minimax-h3
- video
- modular-diffusers
- lightvae
---
# MiniMax-H3 + LightVAE (modular diffusers)
A ready-to-load **modular diffusers** repo for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) in which the
video VAE **decoder** is replaced by [lightx2v/LightVAE](https://huggingface.co/lightx2v/LightVAE)'s block-pruned Lite
decoder (ViT decoder pruned from 36 to 12 transformer blocks). LightVAE preserves H3's latent interface
(24-channel latents, same spatial/temporal compression, same `latents_mean/std`), so the transformer, the audio VAE,
and the scheduler are untouched and the decoder is a drop-in replacement.
Only the `vae` component points here; every other component (`transformer`, `text_encoder`, `audio_vae`, ...) is
loaded from the H3 base and pruned repos through this repo's `modular_model_index.json`. The included VAE also keeps
the **stock H3 encoder** (encode is cheap and leaves the latent interface untouched); only the heavy decode path is
swapped.
## Contents
```
modular_model_index.json # vae -> this repo/vae ; everything else -> MiniMax-H3 / MiniMax-H3-Pruned
vae/ # AutoencoderKLMiniMaxH3, decoder_num_layers=12 (stock encoder + LightVAE Lite decoder)
```
## Usage
Requires `diffusers` with the MiniMax-H3 integration.
Two things to know up front, both inherent to a stand-alone modular index (verified on a live Space):
1. The LightVAE `vae` loads automatically from this repo. The **pruned transformer is remote code in a different
repo**, and `trust_remote_code` is only forwarded to components that live in this index's own repo, so
`transformer_ref` has to be loaded explicitly and injected with `update_components`.
2. Components load on CPU; move the model components to the GPU before calling the pipeline.
```python
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from diffusers import ComponentsManager, AutoModel
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Blocks
from diffusers.modular_pipelines.minimax_h3.references import MiniMaxH3VideoReference
# ref2va workflow: loads only the reference transformer partition (not both)
blocks = MiniMaxH3Blocks(workflow="ref2va")
pipe = blocks.init_pipeline(
"diffusers-modular/MiniMax-H3-LightVAE",
components_manager=ComponentsManager(),
collection="h3",
)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True) # LightVAE `vae` loads here
# the remote-code pruned transformer lives in a different repo, so load + inject it explicitly
if getattr(pipe, "transformer_ref", None) is None:
transformer_ref = AutoModel.from_pretrained(
"multimodalart/MiniMax-H3-Pruned", subfolder="transformer_ref",
trust_remote_code=True, torch_dtype=torch.bfloat16,
)
pipe.update_components(transformer_ref=transformer_ref)
# both autoencoders stay fp32 on the native attention backend
pipe.vae.set_attention_backend("native")
pipe.audio_vae.set_attention_backend("native")
# lightx2v 4-step Ref2V Turbo LoRA. Its keys target the transformer submodules with no `transformer.` prefix,
# so load it onto the component with prefix=None (and drop the PEFT `.default.` adapter segment from the keys).
turbo = load_file(hf_hub_download("lightx2v/Minimax-h3-Turbo", "minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors"))
turbo = {k.replace(".lora_A.default.weight", ".lora_A.weight").replace(".lora_B.default.weight", ".lora_B.weight"): v
for k, v in turbo.items()}
pipe.transformer_ref.load_lora_adapter(turbo, prefix=None, adapter_name="turbo")
pipe.transformer_ref.set_adapters(["turbo"], [1.0])
# components load on CPU; move the models to the GPU (VAEs keep fp32)
for name in ("transformer_ref", "vae", "audio_vae"):
getattr(pipe, name).to("cuda")
state = pipe(
prompt="a red fox walking through snow, cinematic",
references=[MiniMaxH3VideoReference.from_file("reference.mp4")],
height=768,
width=768,
num_frames=124, # must be 17*n + 5
num_inference_steps=4, # 4 with the Turbo LoRA; ~20 without
generator=torch.Generator("cpu").manual_seed(0),
)
frames = state.get("videos")[0] # list of PIL frames
audio = state.get("audio") # waveform (or None)
sampling_rate = state.get("sampling_rate")
```
Swapping back to the stock decoder is just pointing `vae` at `MiniMaxAI/MiniMax-H3` / `vae` in the index.
## How it was built
`vae/` is an `AutoencoderKLMiniMaxH3` instantiated with `decoder_num_layers=12`, loaded with:
- `encoder.*` + `quant_conv.*` from `MiniMaxAI/MiniMax-H3/vae` (stock H3 encoder),
- `decoder.*` + `post_quant_conv.*` from `lightx2v/LightVAE/lightvae-lite-h3-decoder.safetensors`.
`latents_mean/std` are bit-identical between the stock VAE config and the LightVAE decoder checkpoint, confirming the
shared latent interface. State dict loads strict (0 missing, 0 unexpected).
## Roadmap
- **Lite encoder.** LightVAE also ships a Lite *encoder* (`lightvae-lite-h3-encoder.safetensors`, non-uniform resnet
depth `[1,1,1,1,1,2]`). Loading it needs `AutoencoderKLMiniMaxH3` to accept a per-level `layers_per_block`; tracked
as a follow-up.
- **int8 decoder.** `lightvae-lite-h3-decoder-int8-convrot.safetensors` for a cheaper decode once its kernel path is wired.
## Credits
LightVAE by [lightx2v](https://huggingface.co/lightx2v/LightVAE). MiniMax-H3 by [MiniMaxAI](https://huggingface.co/MiniMaxAI/MiniMax-H3).
|