diffusers-modular/MiniMax-H3-LightVAE

MiniMax-H3 + LightVAE (modular diffusers)

A ready-to-load modular diffusers repo for MiniMax-H3 in which the video VAE decoder is replaced by lightx2v/LightVAE's block-pruned Lite decoder (ViT decoder pruned from 36 to 12 transformer blocks). LightVAE preserves H3's latent interface (24-channel latents, same spatial/temporal compression, same latents_mean/std), so the transformer, the audio VAE, and the scheduler are untouched and the decoder is a drop-in replacement.

Only the vae component points here; every other component (transformer, text_encoder, audio_vae, ...) is loaded from the H3 base and pruned repos through this repo's modular_model_index.json. The included VAE also keeps the stock H3 encoder (encode is cheap and leaves the latent interface untouched); only the heavy decode path is swapped.

Contents

modular_model_index.json     # vae -> this repo/vae ; everything else -> MiniMax-H3 / MiniMax-H3-Pruned
vae/                         # AutoencoderKLMiniMaxH3, decoder_num_layers=12 (stock encoder + LightVAE Lite decoder)

Usage

Requires diffusers with the MiniMax-H3 integration.

Two things to know up front, both inherent to a stand-alone modular index (verified on a live Space):

1. The LightVAE vae loads automatically from this repo. The pruned transformer is remote code in a different repo, and trust_remote_code is only forwarded to components that live in this index's own repo, so transformer_ref has to be loaded explicitly and injected with update_components.
2. Components load on CPU; move the model components to the GPU before calling the pipeline.

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from diffusers import ComponentsManager, AutoModel
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Blocks
from diffusers.modular_pipelines.minimax_h3.references import MiniMaxH3VideoReference

# ref2va workflow: loads only the reference transformer partition (not both)
blocks = MiniMaxH3Blocks(workflow="ref2va")
pipe = blocks.init_pipeline(
    "diffusers-modular/MiniMax-H3-LightVAE",
    components_manager=ComponentsManager(),
    collection="h3",
)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)  # LightVAE `vae` loads here

# the remote-code pruned transformer lives in a different repo, so load + inject it explicitly
if getattr(pipe, "transformer_ref", None) is None:
    transformer_ref = AutoModel.from_pretrained(
        "multimodalart/MiniMax-H3-Pruned", subfolder="transformer_ref",
        trust_remote_code=True, torch_dtype=torch.bfloat16,
    )
    pipe.update_components(transformer_ref=transformer_ref)

# both autoencoders stay fp32 on the native attention backend
pipe.vae.set_attention_backend("native")
pipe.audio_vae.set_attention_backend("native")

# lightx2v 4-step Ref2V Turbo LoRA. Its keys target the transformer submodules with no `transformer.` prefix,
# so load it onto the component with prefix=None (and drop the PEFT `.default.` adapter segment from the keys).
turbo = load_file(hf_hub_download("lightx2v/Minimax-h3-Turbo", "minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors"))
turbo = {k.replace(".lora_A.default.weight", ".lora_A.weight").replace(".lora_B.default.weight", ".lora_B.weight"): v
         for k, v in turbo.items()}
pipe.transformer_ref.load_lora_adapter(turbo, prefix=None, adapter_name="turbo")
pipe.transformer_ref.set_adapters(["turbo"], [1.0])

# components load on CPU; move the models to the GPU (VAEs keep fp32)
for name in ("transformer_ref", "vae", "audio_vae"):
    getattr(pipe, name).to("cuda")

state = pipe(
    prompt="a red fox walking through snow, cinematic",
    references=[MiniMaxH3VideoReference.from_file("reference.mp4")],
    height=768,
    width=768,
    num_frames=124,          # must be 17*n + 5
    num_inference_steps=4,   # 4 with the Turbo LoRA; ~20 without
    generator=torch.Generator("cpu").manual_seed(0),
)

frames = state.get("videos")[0]          # list of PIL frames
audio = state.get("audio")               # waveform (or None)
sampling_rate = state.get("sampling_rate")

Swapping back to the stock decoder is just pointing vae at MiniMaxAI/MiniMax-H3 / vae in the index.

How it was built

vae/ is an AutoencoderKLMiniMaxH3 instantiated with decoder_num_layers=12, loaded with:

• encoder.* + quant_conv.* from MiniMaxAI/MiniMax-H3/vae (stock H3 encoder),
• decoder.* + post_quant_conv.* from lightx2v/LightVAE/lightvae-lite-h3-decoder.safetensors.

latents_mean/std are bit-identical between the stock VAE config and the LightVAE decoder checkpoint, confirming the shared latent interface. State dict loads strict (0 missing, 0 unexpected).

Roadmap

• Lite encoder. LightVAE also ships a Lite encoder (lightvae-lite-h3-encoder.safetensors, non-uniform resnet depth [1,1,1,1,1,2]). Loading it needs AutoencoderKLMiniMaxH3 to accept a per-level layers_per_block; tracked as a follow-up.
• int8 decoder. lightvae-lite-h3-decoder-int8-convrot.safetensors for a cheaper decode once its kernel path is wired.

Credits

LightVAE by lightx2v. MiniMax-H3 by MiniMaxAI.