Add optimized MiniMax-H3 NVFP4 Space

README.md

@@ -1,13 +1,135 @@
 ---
-title: Zero Wave2 2
-emoji: 📈
+title: MiniMax H3 Ultra
+emoji: ⚡
 colorFrom: purple
-colorTo: blue
+colorTo: indigo
 sdk: gradio
-sdk_version: 6.19.0
-python_version: '3.13'
+sdk_version: 6.20.0
 app_file: app.py
-pinned: false
+pinned: true
+short_description: Blackwell-native NVFP4 video + synchronized audio generation
+suggested_hardware: zero-a10g
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# MiniMax-H3 Ultra — pruned NVFP4 on Blackwell
+
+Joint video and synchronized sound from MiniMax-H3, with the repeatedly executed transformer rebuilt around the
+Blackwell-native ComfyUI optimization path:
+
+- 12.5 GB pruned NVFP4 transformer instead of the 61.7 GiB BF16 inference transformer.
+- 20.1B effective inference parameters instead of 33.1B; the redundant 13.04B AdaLN projection weights become a
+  compact sampled timestep curve.
+- One fused QKV projection per attention layer.
+- One fused in-place Q/K RMSNorm + partial split-half RoPE kernel.
+- Native CUDA 13 NVFP4 tensor-core GEMMs through `comfy-kitchen`.
+- Segment-wise in-place AdaLN modulation and gated residual accumulation.
+- Video and audio output heads run only on their own rows, not the full packed sequence.
+- Prompt refinement and the rotary table are cached for the request instead of recomputed at every denoising step.
+- The video VAE and audio VAE remain full precision.
+
+The source model is [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3). The pruned NVFP4
+checkpoint is
+[`lilcheaty/MiniMax-H3-NVFP4`](https://huggingface.co/lilcheaty/MiniMax-H3-NVFP4), derived from ComfyUI's
+[`MiniMax-H3`](https://huggingface.co/Comfy-Org/MiniMax-H3) repackage.
+
+## Why the pruned transformer matters
+
+MiniMax-H3's published model card notes that about 13B parameters live in AdaLN-related branches and that their
+outputs can be precomputed for inference. The pruned checkpoint makes that concrete: it samples the shared timestep
+embedding curve at 1025 points and linearly interpolates an 8-value coordinate for each requested timestep. Every
+block's modulation projection consequently becomes `[96768, 8]` instead of `[96768, 2688]`.
+
+| parameter group | original BF16 architecture | pruned architecture |
+|---|---:|---:|
+| AdaLN projections | 13.04B | 0.04B |
+| MLP | 12.02B | 11.56B |
+| attention | 8.02B | 7.71B |
+| refiner, norms and embeddings | 0.05B | 0.80B |
+| **total** | **33.12B** | **20.11B** |
+
+The four large matrices in each of the 50 blocks—fused QKV, attention output, MLP up/gate and MLP down—are NVFP4.
+The modulation curve, norms, embeddings, biases and final heads stay at higher precision.
+
+## Why this is faster than the old 4-bit Space
+
+Quantization alone does not guarantee speed. The older 4-bit comparison paid for CPU offload traffic on every layer
+because its runtime did not keep the transformer resident. This engine is about 12 GB, so the transformer, both VAEs,
+working activations and decoder workspace fit together on the 95 GiB `xlarge` ZeroGPU worker. There is no layerwise
+host-device weight traffic in the denoising loop.
+
+ComfyUI reports about a 2× NVFP4 uplift over FP8/BF16 on Blackwell in supported workloads. The H3 checkpoint author
+measured 1.90 s/iteration for pruned NVFP4 versus 2.17 s/iteration for pruned INT8 ConvRot on an RTX PRO 6000
+Blackwell at 864×480, 39 frames. Those numbers are useful implementation evidence, not a promise for every canvas:
+H3 attention grows quadratically with packed sequence length, so resolution, duration and keyframe vision tokens
+still dominate large requests.
+
+## Split deployment
+
+The full checkpoint cannot fit under a single Space's 150 GB storage ceiling. This Space remains the denoising half:
+
+| component | where it runs | precision / format |
+|---|---|---|
+| Qwen3-VL layer-50 conditioner | [`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner) | BF16 |
+| H3 transformer | this Space | pruned NVFP4 + higher-precision islands |
+| video VAE | this Space | full precision checkpoint policy |
+| audio VAE | this Space | FP32 |
+
+The wire format is `prompt_embeds` plus `text_token_tags` in a safetensors file. The conditioner also returns the
+resolved canvas, aligned frame count and prompt plan. Keyframes are encoded again by this Space's video VAE so the
+conditioning latents exactly match the pixels seen by the conditioner.
+
+## Kernel path
+
+`h3_nvfp4.py` adapts the public ComfyUI H3 implementation to diffusers' packed transformer signature. It deliberately
+does not install or launch the ComfyUI application. The small adapter uses only `comfy-kitchen` for:
+
+1. Dynamic NVFP4 activation quantization and native FP4 matrix multiplication.
+2. Fused in-place Q/K RMSNorm and three-axis split-half rotary embedding.
+
+Attention itself stays on diffusers' `_native_cudnn` backend, which is faster than the default SDPA path on the
+ZeroGPU RTX PRO 6000 pool. The MLP uses one fused QKV-style gate/up matrix, in-place SiLU×up, and the NVFP4 down
+projection.
+
+The old BF16/AoTI engine remains available with `H3_ENGINE=bf16`. It is useful as a quality/debug reference, but it is
+not the default.
+
+## Quality trade-off
+
+NVFP4 is approximate. The checkpoint author reports that 4-bit weights can show more mid-motion artifacts and weaker
+shape retention than the larger INT8 ConvRot checkpoint on difficult 15-second clips. The comparison was not fully
+controlled, so treat it as a real caution rather than a quantified quality score.
+
+The Space keeps both VAEs full precision and leaves AdaLN, norms, embeddings and output heads out of NVFP4. For the
+exact original denoiser, set `H3_ENGINE=bf16`; this restores the 61.7 GiB unquantized transformer and its AoTI option.
+
+## Space variables
+
+| variable | default | meaning |
+|---|---|---|
+| `H3_ENGINE` | `nvfp4` | `nvfp4` ultra engine or `bf16` reference engine. |
+| `H3_NVFP4_REPO` | `lilcheaty/MiniMax-H3-NVFP4` | Repository containing the pruned Comfy-format transformer. |
+| `H3_NVFP4_FILE` | `minimax_h3_fl2va_pruned_nvfp4.safetensors` | FL2VA/T2VA transformer file. |
+| `H3_MODEL_REPO` | `MiniMaxAI/MiniMax-H3` | Canonical schedulers and VAE checkpoint. |
+| `H3_CONDITIONER` | `multimodalart/qwen3vl-conditioner` | Remote layer-50 conditioner Space. |
+| `H3_PLACEMENT` | `lazy` (`nvfp4`) | Move the compact transformer and VAEs on the first GPU call, then keep them resident. |
+| `H3_ATTENTION` | `_native_cudnn` | Attention backend for both the main stack and text refiner. |
+| `H3_GPU_SIZE` | `xlarge` | 95 GiB Blackwell ZeroGPU allocation. |
+| `H3_AOTI` | `0` | BF16 engine only: load the optional repeated-block AoTI package. |
+
+## Runtime requirements
+
+- PyTorch 2.11 with CUDA 13.0.
+- A Blackwell GPU (`sm120` for this Space). NVFP4 on older architectures is emulated and can be slower than BF16.
+- `comfy-kitchen==0.2.26` for the native layouts and fused Q/K kernel.
+- The pinned MiniMax-H3 diffusers pull request for the modular schedulers, packing and VAE decode path.
+
+No secret is required. All model artifacts are public, and `gradio_client` forwards the requesting user's ZeroGPU
+identity to the conditioner.
+
+## Attribution
+
+The fused/pruned model structure follows
+[`comfy/ldm/minimax/model.py`](https://github.com/Comfy-Org/ComfyUI/blob/master/comfy/ldm/minimax/model.py) from
+ComfyUI (Apache-2.0). The quantized checkpoint and its conversion notes are from
+[`lilcheaty/MiniMax-H3-NVFP4`](https://huggingface.co/lilcheaty/MiniMax-H3-NVFP4). MiniMax-H3 weights remain governed
+by the MiniMax-H3 Community License Agreement.

app.py

@@ -0,0 +1,451 @@
+"""MiniMax-H3 `t2va` / `fl2va`, split deployment — the denoising half."""
+
+from __future__ import annotations
+
+import os
+import tempfile
+import time
+import traceback
+from functools import cache
+
+# Before anything that could initialize CUDA: `import spaces` patches `torch.cuda` so model loading can happen at
+# startup rather than on GPU time.
+import spaces
+import gradio as gr
+
+MODEL_REPO = os.environ.get("H3_MODEL_REPO", "MiniMaxAI/MiniMax-H3")
+CONDITIONER_SPACE = os.environ.get("H3_CONDITIONER", "multimodalart/qwen3vl-conditioner")
+# `nvfp4` is the Blackwell-native ultra path; `bf16` preserves the original 33B diffusers transformer as a fallback.
+ENGINE = os.environ.get("H3_ENGINE", "nvfp4").lower()
+# `pack` places the transformer at startup, `lazy` moves everything on the first GPU call, `offload` hands placement to
+# `ComponentsManager.enable_auto_cpu_offload`.
+PLACEMENT = os.environ.get("H3_PLACEMENT", "lazy" if ENGINE == "nvfp4" else "pack").lower()
+# cuDNN's fused attention is 10-20% faster than the SDPA default on this pool and needs nothing installed.
+ATTENTION = os.environ.get("H3_ATTENTION", "_native_cudnn").lower()
+GPU_SIZE = os.environ.get("H3_GPU_SIZE", "xlarge")
+
+# Must stay identical to the conditioner's table: the *label* goes over the wire, so a canvas that half does not know
+# is rejected there and surfaces as a failure here.
+CANVASES = {
+    # 16:9
+    "960x544 · 16:9 fast": (544, 960),
+    "1024x576 · 16:9 fast": (576, 1024),
+    "1152x640 · 16:9": (640, 1152),
+    "1280x704 · 16:9": (704, 1280),
+    "1344x768 · 16:9 full": (768, 1344),
+    # 9:16
+    "544x960 · 9:16 fast": (960, 544),
+    "640x1152 · 9:16": (1152, 640),
+    "768x1344 · 9:16 full": (1344, 768),
+    # 1:1
+    "544x544 · 1:1 fast": (544, 544),
+    "768x768 · 1:1 full": (768, 768),
+    # 4:3 / 3:4
+    "768x576 · 4:3 fast": (576, 768),
+    "1024x768 · 4:3 full": (768, 1024),
+    "576x768 · 3:4 fast": (768, 576),
+    "768x1024 · 3:4 full": (1024, 768),
+    # 21:9
+    "1152x512 · 21:9 fast": (512, 1152),
+    "1536x672 · 21:9 full": (672, 1536),
+}
+DEFAULT_CANVAS = "960x544 · 16:9 fast"
+FPS, FRAMES_PER_CHUNK, LATENTS_PER_CHUNK = 24, 17, 5
+# It is the *snapped* frame count the ceiling has to hold for: 15 s is 360 frames, which rounds up to 362, i.e.
+# 15.083 s, and is refused.
+MIN_UI_DURATION, MAX_UI_DURATION = 2, 14
+
+
+def snap_frames(seconds: float) -> int:
+    """The frame count MiniMax-H3's video VAE can decode: the next `17 * n + 5` at 24 fps."""
+    frames = max(1, round(float(seconds) * FPS))
+    while frames % FRAMES_PER_CHUNK != LATENTS_PER_CHUNK:
+        frames += 1
+    return frames
+
+
+def lower_duration_floor(seconds: float = MIN_UI_DURATION) -> None:
+    """Let the pipeline generate below its 5 s floor. 56 frames (2.33 s) is fine on the released checkpoint."""
+    from diffusers.modular_pipelines.minimax_h3.modular_pipeline import MiniMaxH3ModularPipeline
+
+    MiniMaxH3ModularPipeline.min_duration = property(lambda self: float(seconds))
+
+
+PIPE = None
+MANAGER = None
+LOAD_ERROR: str | None = None
+LOADED_IN: float | None = None
+
+
+def status() -> str:
+    if LOAD_ERROR:
+        return LOAD_ERROR
+    if PIPE is None:
+        payload = (
+            "pruned NVFP4 transformer + full-precision VAEs (~28 GB)"
+            if ENGINE == "nvfp4"
+            else "BF16 transformer + VAEs (77.3 GB)"
+        )
+        return f"Loading {payload}. Watch the Space logs."
+    if ENGINE == "nvfp4":
+        import h3_nvfp4
+
+        engine_status = h3_nvfp4.status()
+    else:
+        import h3_aoti
+
+        engine_status = f"BF16, unquantized · {h3_aoti.status()}"
+    return (
+        f"Ready · **{engine_status}** · VAEs full precision · placement `{PLACEMENT}` · attention `{ATTENTION}` · "
+        f"loaded in {LOADED_IN:.0f}s · conditioner `{CONDITIONER_SPACE}`"
+    )
+
+
+def load_models() -> str | None:
+    """Load the denoising half at startup.
+
+    `MiniMaxH3GeneratorBlocks` declares `transformer`, `vae`, `audio_vae`, the two schedulers and `video_processor`,
+    so `load_components` fetches exactly those subfolders — `text_encoder/` and `transformer_ref/` are never touched.
+    Both autoencoders carry `_keep_in_fp32_modules` over every module and stay float32: a bfloat16 audio VAE decodes
+    the soundtrack roughly 20 dB too quiet.
+    """
+    global PIPE, MANAGER, LOAD_ERROR, LOADED_IN
+
+    if PIPE is not None or LOAD_ERROR is not None:
+        return LOAD_ERROR
+
+    started = time.time()
+    try:
+        import torch
+        from diffusers import ComponentsManager
+
+        from h3_split_blocks import MiniMaxH3GeneratorBlocks
+
+        lower_duration_floor()
+        manager = ComponentsManager()
+        blocks = MiniMaxH3GeneratorBlocks()
+        print(f"[gen] loading {[c.name for c in blocks.expected_components]} from {MODEL_REPO} ...", flush=True)
+        pipe = blocks.init_pipeline(MODEL_REPO, components_manager=manager, collection="h3")
+        if ENGINE == "nvfp4":
+            # Do not download the 61.7 GiB BF16 transformer. The schedulers and full-precision VAEs stay canonical;
+            # only the repeatedly executed DiT is replaced with the pruned Blackwell-native checkpoint.
+            pipe.load_components(
+                names=["vae", "audio_vae", "scheduler", "audio_scheduler", "video_processor"],
+                dtype=torch.bfloat16,
+            )
+            from h3_nvfp4 import load_transformer
+
+            pipe.update_components(transformer=load_transformer())
+        elif ENGINE == "bf16":
+            pipe.load_components(dtype=torch.bfloat16)
+        else:
+            raise ValueError(f"H3_ENGINE must be `nvfp4` or `bf16`, got {ENGINE!r}")
+        pipe.transformer.set_attention_backend(ATTENTION)
+
+        # Still startup, still free: an AoTI package carries no weights and opens its archive lazily inside the GPU
+        # worker. Off unless `H3_AOTI=1`.
+        if ENGINE == "bf16":
+            import h3_aoti
+
+            h3_aoti.maybe_load(pipe.transformer)
+
+        if PLACEMENT == "pack":
+            # Scoped to the transformer. `spaces` packs every startup-resident CUDA tensor into a second on-disk copy,
+            # and packing all 77.3 GB busts the 150 GB storage quota; the 61.7 GB transformer alone fits. The ~10 GB of
+            # fp32 VAEs move on the first GPU call instead.
+            pipe.transformer.to("cuda")
+
+        if PLACEMENT == "offload":
+            manager.enable_auto_cpu_offload(device="cuda")
+            _arm_decode_hooks(pipe)
+
+        PIPE, MANAGER = pipe, manager
+        LOADED_IN = time.time() - started
+        print(f"[gen] ready in {LOADED_IN:.0f}s", flush=True)
+    except Exception as error:
+        traceback.print_exc()
+        LOAD_ERROR = f"**Loading `{MODEL_REPO}` failed** after {time.time() - started:.0f}s: `{type(error).__name__}: {error}`"
+    return LOAD_ERROR
+
+
+def _arm_decode_hooks(pipe):
+    """Make the offload hooks fire for the two VAEs.
+
+    `enable_auto_cpu_offload` wraps `forward`, and the decode blocks call `vae.decode(...)` directly, so the hook
+    never runs and the VAE is still on the host when the latents arrive on the card.
+    """
+    for name in ("vae", "audio_vae"):
+        module = getattr(pipe, name)
+        inner = module.decode
+
+        def armed(*args, _module=module, _decode=inner, **kwargs):
+            hook = getattr(_module, "_hf_hook", None)
+            if hook is not None:
+                hook.pre_forward(_module)
+            return _decode(*args, **kwargs)
+
+        module.decode = armed
+
+
+@cache
+def conditioner():
+    """The other half, over the gradio API. `gradio_client` attaches the caller's own ZeroGPU token per call, so the
+    conditioner's booking is billed to whoever asked for the video."""
+    from gradio_client import Client
+
+    return Client(CONDITIONER_SPACE)
+
+
+def encode_remote(prompt, image_path, last_image_path, canvas, num_frames, rewrite_prompt=False):
+    """`/encode` on the conditioner Space: a safetensors file holding `prompt_embeds` + `text_token_tags`, with the
+    resolved `height` / `width` / `num_frames` in its metadata, plus the plan. `canvas` is the label."""
+    from gradio_client import handle_file
+    from safetensors import safe_open
+
+    path, plan = conditioner().predict(
+        prompt=prompt,
+        image_path=handle_file(image_path) if image_path else None,
+        last_image_path=handle_file(last_image_path) if last_image_path else None,
+        canvas=canvas,
+        num_frames=num_frames,
+        rewrite_prompt=bool(rewrite_prompt),
+        api_name="/encode",
+    )
+    with safe_open(path, framework="pt") as handle:
+        metadata = handle.metadata()
+        return handle.get_tensor("prompt_embeds"), handle.get_tensor("text_token_tags"), metadata, plan
+
+
+# Seconds of GPU one request needs, from the packed video rows it is about to denoise: linear in the rows for the
+# matmuls, quadratic for the attention, against the AoTI block package this Space runs.
+_DUR_B, _DUR_C = 1.1745e-4, 3.8396e-9
+# The two resident decoders and the mux, which scale with the output rather than with the step count.
+_DECODE_BASE, _DECODE_PER_DEFAULT_CANVAS, _DEFAULT_CANVAS_PIXELS = 15, 15, 960 * 544 * 124
+# `pack` mode: only the ~10 GB of VAEs move on a cold worker.
+_PLACEMENT_ALLOWANCE, _PAD = 12, 10
+
+
+def get_duration(prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, steps, seed, *a, **k):
+    height, width, num_frames, steps = int(height), int(width), int(num_frames), int(steps)
+    latent_frames = (num_frames - LATENTS_PER_CHUNK) // FRAMES_PER_CHUNK * LATENTS_PER_CHUNK + 2
+    patches = (height // 32) * (width // 32)
+    rows = latent_frames * patches + (int(image is not None) + int(last_image is not None)) * patches
+    denoise = steps * (_DUR_B * rows + _DUR_C * rows**2)
+    decode = _DECODE_BASE + _DECODE_PER_DEFAULT_CANVAS * (height * width * num_frames) / _DEFAULT_CANVAS_PIXELS
+    return max(60, int(denoise + decode) + _PLACEMENT_ALLOWANCE + _PAD)
+
+
+@spaces.GPU(duration=get_duration, size=GPU_SIZE)
+def _generate(prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, steps, seed):
+    """The only thing on GPU time: the packed-sequence denoise loop and the two decoders.
+
+    Only the three generated outputs come back — a `@spaces.GPU` return crosses a process boundary by pickling, and
+    the full `PipelineState` still holds the packed latents, the rotary grid and the row indices on the card.
+    """
+    import torch
+
+    if PLACEMENT == "lazy":
+        PIPE.to("cuda")
+    elif PLACEMENT == "pack":
+        PIPE.vae.to("cuda")
+        PIPE.audio_vae.to("cuda")
+
+    begin_request = getattr(PIPE.transformer, "begin_request", None)
+    end_request = getattr(PIPE.transformer, "end_request", None)
+    if begin_request is not None:
+        begin_request()
+    try:
+        with torch.inference_mode():
+            state = PIPE(
+                prompt_embeds=prompt_embeds.to("cuda", non_blocking=True),
+                text_token_tags=text_token_tags,
+                image=image,
+                last_image=last_image,
+                height=height,
+                width=width,
+                num_frames=num_frames,
+                num_inference_steps=int(steps),
+                generator=torch.Generator("cpu").manual_seed(int(seed)),
+            )
+    finally:
+        if end_request is not None:
+            end_request()
+    return state.get("videos")[0], state.get("audio")[0].cpu(), state.get("sampling_rate")
+
+
+def generate(prompt, image_path=None, last_image_path=None, canvas=DEFAULT_CANVAS, duration=5, steps=28, seed=42, upsample=False, progress=gr.Progress(track_tqdm=True)):
+    """One request. `upsample` is last and defaults off, so a positional API client that predates it is unaffected."""
+    if LOAD_ERROR:
+        raise gr.Error(LOAD_ERROR)
+    if PIPE is None:
+        raise gr.Error("The denoiser is still loading.")
+    if not prompt or not prompt.strip():
+        raise gr.Error("MiniMax-H3 always takes a prompt, keyframes or not.")
+
+    from PIL import Image, ImageOps
+
+    from diffusers.utils import encode_video
+
+    num_frames = snap_frames(duration)
+
+    progress(0.0, desc=f"Upsampling the prompt on {CONDITIONER_SPACE} ..." if upsample else f"Conditioning on {CONDITIONER_SPACE} ...")
+    conditioned = time.time()
+    prompt_embeds, text_token_tags, metadata, plan = encode_remote(
+        prompt, image_path, last_image_path, canvas, num_frames, rewrite_prompt=upsample
+    )
+    condition_seconds = time.time() - conditioned
+    height, width, num_frames = (int(metadata[key]) for key in ("height", "width", "num_frames"))
+    refined = plan.get("refined_prompt") or ""
+
+    def keyframe(path):
+        # The conditioning latents encoded here have to be of the image the conditioner looked at, which it prepares
+        # exactly this way.
+        return ImageOps.exif_transpose(Image.open(path)).convert("RGB") if path else None
+
+    progress(0.1, desc=f"Denoising {steps} steps at {width}x{height}, {num_frames} frames ...")
+    started = time.time()
+    frames, audio, sampling_rate = _generate(
+        prompt_embeds,
+        text_token_tags,
+        keyframe(image_path),
+        keyframe(last_image_path),
+        height,
+        width,
+        num_frames,
+        steps,
+        seed,
+    )
+    generate_seconds = time.time() - started
+
+    directory = os.path.join(tempfile.gettempdir(), "h3-outputs")
+    os.makedirs(directory, exist_ok=True)
+    path = os.path.join(directory, f"h3-{int(time.time() * 1000)}.mp4")
+    encode_video(frames, fps=FPS, output_path=path, audio=audio, audio_sample_rate=sampling_rate)
+
+    report = (
+        f"`{width}x{height}`, {num_frames} frames ({num_frames / FPS:.3f} s), {int(steps)} steps · "
+        f"conditioner {condition_seconds:.0f}s ({plan['num_text_tokens']} tokens"
+        f"{', upsampled' if refined else ''}) · "
+        f"denoise + decode {generate_seconds:.0f}s ({generate_seconds / int(steps):.1f} s/step) · seed {int(seed)}"
+    )
+    print(f"[gen] {report}", flush=True)
+    return path, report, refined, gr.update(visible=bool(refined))
+
+
+def _fit_keyframe(image_path, current_canvas):
+    """Cover-crop an uploaded keyframe to the closest supported aspect ratio and select that ratio's smallest
+    (fastest) canvas, unless the user already picked a matching ratio."""
+    if not image_path:
+        return gr.update(), gr.update()
+    from PIL import Image as _Image
+
+    img = _Image.open(image_path)
+    aspect = img.width / img.height
+    fastest = {}
+    for label, (h, w) in CANVASES.items():
+        r = w / h
+        if r not in fastest or w * h < fastest[r][1][0] * fastest[r][1][1]:
+            fastest[r] = (label, (h, w))
+    ratio = min(fastest, key=lambda r: abs(r - aspect))
+    label, (h, w) = fastest[ratio]
+
+    cur_h, cur_w = CANVASES[current_canvas]
+    if abs(cur_w / cur_h - aspect) <= abs(ratio - aspect):
+        label = current_canvas
+        h, w = cur_h, cur_w
+
+    target = w / h
+    if abs(img.width / img.height - target) <= 1e-3:
+        return gr.update(), gr.update(value=label)
+    if img.width / img.height > target:
+        new_w = int(img.height * target)
+        left = (img.width - new_w) // 2
+        img = img.crop((left, 0, left + new_w, img.height))
+    else:
+        new_h = int(img.width / target)
+        top = (img.height - new_h) // 2
+        img = img.crop((0, top, img.width, top + new_h))
+    img.save(image_path)
+    return gr.update(value=image_path), gr.update(value=label)
+
+
+load_models()
+
+INTRO = """# MiniMax-H3 Ultra
+
+<div align="center">
+  <a href="https://huggingface.co/MiniMaxAI/MiniMax-H3" target="_blank" rel="noopener"><strong>[ model ]</strong></a> &nbsp;
+  <a href="https://huggingface.co/lilcheaty/MiniMax-H3-NVFP4" target="_blank" rel="noopener"><strong>[ NVFP4 ]</strong></a> &nbsp;
+  <a href="https://www.minimax.io/blog/minimax-h3" target="_blank" rel="noopener"><strong>[ blog ]</strong></a> &nbsp;
+  <a href="https://huggingface.co/spaces/multimodalart/minimax-h3-reference" target="_blank" rel="noopener"><strong>[ reference to video ]</strong></a>
+</div>
+
+**MiniMax-H3 Ultra** runs the pruned Blackwell-native NVFP4 transformer with fused QKV, fused Q/K norm + RoPE,
+full-precision video/audio decoders, and the original synchronized soundtrack generation.
+"""
+
+CSS = """
+.main.fillable {max-width: 1250px !important}
+.dark .gradio-container { color: var(--body-text-color); }
+"""
+
+with gr.Blocks(title="MiniMax-H3") as demo:
+    gr.Markdown(INTRO)
+
+    with gr.Row():
+        with gr.Column():
+            prompt = gr.Textbox(
+                label="Prompt",
+                lines=3,
+                value="A red fox trotting through a snowy pine forest at dawn, snow crunching underfoot",
+            )
+            upsample = gr.Checkbox(label="Upsample prompt", value=False)
+            with gr.Row():
+                image = gr.Image(label="First frame (optional)", type="filepath")
+                last_image = gr.Image(label="Last frame (optional)", type="filepath")
+            run = gr.Button("Generate", variant="primary")
+            with gr.Accordion("Advanced options", open=False):
+                canvas = gr.Dropdown(label="Canvas", choices=list(CANVASES), value=DEFAULT_CANVAS)
+                duration = gr.Slider(label="Duration (s)", minimum=MIN_UI_DURATION, maximum=MAX_UI_DURATION, step=1, value=5)
+                steps = gr.Slider(label="Steps", minimum=10, maximum=40, step=1, value=28)
+                seed = gr.Number(label="Seed", value=42, precision=0)
+
+        with gr.Column():
+            video = gr.Video(label="Video + soundtrack")
+            report = gr.Markdown(visible=False)
+            # An output, so it can be revealed only for a request that asked for a rewrite.
+            with gr.Accordion("Upsampled prompt", open=False, visible=False) as upsampled_panel:
+                upsampled = gr.Textbox(show_label=False, lines=8, interactive=False)
+
+    image.upload(_fit_keyframe, [image, canvas], [image, canvas])
+
+    gr.Examples(
+        examples=[
+            ["A red fox trotting through a snowy pine forest at dawn, snow crunching underfoot", None, None, "1344x768 · 16:9 full"],
+            ["A busy night market, neon signs reflecting in puddles, sizzling street food", None, None, "768x1344 · 9:16 full"],
+            ["A cellist playing a slow melody in an empty concert hall", None, None, "768x768 · 1:1 full"],
+            ["The fox looks around, then trots deeper into the forest", "examples/first.png", None, "1344x768 · 16:9 full"],
+            ["A slow seamless camera move from the first view to the last", "examples/first.png", "examples/last.png", "1344x768 · 16:9 full"],
+        ],
+        inputs=[prompt, image, last_image, canvas],
+        outputs=[video, report, upsampled, upsampled_panel],
+        fn=generate,
+        cache_examples=True,
+        cache_mode="lazy",
+    )
+
+    run.click(
+        generate,
+        [prompt, image, last_image, canvas, duration, steps, seed, upsample],
+        [video, report, upsampled, upsampled_panel],
+        api_name="generate",
+    )
+
+    gr.Markdown(
+        '<div style="text-align:center"><a href="https://x.com/realmrfakename" target="_blank" '
+        'rel="noopener">@realmrfakename</a></div>'
+    )
+
+
+if __name__ == "__main__":
+    demo.queue().launch(show_error=True, theme=gr.themes.Citrus(), css=CSS)

h3_aoti.py

@@ -0,0 +1,307 @@
+"""ZeroGPU AoTI for MiniMax-H3: one compiled `MiniMaxH3TransformerBlock` package, reused by all 50 blocks.
+
+Shared byte-identically by every MiniMax-H3 Space. A Space only calls `maybe_load()`; the rest is the build path.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AOTI = os.environ.get("H3_AOTI", "0") == "1"
+AOTI_REPO = os.environ.get("H3_AOTI_REPO", "multimodalart/minimax-h3-aoti")
+AOTI_REPO_TYPE = os.environ.get("H3_AOTI_REPO_TYPE", "model")
+# A package is valid for exactly one `<width>/torch<X.Y>/sm<cc>/<shape>`, and a mismatched one segfaults rather than
+# raising, so `maybe_load` refuses anything but this key.
+AOTI_KEY = os.environ.get("H3_AOTI_KEY", "bf16/torch2.11/sm120/dynamic")
+# `dynamic` is the sequence dimension: `build_packed_sequence` pads nothing, so `S` moves with the prompt as well as
+# the canvas and a static package would serve one prompt length.
+AOTI_SHAPE = os.environ.get("H3_AOTI_SHAPE", "dynamic")
+AOTI_DURATION = int(os.environ.get("H3_AOTI_DURATION", "1500"))
+
+# Where a step spends its time. `MiniMaxH3TokenRefinerBlock` is also repeated but runs a handful of text rows.
+BLOCK_CONTAINER = "transformer_blocks"
+
+# Height of the AdaLN table baked into the package. `temb` grows from 1 row (step 0, both streams at one noise level)
+# to 2 (from step 1, sigmas diverged), and the block gathers from `3 * rows`, so the row count is part of the compiled
+# shape and is pinned by padding on both sides of the compile. Must match the package's `H3_AOTI_TEMB_ROWS`.
+TEMB_ROWS = int(os.environ.get("H3_AOTI_TEMB_ROWS", "4"))
+
+_LOADED: set[int] = set()
+
+
+def pad_temb(temb, rows: int = TEMB_ROWS):
+    """Grow `temb` to exactly `rows` timestep rows by repeating its last one."""
+    present = temb.shape[0]
+    if present == rows:
+        return temb
+    if present > rows:
+        raise RuntimeError(
+            f"{present} distinct timesteps, but this AoTI package holds at most {rows}. "
+            f"Recompile with H3_AOTI_TEMB_ROWS>={present}."
+        )
+    import torch
+
+    return torch.cat([temb, temb[-1:].expand(rows - present, *temb.shape[1:])], dim=0)
+
+
+def width() -> str:
+    """Which transformer these artifacts belong to: `bf16`, `fp8`, `nvfp4`, ..."""
+    if explicit := os.environ.get("H3_WIDTH"):
+        return explicit.lower()
+    try:
+        import h3_core
+
+        return h3_core.WIDTH
+    except Exception:
+        return "bf16"
+
+
+def artifact_key() -> str | None:
+    """`<width>/torch<X.Y>/sm<cc>/<shape>` of the card this process is on, or `None` when there is no CUDA."""
+    try:
+        import torch
+
+        torch_version = ".".join(torch.__version__.split(".")[:2])
+        major, minor = torch.cuda.get_device_capability()
+    except Exception:
+        return None
+    return f"{width()}/torch{torch_version}/sm{major}{minor}/{AOTI_SHAPE}"
+
+
+def status() -> str:
+    return (
+        f"AoTI **on** · `{AOTI_REPO}` ({AOTI_REPO_TYPE}) · shape `{AOTI_SHAPE}`"
+        if AOTI
+        else "AoTI **off** (`H3_AOTI=1` to load compiled blocks)"
+    )
+
+
+def patch_blocks(transformer, package_dir) -> None:
+    """Point all 50 blocks at the one compiled package, binding each block's own weights on its first call.
+
+    `spaces.aoti_load_from_package_dir` with two changes. Weights are read on the first forward rather than at patch
+    time, because this runs at startup and `Module.to` later rebinds `param.data` to fresh CUDA tensors. And `temb` is
+    padded to the height the package was exported with — see `TEMB_ROWS`.
+    """
+    from spaces.zero.torch.aoti import LazyAOTIModel, _shallow_clone_module
+    from torch._functorch._aot_autograd.subclass_parametrization import unwrap_tensor_subclass_parameters
+
+    # `LazyAOTIModel` binds constants by name and silently keeps what it cannot match, which is a SIGSEGV rather than
+    # an error. The patch resolves anonymous names through the compile side's sidecar and raises if it still cannot.
+    try:
+        from spaces_constant_binding_patch import apply_spaces_constant_binding_patch
+
+        apply_spaces_constant_binding_patch()
+    except ImportError:
+        print("[h3-aoti] spaces_constant_binding_patch.py is missing; an unbindable constant would segfault", flush=True)
+
+    model = LazyAOTIModel(Path(package_dir) / "submodules" / BLOCK_CONTAINER / "package.pt2")
+
+    for block in getattr(transformer, BLOCK_CONTAINER):
+        bound: dict = {}
+
+        def forward(hidden_states, temb, *rest, _block=block, _bound=bound):
+            first = not _bound
+            if first:
+                clone = _shallow_clone_module(_block)
+                unwrap_tensor_subclass_parameters(clone)
+                _bound["weights"] = clone.state_dict()
+            return model(_bound["weights"], first, hidden_states, pad_temb(temb), *rest)
+
+        block.forward = forward
+    print(f"[h3-aoti] {len(getattr(transformer, BLOCK_CONTAINER))} blocks patched (temb padded to {TEMB_ROWS})", flush=True)
+
+
+def maybe_load(transformer) -> None:
+    """Patch the block stack with its compiled package, or leave it eager. Safe to call at **startup**.
+
+    Off unless `H3_AOTI=1`, and anything that does not line up — another card, another torch, no `spaces` AoTI
+    helpers, no published package — falls back to eager with one line rather than raising or segfaulting. Nothing here
+    touches a GPU: the download is CPU work and the `.pt2` is not opened until the first forward.
+    """
+    if not AOTI or id(transformer) in _LOADED:
+        return
+
+    key = artifact_key()
+    if key is None:
+        print("[h3-aoti] no CUDA device visible; running eager", flush=True)
+        return
+    if key != AOTI_KEY:
+        print(f"[h3-aoti] this card wants `{key}`, only `{AOTI_KEY}` is published; running eager", flush=True)
+        return
+
+    try:
+        from huggingface_hub import snapshot_download
+        from spaces.zero.torch.aoti import LazyAOTIModel  # noqa: F401
+    except Exception as error:
+        print(f"[h3-aoti] no AoTI loader here ({type(error).__name__}: {error}); running eager", flush=True)
+        return
+
+    print(f"[h3-aoti] loading {AOTI_REPO}:{key} ...", flush=True)
+    try:
+        local = snapshot_download(repo_id=AOTI_REPO, repo_type=AOTI_REPO_TYPE, allow_patterns=f"{key}/package/*")
+    except Exception as error:
+        print(f"[h3-aoti] {AOTI_REPO}:{key} unreachable ({type(error).__name__}: {error}); running eager", flush=True)
+        return
+    package_dir = Path(local) / key / "package"
+    if not package_dir.is_dir():
+        print(f"[h3-aoti] no package at `{AOTI_REPO}:{key}/package`; running eager", flush=True)
+        return
+
+    patch_blocks(transformer, package_dir)
+    _LOADED.add(id(transformer))
+    print(f"[h3-aoti] compiled blocks in place (temb padded to {TEMB_ROWS} rows)", flush=True)
+
+
+def export_block(pipe, height: int, width: int, num_frames: int, prompt: str):
+    """Capture one block call out of a real request and export it with a dynamic sequence dimension.
+
+    Runs on the GPU, after the transformer has been quantized and moved there: a package compiled for one
+    quantization mode is meaningless for another.
+    """
+    import torch
+    import spaces
+
+    import h3_core as h3
+
+    transformer = h3.transformer_of(pipe)
+    blocks = getattr(transformer, BLOCK_CONTAINER)
+
+    # Keep the widest `temb` over a short real run rather than `spaces.aoti_capture`'s first call, which is the
+    # 1-row one — see `TEMB_ROWS`.
+    original_forward = blocks[0].forward
+    widest = {"args": (), "kwargs": {}, "rows": -1}
+    seen = []
+
+    def recording(*args, **kwargs):
+        rows = int(args[1].shape[0]) if len(args) > 1 and hasattr(args[1], "shape") else -1
+        seen.append(rows)
+        if rows > widest["rows"]:
+            widest.update(args=args, kwargs=kwargs, rows=rows)
+        return original_forward(*args, **kwargs)
+
+    blocks[0].forward = recording
+    try:
+        pipe(
+            prompt=prompt,
+            height=height,
+            width=width,
+            num_frames=num_frames,
+            num_inference_steps=int(os.environ.get("H3_AOTI_CAPTURE_STEPS", "4")),
+            generator=torch.Generator("cpu").manual_seed(42),
+        )
+    finally:
+        blocks[0].forward = original_forward
+    call = type("Captured", (), widest)
+    if not call.args:
+        raise RuntimeError("Nothing was captured — the transformer block was never called.")
+    print(f"[h3-aoti] temb rows seen: {sorted(set(seen))}; exporting with {TEMB_ROWS} (padded)", flush=True)
+
+    # `block(hidden_states, temb, adaln_indices, rotary_emb, attention_mask)`, `attention_mask` being `None` for the
+    # padless sequences these pipelines build. Only the sequence is dynamic: `torch.export` specializes size-1
+    # dimensions unconditionally, so a `Dim` on `temb`'s rows cannot be expressed at all.
+    if AOTI_SHAPE == "dynamic":
+        sequence = torch.export.Dim("sequence", min=2048, max=262144)
+        dynamic_shapes = ({1: sequence}, None, {0: sequence}, ({0: sequence}, {0: sequence}), None)
+        dynamic_shapes = dynamic_shapes[: len(call.args)]
+    else:
+        dynamic_shapes = None
+
+    args = (call.args[0], pad_temb(call.args[1]), *call.args[2:])
+
+    # Export the **live** block, non-strict. A shallow clone under non-strict tracing lifts every weight twice — once
+    # named, once as an anonymous `CONSTANT_TENSOR` aliasing it — and the loader binds by name, so the compiled block
+    # dereferences constants nobody set. The clone is only for flattening tensor-subclass parameters, which inductor's
+    # constant handling cannot wrap back into a `Parameter`, and it needs `strict=True`.
+    from spaces.zero.torch.aoti import _shallow_clone_module
+    from torch._functorch._aot_autograd.subclass_parametrization import unwrap_tensor_subclass_parameters
+
+    subclassed = sorted({type(p).__name__ for p in blocks[0].parameters()} - {"Parameter"})
+    if subclassed:
+        block = _shallow_clone_module(blocks[0])
+        unwrap_tensor_subclass_parameters(block)
+        strict = True
+        print(f"[h3-aoti] tensor-subclass parameters {subclassed}: exporting a flattened clone, strict=True", flush=True)
+    else:
+        block = blocks[0]
+        strict = False
+        print("[h3-aoti] plain parameters: exporting the live block, non-strict", flush=True)
+
+    # `torch.export` only gives a lifted tensor a real FQN when it is a registered parameter or buffer; a plain
+    # attribute becomes an anonymous constant the loader can never match. Only ever on the clone, since this
+    # re-registers attributes and the live block is what the eager path runs.
+    if block is not blocks[0]:
+        try:
+            from spaces_constant_binding_patch import register_loose_tensors
+
+            if loose := register_loose_tensors(block):
+                print(f"[h3-aoti] re-registered {len(loose)} loose tensors as buffers: {loose[:6]}", flush=True)
+        except ImportError:
+            pass
+
+    print(f"[h3-aoti] exporting {type(blocks[0]).__name__}, shapes={AOTI_SHAPE}, strict={strict} ...", flush=True)
+    try:
+        exported = torch.export.export(block, args, call.kwargs or None, dynamic_shapes=dynamic_shapes, strict=strict)
+    except Exception as error:
+        if not strict:
+            raise
+        print(f"[h3-aoti] strict export failed ({type(error).__name__}: {error}); retrying non-strict", flush=True)
+        exported = torch.export.export(block, args, call.kwargs or None, dynamic_shapes=dynamic_shapes)
+
+    anonymous = [
+        spec.target for spec in exported.graph_signature.input_specs if spec.kind.name == "CONSTANT_TENSOR"
+    ]
+    if anonymous:
+        print(
+            f"[h3-aoti] WARNING {len(anonymous)} constants lifted anonymously: {anonymous[:6]}. The loader binds by "
+            f"name, so `compile_and_save` writes the alias sidecar and `patch_blocks` raises rather than segfaulting.",
+            flush=True,
+        )
+    return exported
+
+
+def compile_and_save(exported_program, destination: str | os.PathLike[str]) -> Path:
+    """Inductor-compile the exported block into `<destination>/package/submodules/transformer_blocks/package.pt2`.
+
+    That layout is what `aoti_load_from_package_dir` walks, resolving the submodule name to the transformer's
+    `transformer_blocks` `ModuleList` and patching every block in it with this one package.
+    """
+    import spaces
+
+    package_dir = Path(destination) / "package"
+    print("[h3-aoti] inductor compile (minutes) ...", flush=True)
+    spaces.aoti_compile_and_save(package_dir, exported_program, submodule=BLOCK_CONTAINER)
+
+    # The compiled artifact drops a constant's FQN when the export lifted it anonymously; the `ExportedProgram` still
+    # has the real names, so record the mapping for the loader while it is available.
+    try:
+        from spaces_constant_binding_patch import write_constant_aliases
+
+        if sidecar := write_constant_aliases(package_dir, exported_program, submodule=BLOCK_CONTAINER):
+            print(f"[h3-aoti] constant alias sidecar written: {sidecar.name}", flush=True)
+    except ImportError:
+        pass
+
+    files = sorted(str(path.relative_to(package_dir)) for path in package_dir.rglob("*") if path.is_file())
+    print(f"[h3-aoti] package written: {files}", flush=True)
+    return package_dir
+
+
+def upload(package_dir: str | os.PathLike[str], key: str) -> str:
+    """Push the package under its `<width>/torch<X.Y>/sm<cc>/<shape>` key. CPU work — never inside GPU time."""
+    from huggingface_hub import HfApi
+
+    token = os.environ.get("HF_TOKEN")
+    if not token:
+        raise RuntimeError("`HF_TOKEN` is needed to push the AoTI package.")
+    api = HfApi(token=token)
+    api.create_repo(repo_id=AOTI_REPO, repo_type=AOTI_REPO_TYPE, private=False, exist_ok=True)
+    api.upload_folder(
+        folder_path=str(package_dir),
+        path_in_repo=f"{key}/package",
+        repo_id=AOTI_REPO,
+        repo_type=AOTI_REPO_TYPE,
+        commit_message=f"AoTI package for {key}",
+    )
+    return f"https://huggingface.co/{'datasets/' if AOTI_REPO_TYPE == 'dataset' else ''}{AOTI_REPO}/tree/main/{key}"

h3_nvfp4.py

@@ -0,0 +1,493 @@
+"""Blackwell-native MiniMax-H3 transformer for the pruned ComfyUI NVFP4 checkpoint.
+
+The public diffusers checkpoint spends 13.04B of its 33.12B parameters on per-block
+AdaLN projections.  ComfyUI's pruned checkpoint replaces those projections with an
+interpolated 1025-point timestep curve, fuses Q/K/V, and stores the four large linear
+layers in every block as NVFP4.  This adapter keeps diffusers' packed-sequence contract
+so the rest of the split Space (schedulers, VAEs and remote conditioner) stays unchanged.
+
+The kernel/layout conventions follow ComfyUI's Apache-2.0 implementation:
+https://github.com/Comfy-Org/ComfyUI/blob/master/comfy/ldm/minimax/model.py
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from types import SimpleNamespace
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+NVFP4_REPO = os.environ.get("H3_NVFP4_REPO", "lilcheaty/MiniMax-H3-NVFP4")
+NVFP4_FILE = os.environ.get("H3_NVFP4_FILE", "minimax_h3_fl2va_pruned_nvfp4.safetensors")
+
+HIDDEN = 5376
+HEADS = 56
+HEAD_DIM = 128
+FFN = 14336
+TEXT_DIM = 5120
+TIME_DIM = 8
+VIDEO_DIM = 24 * 1 * 2 * 2
+AUDIO_DIM = 32
+LAYERS = 50
+REFINER_LAYERS = 2
+EPS = 1e-5
+
+
+def _quant_config(handle, prefix: str) -> dict | None:
+    key = f"{prefix}.comfy_quant"
+    if key not in handle.keys():
+        return None
+    return json.loads(handle.get_tensor(key).numpy().tobytes())
+
+
+class H3Linear(nn.Module):
+    """A plain or comfy-kitchen NVFP4 linear, selected by checkpoint metadata."""
+
+    def __init__(
+        self,
+        in_features: int,
+        out_features: int,
+        bias: bool = False,
+        compute_dtype: torch.dtype | None = None,
+    ):
+        super().__init__()
+        self.in_features = in_features
+        self.out_features = out_features
+        self.compute_dtype = compute_dtype
+        self.register_parameter("weight", None)
+        self.register_parameter("bias", None)
+        self.register_buffer("input_scale", None)
+        self.register_buffer("pre_quant_scale", None)
+        self.quantized = False
+
+    def load(self, handle, prefix: str) -> None:
+        config = _quant_config(handle, prefix)
+        weight = handle.get_tensor(f"{prefix}.weight")
+
+        if config is None:
+            self.weight = nn.Parameter(
+                weight if self.compute_dtype is None else weight.to(self.compute_dtype), requires_grad=False
+            )
+        elif config.get("format") == "nvfp4":
+            from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
+
+            block_scale = handle.get_tensor(f"{prefix}.weight_scale")
+            if block_scale.dtype == torch.uint8:
+                block_scale = block_scale.view(torch.float8_e4m3fn)
+            tensor_scale = handle.get_tensor(f"{prefix}.weight_scale_2").float()
+            params = TensorCoreNVFP4Layout.Params(
+                scale=tensor_scale,
+                block_scale=block_scale,
+                orig_dtype=torch.bfloat16,
+                orig_shape=(self.out_features, self.in_features),
+            )
+            quantized = QuantizedTensor(weight.to(torch.uint8), "TensorCoreNVFP4Layout", params)
+            self.weight = nn.Parameter(quantized, requires_grad=False)
+            self.quantized = True
+            for name in ("input_scale", "pre_quant_scale"):
+                key = f"{prefix}.{name}"
+                if key in handle.keys():
+                    setattr(self, name, handle.get_tensor(key))
+        else:
+            raise ValueError(f"Unsupported quantization on {prefix}: {config}")
+
+        bias_key = f"{prefix}.bias"
+        if bias_key in handle.keys():
+            bias = handle.get_tensor(bias_key)
+            self.bias = nn.Parameter(
+                bias if self.compute_dtype is None else bias.to(self.compute_dtype), requires_grad=False
+            )
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        if self.pre_quant_scale is not None:
+            hidden_states = hidden_states * self.pre_quant_scale.to(
+                device=hidden_states.device, dtype=hidden_states.dtype
+            )
+        if not self.quantized:
+            hidden_states = hidden_states.to(self.weight.dtype)
+            return F.linear(
+                hidden_states,
+                self.weight,
+                self.bias,
+            )
+
+        from comfy_kitchen.tensor import QuantizedTensor
+
+        shape = hidden_states.shape
+        flat = hidden_states.reshape(-1, shape[-1])
+        scale = None if self.input_scale is None else self.input_scale.to(flat.device)
+        quantized_input = QuantizedTensor.from_float(flat, "TensorCoreNVFP4Layout", scale=scale)
+        output = F.linear(
+            quantized_input,
+            self.weight,
+            None if self.bias is None else self.bias.to(hidden_states.dtype),
+        )
+        return output.reshape(*shape[:-1], self.out_features)
+
+
+class H3RMSNorm(nn.Module):
+    def __init__(self, width: int, eps: float = EPS):
+        super().__init__()
+        self.width = width
+        self.eps = eps
+        self.register_parameter("weight", None)
+
+    def load(self, handle, prefix: str) -> None:
+        self.weight = nn.Parameter(handle.get_tensor(f"{prefix}.weight"), requires_grad=False)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        return F.rms_norm(
+            hidden_states,
+            (self.width,),
+            self.weight.to(device=hidden_states.device, dtype=hidden_states.dtype),
+            self.eps,
+        )
+
+
+class H3Attention(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.qkv_proj = H3Linear(HIDDEN, 3 * HEADS * HEAD_DIM)
+        self.q_norm = H3RMSNorm(HEAD_DIM)
+        self.k_norm = H3RMSNorm(HEAD_DIM)
+        self.out_proj = H3Linear(HEADS * HEAD_DIM, HIDDEN)
+
+    def load(self, handle, prefix: str) -> None:
+        self.qkv_proj.load(handle, f"{prefix}.qkv_proj")
+        self.q_norm.load(handle, f"{prefix}.q_norm")
+        self.k_norm.load(handle, f"{prefix}.k_norm")
+        self.out_proj.load(handle, f"{prefix}.out_proj")
+
+    def forward(self, hidden_states, rope_table, backend: str):
+        import comfy_kitchen as kitchen
+        from diffusers.models.attention_dispatch import dispatch_attention_fn
+
+        sequence = hidden_states.shape[0]
+        qkv = self.qkv_proj(hidden_states)
+        query, key, value = qkv.split(HEADS * HEAD_DIM, dim=-1)
+        query = query.view(1, sequence, HEADS, HEAD_DIM)
+        key = key.view(1, sequence, HEADS, HEAD_DIM)
+        value = value.view(1, sequence, HEADS, HEAD_DIM)
+
+        # One in-place kernel replaces Q RMSNorm, K RMSNorm and both partial RoPE applications.
+        kitchen.rms_rope_split_half_(
+            query,
+            key,
+            rope_table,
+            self.q_norm.weight.to(query.device),
+            self.k_norm.weight.to(key.device),
+            epsilon=self.q_norm.eps,
+            rot_dim=rope_table.shape[-3] * 2,
+        )
+        attended = dispatch_attention_fn(
+            query,
+            key,
+            value,
+            attn_mask=None,
+            dropout_p=0.0,
+            is_causal=False,
+            backend=backend,
+        )
+        return self.out_proj(attended.reshape(sequence, HEADS * HEAD_DIM))
+
+
+class H3MLP(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.fc1 = H3Linear(HIDDEN, 2 * FFN)
+        self.fc2 = H3Linear(FFN, HIDDEN)
+
+    def load(self, handle, prefix: str) -> None:
+        self.fc1.load(handle, f"{prefix}.fc1")
+        self.fc2.load(handle, f"{prefix}.fc2")
+
+    def forward(self, hidden_states):
+        gate, up = self.fc1(hidden_states).chunk(2, dim=-1)
+        return self.fc2(F.silu(gate).mul_(up))
+
+
+class H3RefinerBlock(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.norm1 = H3RMSNorm(HIDDEN)
+        self.attn = H3Attention()
+        self.norm2 = H3RMSNorm(HIDDEN)
+        self.mlp = H3MLP()
+
+    def load(self, handle, prefix: str) -> None:
+        self.norm1.load(handle, f"{prefix}.norm1")
+        self.attn.load(handle, f"{prefix}.attn")
+        self.norm2.load(handle, f"{prefix}.norm2")
+        self.mlp.load(handle, f"{prefix}.mlp")
+
+
+class H3AdaLN(nn.Module):
+    def __init__(self, expand: int, modalities: int):
+        super().__init__()
+        self.expand = expand
+        self.modalities = modalities
+        # Curve checkpoints deliberately evaluate interpolation and modulation projection in FP32. Expanding the
+        # checkpoint's tiny FP16 [*, 8] matrices once at load avoids 51 request-step casts.
+        self.linear = H3Linear(
+            TIME_DIM, expand * HIDDEN * modalities, bias=True, compute_dtype=torch.float32
+        )
+
+    def load(self, handle, prefix: str) -> None:
+        self.linear.load(handle, f"{prefix}.linear")
+
+    def forward(self, time_embedding):
+        projected = self.linear(time_embedding)
+        projected = projected.view(-1, self.expand * HIDDEN)
+        return projected.chunk(self.expand, dim=-1)
+
+
+class H3Block(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.norm1 = H3RMSNorm(HIDDEN)
+        self.attn = H3Attention()
+        self.norm2 = H3RMSNorm(HIDDEN)
+        self.mlp = H3MLP()
+        self.adaln_proj = H3AdaLN(6, 3)
+
+    def load(self, handle, prefix: str) -> None:
+        self.norm1.load(handle, f"{prefix}.norm1")
+        self.attn.load(handle, f"{prefix}.attn")
+        self.norm2.load(handle, f"{prefix}.norm2")
+        self.mlp.load(handle, f"{prefix}.mlp")
+        self.adaln_proj.load(handle, f"{prefix}.adaln_proj")
+
+
+class H3FinalLayer(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.norm = H3RMSNorm(HIDDEN)
+        self.adaln_proj = H3AdaLN(2, 1)
+        self.video_out = H3Linear(HIDDEN, VIDEO_DIM, bias=True, compute_dtype=torch.float32)
+        self.audio_out = H3Linear(HIDDEN, AUDIO_DIM, bias=True, compute_dtype=torch.float32)
+
+    def load(self, handle, prefix: str) -> None:
+        self.norm.load(handle, f"{prefix}.norm")
+        self.adaln_proj.load(handle, f"{prefix}.adaln_proj")
+        self.video_out.load(handle, f"{prefix}.video_out")
+        self.audio_out.load(handle, f"{prefix}.audio_out")
+
+
+class H3NVFP4Transformer(nn.Module):
+    """Diffusers-compatible H3 transformer backed by fused comfy-kitchen NVFP4 kernels."""
+
+    def __init__(self):
+        super().__init__()
+        # The modular pipeline reads these values through the diffusers component config rather than inspecting the
+        # module itself. Keep the public transformer contract even though this lean adapter is not a ConfigMixin.
+        self.config = SimpleNamespace(
+            patch_size=(1, 2, 2),
+            in_channels=24,
+            audio_in_channels=AUDIO_DIM,
+            text_dim=TEXT_DIM,
+        )
+        self.video_patch_proj = H3Linear(VIDEO_DIM, HIDDEN, bias=True, compute_dtype=torch.float32)
+        self.audio_patch_proj = H3Linear(AUDIO_DIM, HIDDEN, bias=True, compute_dtype=torch.float32)
+        self.condition_proj = H3Linear(TEXT_DIM, HIDDEN, bias=True)
+        self.token_refiner = nn.ModuleList([H3RefinerBlock() for _ in range(REFINER_LAYERS)])
+        self.token_refiner_norm = H3RMSNorm(HIDDEN)
+        self.blocks = nn.ModuleList([H3Block() for _ in range(LAYERS)])
+        self.final_layer = H3FinalLayer()
+        self.register_buffer("adaln_t_table", None)
+        self.register_buffer("rope_inv_freq", None)
+        self.attention_backend = "_native_cudnn"
+        self._text_cache = None
+        self._rope_cache = None
+        self._segment_boundaries = None
+
+    @property
+    def dtype(self) -> torch.dtype:
+        """Match ModelMixin's placement contract used by ModularPipeline.to()."""
+        return self.condition_proj.weight.dtype
+
+    @property
+    def device(self) -> torch.device:
+        return self.adaln_t_table.device
+
+    def load(self, path: str) -> None:
+        from safetensors import safe_open
+
+        with safe_open(path, framework="pt", device="cpu") as handle:
+            self.video_patch_proj.load(handle, "video_patch_proj")
+            self.audio_patch_proj.load(handle, "audio_patch_proj")
+            self.condition_proj.load(handle, "condition_proj")
+            for index, block in enumerate(self.token_refiner):
+                block.load(handle, f"token_refiner.blocks.{index}")
+            self.token_refiner_norm.load(handle, "token_refiner.final_norm")
+            for index, block in enumerate(self.blocks):
+                block.load(handle, f"blocks.{index}")
+            self.final_layer.load(handle, "final_layer")
+            self.adaln_t_table = handle.get_tensor("adaln_t_table")
+            self.rope_inv_freq = handle.get_tensor("rope.inv_freq")
+        # Every loaded tensor is already a frozen Parameter (or a buffer). Avoid mutating the quantized tensor
+        # subclass through a redundant requires_grad_ dispatch.
+        self.eval()
+
+    def set_attention_backend(self, backend: str) -> None:
+        self.attention_backend = backend
+
+    def begin_request(self) -> None:
+        self._text_cache = None
+        self._rope_cache = None
+        self._segment_boundaries = None
+
+    def end_request(self) -> None:
+        self.begin_request()
+
+    def _refine_text(self, text_states: torch.Tensor) -> torch.Tensor:
+        key = (text_states.data_ptr(), tuple(text_states.shape), text_states.device)
+        if self._text_cache is not None and self._text_cache[0] == key:
+            return self._text_cache[1]
+        hidden = self.condition_proj(text_states)
+        # Text is tiny compared with the video sequence; use the same fused QKV path with an identity RoPE omitted.
+        for block in self.token_refiner:
+            residual = hidden
+            normalized = block.norm1(hidden)
+            qkv = block.attn.qkv_proj(normalized)
+            query, key_states, value = qkv.split(HEADS * HEAD_DIM, dim=-1)
+            query = block.attn.q_norm(query.view(1, -1, HEADS, HEAD_DIM))
+            key_states = block.attn.k_norm(key_states.view(1, -1, HEADS, HEAD_DIM))
+            value = value.view(1, -1, HEADS, HEAD_DIM)
+            from diffusers.models.attention_dispatch import dispatch_attention_fn
+
+            attended = dispatch_attention_fn(
+                query,
+                key_states,
+                value,
+                attn_mask=None,
+                dropout_p=0.0,
+                is_causal=False,
+                backend=self.attention_backend,
+            ).reshape(-1, HEADS * HEAD_DIM)
+            hidden = residual + block.attn.out_proj(attended)
+            hidden = hidden + block.mlp(block.norm2(hidden))
+        hidden = self.token_refiner_norm(hidden)
+        self._text_cache = (key, hidden)
+        return hidden
+
+    def _rope(self, position_ids: torch.Tensor, dtype: torch.dtype) -> torch.Tensor:
+        key = (position_ids.data_ptr(), tuple(position_ids.shape), position_ids.device, dtype)
+        if self._rope_cache is not None and self._rope_cache[0] == key:
+            return self._rope_cache[1]
+        positions = position_ids.to(torch.float32)
+        frequencies = positions.unsqueeze(-1) * self.rope_inv_freq.to(position_ids.device).view(1, 1, -1)
+        temporal, height, width = frequencies.unbind(dim=1)
+        angles = torch.cat((temporal, height, width), dim=-1)
+        cosine, sine = angles.cos(), angles.sin()
+        table = torch.stack((cosine, -sine, sine, cosine), dim=-1)
+        table = table.reshape(1, position_ids.shape[0], 1, angles.shape[-1], 2, 2).to(dtype)
+        self._rope_cache = (key, table)
+        return table
+
+    def _time_embedding(self, timestep: torch.Tensor) -> torch.Tensor:
+        table = self.adaln_t_table.to(timestep.device)
+        position = timestep.float().clamp(0.0, 1.0) * (table.shape[0] - 1)
+        lower = position.floor().long().clamp(max=table.shape[0] - 2)
+        return torch.lerp(table[lower], table[lower + 1], (position - lower).unsqueeze(1))
+
+    def _segments(self, indices: torch.Tensor):
+        if self._segment_boundaries is None:
+            host = indices.detach().cpu()
+            changes = (host[1:] != host[:-1]).nonzero().flatten().add(1).tolist()
+            self._segment_boundaries = [0, *changes, len(host)]
+        bounds = self._segment_boundaries
+        return [(a, b, indices[a]) for a, b in zip(bounds[:-1], bounds[1:])]
+
+    @staticmethod
+    def _modulate(hidden, shift, scale, segments):
+        for start, stop, row in segments:
+            hidden[start:stop].mul_(1.0 + scale[row].to(hidden.dtype)).add_(shift[row].to(hidden.dtype))
+        return hidden
+
+    @staticmethod
+    def _gate(hidden, update, gate, segments):
+        for start, stop, row in segments:
+            hidden[start:stop].addcmul_(update[start:stop], gate[row].to(hidden.dtype))
+        return hidden
+
+    def forward(
+        self,
+        hidden_states,
+        audio_hidden_states,
+        encoder_hidden_states,
+        timestep,
+        timestep_indices,
+        token_tags,
+        position_ids,
+        video_indices,
+        audio_indices,
+        text_indices,
+        attention_kwargs=None,
+        return_dict=True,
+    ):
+        from diffusers.models.transformers.transformer_minimax_h3 import MiniMaxH3TransformerOutput
+
+        if hidden_states.shape[0] != 1:
+            raise ValueError("The NVFP4 MiniMax-H3 engine supports batch size 1.")
+
+        text = self._refine_text(encoder_hidden_states[0].to(torch.bfloat16))
+        video = self.video_patch_proj(hidden_states[0].float()).to(text.dtype)
+        audio = self.audio_patch_proj(audio_hidden_states[0].float()).to(text.dtype)
+        packed = text.new_zeros((position_ids.shape[0], HIDDEN))
+        packed.index_copy_(0, text_indices, text)
+        packed.index_copy_(0, video_indices, video)
+        packed.index_copy_(0, audio_indices, audio)
+
+        time_embedding = self._time_embedding(timestep)
+        adaln_indices = timestep_indices * 3 + token_tags.clamp(min=0)
+        segments = self._segments(adaln_indices)
+        rope = self._rope(position_ids, packed.dtype)
+
+        for block in self.blocks:
+            shift_attn, scale_attn, gate_attn, shift_mlp, scale_mlp, gate_mlp = block.adaln_proj(time_embedding)
+            normalized = self._modulate(block.norm1(packed), shift_attn, scale_attn, segments)
+            packed = self._gate(
+                packed,
+                block.attn(normalized, rope, self.attention_backend),
+                gate_attn,
+                segments,
+            )
+            normalized = self._modulate(block.norm2(packed), shift_mlp, scale_mlp, segments)
+            packed = self._gate(packed, block.mlp(normalized), gate_mlp, segments)
+
+        normalized = self.final_layer.norm(packed)
+        shift, scale = self.final_layer.adaln_proj(time_embedding)
+
+        video_times = timestep_indices.index_select(0, video_indices)
+        video_hidden = normalized.index_select(0, video_indices)
+        video_hidden = video_hidden * (1.0 + scale.index_select(0, video_times)) + shift.index_select(0, video_times)
+        video_output = self.final_layer.video_out(video_hidden.float()).unsqueeze(0)
+
+        audio_times = timestep_indices.index_select(0, audio_indices)
+        audio_hidden = normalized.index_select(0, audio_indices)
+        audio_hidden = audio_hidden * (1.0 + scale.index_select(0, audio_times)) + shift.index_select(0, audio_times)
+        audio_output = self.final_layer.audio_out(audio_hidden.float()).unsqueeze(0)
+
+        if not return_dict:
+            return video_output, audio_output
+        return MiniMaxH3TransformerOutput(sample=video_output, audio_sample=audio_output)
+
+
+def load_transformer() -> H3NVFP4Transformer:
+    if torch.version.cuda is None or int(torch.version.cuda.split(".")[0]) < 13:
+        raise RuntimeError("NVFP4 requires the CUDA 13 PyTorch build.")
+    from huggingface_hub import hf_hub_download
+
+    path = hf_hub_download(repo_id=NVFP4_REPO, filename=NVFP4_FILE)
+    transformer = H3NVFP4Transformer()
+    transformer.load(path)
+    print(f"[h3-nvfp4] loaded {NVFP4_REPO}/{NVFP4_FILE}", flush=True)
+    return transformer
+
+
+def status() -> str:
+    return f"NVFP4 · pruned AdaLN curve · fused QKV/QK-norm/RoPE · `{NVFP4_REPO}`"

h3_split_blocks.py

@@ -0,0 +1,147 @@
+"""The halves of a **split** MiniMax-H3 deployment, for both of its checkpoint partitions.
+
+MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so `MiniMaxH3Blocks` is cut
+at its `text_encoder` step: the 62.14 GiB Qwen3-VL runs in the conditioner Space, everything else in a generator
+Space, and `prompt_embeds` + `text_token_tags` is the whole wire format between them.
+
+`resize` / `setup` run on **both** sides: they own no pretrained component, and each half needs the canvas and the
+prepared keyframes or normalized references. Both conditioner halves also return the resolved `height` / `width` /
+`num_frames`, which the generating half pins rather than re-deriving.
+
+Two things the blocks leave to the caller: a keyframe reaches them EXIF-transposed and in RGB, and the `t2va` / `fl2va`
+frame count is aligned to `17 * n + 5` before the call, since that arithmetic lives on the denoising side of the cut.
+"""
+
+from diffusers.modular_pipelines.minimax_h3.before_encoder import MiniMaxH3Ref2VASetupStep
+from diffusers.modular_pipelines.minimax_h3.decoders import MiniMaxH3AfterDenoiseStep
+from diffusers.modular_pipelines.minimax_h3.encoders import (
+    MiniMaxH3Ref2VAReferenceEncoderStep,
+    MiniMaxH3Ref2VATextEncoderStep,
+    MiniMaxH3TextEncoderStep,
+)
+from diffusers.modular_pipelines.minimax_h3.modular_blocks_minimax_h3 import (
+    MiniMaxH3AutoKeyframeVaeEncoderStep,
+    MiniMaxH3AutoResizeStep,
+    MiniMaxH3CoreDenoiseStep,
+    MiniMaxH3DecodeStep,
+    MiniMaxH3Ref2VACoreDenoiseStep,
+    _generation_outputs,
+)
+from diffusers.modular_pipelines.modular_pipeline import SequentialPipelineBlocks
+from diffusers.modular_pipelines.modular_pipeline_utils import OutputParam
+
+
+def _wire_outputs(num_frames: bool = True) -> list[OutputParam]:
+    """The wire format of the split. `num_frames` is declared by the `ref2va` half alone, whose setup resolves one."""
+    return [
+        OutputParam.template("prompt_embeds"),
+        OutputParam("text_token_tags", description="The per-row modality tag of every row of `prompt_embeds`."),
+        OutputParam("height", type_hint=int, description="Resolved height of the generated video in pixels."),
+        OutputParam("width", type_hint=int, description="Resolved width of the generated video in pixels."),
+        *(
+            [OutputParam("num_frames", type_hint=int, description="Resolved number of frames, of the form 17 * n + 5.")]
+            if num_frames
+            else []
+        ),
+    ]
+
+
+class MiniMaxH3ConditionerBlocks(SequentialPipelineBlocks):
+    """The conditioner half of a split MiniMax-H3: the keyframes on the canvas plus the Qwen3-VL read at layer 50."""
+
+    model_name = "minimax-h3"
+    block_classes = [MiniMaxH3AutoResizeStep, MiniMaxH3TextEncoderStep]
+    block_names = ["resize", "text_encoder"]
+
+    @property
+    def description(self):
+        return (
+            "The conditioner half of a split MiniMax-H3 deployment: puts the keyframes onto the target canvas and "
+            "encodes MiniMax-H3's presentation of the request into the `prompt_embeds` / `text_token_tags` pair the "
+            "denoising half consumes. The frame count is the caller's to align."
+        )
+
+    @property
+    def outputs(self):
+        return _wire_outputs(num_frames=False)
+
+
+class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
+    """The denoising half of a split MiniMax-H3: `MiniMaxH3Blocks` with its `text_encoder` step removed."""
+
+    model_name = "minimax-h3"
+    block_classes = [
+        MiniMaxH3AutoResizeStep,
+        MiniMaxH3AutoKeyframeVaeEncoderStep,
+        MiniMaxH3CoreDenoiseStep,
+        MiniMaxH3AfterDenoiseStep,
+        MiniMaxH3DecodeStep,
+    ]
+    block_names = ["resize", "vae_encoder", "denoise", "after_denoise", "decode"]
+
+    @property
+    def description(self):
+        return (
+            "The denoising half of a split MiniMax-H3 deployment: the `t2va` / `fl2va` branch of `MiniMaxH3Blocks` "
+            "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
+            "62.14 GiB Qwen3-VL conditioner is never loaded here."
+        )
+
+    @property
+    def outputs(self):
+        return _generation_outputs()
+
+
+class MiniMaxH3Ref2VAConditionerBlocks(SequentialPipelineBlocks):
+    """The conditioner half of a split `ref2va`: the resolved plan plus the Qwen3-VL read at its 50th layer.
+
+    Component for component this is `MiniMaxH3ConditionerBlocks`, so one conditioner Space serves both partitions.
+    What differs is the presentation: `ref2va` prepends a label per reference and a vision block per image and per
+    merged video frame pair, so the references themselves have to reach this half.
+    """
+
+    model_name = "minimax-h3"
+    block_classes = [MiniMaxH3Ref2VASetupStep, MiniMaxH3Ref2VATextEncoderStep]
+    block_names = ["setup", "text_encoder"]
+
+    @property
+    def description(self):
+        return (
+            "The conditioner half of a split MiniMax-H3 `ref2va` deployment: resolves the request plan (canvas, frame "
+            "count, references normalized onto MiniMax-H3's own rates and resolutions) and encodes MiniMax-H3's "
+            "presentation of it into the `prompt_embeds` / `text_token_tags` pair the denoising half consumes."
+        )
+
+    @property
+    def outputs(self):
+        return _wire_outputs()
+
+
+class MiniMaxH3Ref2VAGeneratorBlocks(SequentialPipelineBlocks):
+    """The denoising half of a split `ref2va`: the `ref2va` branch with its `text_encoder` step removed.
+
+    `reference_encoder` stays here, next to the two autoencoders it runs: its output shapes are where every reference
+    block's geometry in the packed layout comes from.
+    """
+
+    model_name = "minimax-h3"
+    block_classes = [
+        MiniMaxH3Ref2VASetupStep,
+        MiniMaxH3Ref2VAReferenceEncoderStep,
+        MiniMaxH3Ref2VACoreDenoiseStep,
+        MiniMaxH3AfterDenoiseStep,
+        MiniMaxH3DecodeStep,
+    ]
+    block_names = ["setup", "reference_encoder", "denoise", "after_denoise", "decode"]
+
+    @property
+    def description(self):
+        return (
+            "The denoising half of a split MiniMax-H3 `ref2va` deployment: the `ref2va` branch of `MiniMaxH3Blocks` "
+            "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
+            "62.14 GiB Qwen3-VL conditioner is never loaded here. The transformer is the `transformer_ref` partition."
+        )
+
+    @property
+    def outputs(self):
+        return _generation_outputs()

packages.txt

@@ -0,0 +1 @@
+ffmpeg

requirements.txt

@@ -0,0 +1,29 @@
+# `diffusers` is installed from the canonical MiniMax-H3 pull request,
+# https://github.com/huggingface/diffusers/pull/14371 ("Minimax h3 follow up (review & refactor)"), pinned to a
+# **commit** rather than to its `minimax-h3-refactor` branch: the PR is a WIP and its head moves, and this Space's
+# blocks subclass its block classes. Re-pin — and re-check `h3_split_blocks.py` against the block names of the new
+# head — whenever the PR updates.
+#
+#   665f578278365ea4a3318cb8c9b66ce6c01204b9 = refs/pull/14371/head at the time of this deploy
+--extra-index-url https://download.pytorch.org/whl/cu130
+diffusers @ git+https://github.com/huggingface/diffusers.git@665f578278365ea4a3318cb8c9b66ce6c01204b9
+torch==2.11.0
+torchvision==0.26.0
+# The Qwen3-VL processor decides the vision patch count, so a different minor changes the conditioning.
+transformers==5.8.0
+accelerate==1.14.0
+# diffusers pins <2.
+huggingface-hub==1.24.0
+gradio==6.20.0
+spaces==0.51.1
+# Blackwell-native NVFP4 GEMMs and the fused Q/K RMSNorm + split-half RoPE kernel used by h3_nvfp4.py.
+# CUDA 13 is mandatory: older builds emulate this path and are slower than BF16.
+comfy-kitchen==0.2.26
+# No `kernels` pin on purpose: the Hub attention backends want `kernels>=0.12.3`, and that version breaks
+# transformers 5.8.0 at import.
+# PyAV muxes the generated soundtrack onto the frames (`encode_video`).
+av
+pillow
+numpy
+requests
+safetensors>=0.8.0

spaces_constant_binding_patch.py

@@ -0,0 +1,202 @@
+"""Bind AoTI constants that `torch.export` lifted anonymously.
+
+`spaces.zero.torch.aoti.LazyAOTIModel` binds a package's constants by intersecting the module's `state_dict()` with
+`compiled_model.get_constant_fqns()`, and keeps whatever it cannot match. `torch.export` only gives a lifted tensor a
+real FQN when it was a registered parameter or buffer; anything else is named `_tensor_constant<N>`, which no
+`state_dict()` can contain, so the compiled model runs against constants nobody set — a SIGSEGV rather than an error.
+
+`write_constant_aliases` records the real names on the compile side; `apply_spaces_constant_binding_patch` uses that
+sidecar on the load side, falls back to matching by dtype+shape, and raises if the binding is still not total.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import re
+import zipfile
+from pathlib import Path
+
+import torch
+
+ALIASES_FILENAME = "constant_aliases.json"
+
+_DTYPES = {
+    "float32": torch.float32, "float64": torch.float64, "float16": torch.float16,
+    "bfloat16": torch.bfloat16, "float8_e4m3fn": torch.float8_e4m3fn,
+    "float8_e5m2": torch.float8_e5m2, "float8_e4m3fnuz": torch.float8_e4m3fnuz,
+    "float8_e5m2fnuz": torch.float8_e5m2fnuz, "int8": torch.int8, "uint8": torch.uint8,
+    "int16": torch.int16, "int32": torch.int32, "int64": torch.int64, "bool": torch.bool,
+}
+
+
+# --------------------------------------------------------------------------- compile side
+
+
+def register_loose_tensors(module: torch.nn.Module, prefix: str = "") -> list[str]:
+    """Re-register plain tensor attributes as buffers so `torch.export` gives them real FQNs.
+
+    Run on the shallow clone, right before `torch.export.export`. Returns the names it re-registered.
+    """
+    registered = []
+    for name, value in list(vars(module).items()):
+        if not isinstance(value, torch.Tensor) or name.startswith("_"):
+            continue
+        if name in module._parameters or name in module._buffers:
+            continue
+        object.__delattr__(module, name)
+        module.register_buffer(name, value, persistent=True)
+        registered.append(f"{prefix}{name}")
+    for child_name, child in module.named_children():
+        registered += register_loose_tensors(child, f"{prefix}{child_name}.")
+    return registered
+
+
+def constant_aliases_from_exported_program(exported_program) -> dict[str, str]:
+    """`{'_tensor_constant<N>': '<real dotted fqn>'}` for every anonymously lifted constant.
+
+    AOT Inductor numbers its slots in the order the `CONSTANT_TENSOR` inputs appear in the graph signature, which
+    still carries each one's real FQN.
+    """
+    targets = [
+        spec.target
+        for spec in exported_program.graph_signature.input_specs
+        if spec.kind.name == "CONSTANT_TENSOR"
+    ]
+    return {f"_tensor_constant{index}": target for index, target in enumerate(targets)}
+
+
+def write_constant_aliases(package_dir, exported_program, submodule: str | None = None) -> Path | None:
+    """Drop the alias sidecar next to the `package.pt2` `aoti_compile_and_save` just wrote."""
+    aliases = constant_aliases_from_exported_program(exported_program)
+    if not aliases:
+        return None
+    subdir = Path(package_dir) / ("submodules/" + submodule if submodule else "root")
+    path = subdir / ALIASES_FILENAME
+    path.write_text(json.dumps(aliases, indent=2))
+    return path
+
+
+# --------------------------------------------------------------------------- load side
+
+
+def _package_constants_info(archive_file) -> list[dict]:
+    """Read `constants_info_` (dtype, shape, in slot order) out of a `.pt2`'s wrapper source."""
+    if isinstance(archive_file, (str, Path)):
+        handle: object = str(archive_file)
+    else:
+        position = archive_file.tell()
+        archive_file.seek(0)
+        handle = io.BytesIO(archive_file.read())
+        archive_file.seek(position)
+    with zipfile.ZipFile(handle) as archive:  # pyright: ignore[reportArgumentType]
+        names = [n for n in archive.namelist() if n.endswith(".wrapper.cpp")]
+        if not names:
+            return []
+        source = archive.read(names[0]).decode()
+    info: dict[int, dict] = {}
+    for match in re.finditer(r"constants_info_\[(\d+)\]\.(\w+) = ([^;]+);", source):
+        index, field, value = int(match.group(1)), match.group(2), match.group(3).strip()
+        entry = info.setdefault(index, {})
+        if field == "dtype":
+            entry["dtype"] = _DTYPES.get(value.replace("cached_torch_dtype_", ""))
+        elif field == "shape":
+            entry["shape"] = tuple(int(x) for x in re.findall(r"-?\d+", value))
+        elif field in ("name", "original_fqn"):
+            entry[field] = value.strip('"')
+    return [info[index] for index in sorted(info)]
+
+
+def resolve_constant_map(
+    archive_file,
+    constant_fqns,
+    weights: dict[str, torch.Tensor],
+    aliases=None,
+    allow_shape_fallback: bool = False,
+):
+    """Map every compiled constant FQN onto one of `weights`, or report what is left over."""
+    constant_map = {name: weights[name] for name in constant_fqns if name in weights}
+    missing = [name for name in constant_fqns if name not in constant_map]
+    if not missing:
+        return constant_map, []
+
+    aliases = aliases or {}
+    for name in list(missing):
+        target = aliases.get(name)
+        if target is not None and target in weights:
+            constant_map[name] = weights[target]
+            missing.remove(name)
+    if not missing or not allow_shape_fallback:
+        return constant_map, missing
+
+    # Match by dtype+shape against the unclaimed `state_dict()` entries, preserving each side's own order inside a
+    # (dtype, shape) group. `get_constant_fqns()` returns slots lexicographically (`_tensor_constant10` before
+    # `_tensor_constant2`), so the package's own `constants_info_` index is the only correct order to walk them in.
+    info = _package_constants_info(archive_file)
+    by_name = {entry.get("name"): entry for entry in info}
+    slot_index = {entry.get("name"): index for index, entry in enumerate(info)}
+    taken = {id(tensor) for tensor in constant_map.values()}
+    buckets: dict[tuple, list[torch.Tensor]] = {}
+    for tensor in weights.values():
+        if id(tensor) not in taken:
+            buckets.setdefault((tensor.dtype, tuple(tensor.shape)), []).append(tensor)
+    for name in sorted(list(missing), key=lambda n: slot_index.get(n, 1 << 30)):
+        entry = by_name.get(name)
+        if entry is None or entry.get("dtype") is None:
+            continue
+        bucket = buckets.get((entry["dtype"], entry["shape"]))
+        if bucket:
+            constant_map[name] = bucket.pop(0)
+            missing.remove(name)
+    return constant_map, missing
+
+
+def apply_spaces_constant_binding_patch(strict: bool = True, allow_shape_fallback: bool = False):
+    """Make `spaces`' AoTI loader bind anonymous constants, and fail loudly if it still cannot.
+
+    Call once, before any `spaces.aoti_*` loading. Idempotent.
+    """
+    from spaces.zero.torch import aoti as spaces_aoti
+
+    if getattr(spaces_aoti.LazyAOTIModel, "_constant_binding_patched", False):
+        return
+
+    original_call = spaces_aoti.LazyAOTIModel.__call__
+
+    def patched_call(self, weights, check_full_update, *args, **kwargs):
+        compiled_model = self.compiled_model.get()
+        if compiled_model is None:
+            with spaces_aoti._register_aoti_cleanup():
+                compiled_model = torch._inductor.aoti_load_package(self.archive_file)
+            self.compiled_model.set(compiled_model)
+        loaded = self.loaded_weights.get()
+        if loaded is None or loaded is not weights:
+            fqns = compiled_model.get_constant_fqns()
+            aliases = getattr(self, "_constant_aliases", None)
+            if aliases is None:
+                aliases = {}
+                if isinstance(self.archive_file, (str, Path)):
+                    sidecar = Path(self.archive_file).with_name(ALIASES_FILENAME)
+                    if sidecar.is_file():
+                        aliases = json.loads(sidecar.read_text())
+                self._constant_aliases = aliases
+            constant_map, missing = resolve_constant_map(
+                self.archive_file, fqns, weights, aliases, allow_shape_fallback
+            )
+            if missing and strict:
+                raise RuntimeError(
+                    f"{len(missing)} of {len(fqns)} AoTI constants could not be bound to the module's "
+                    f"state_dict: {missing[:8]}. Anonymous `_tensor_constant*` names mean the export saw "
+                    f"plain tensor attributes rather than registered parameters or buffers. Register them "
+                    f"(or write a {ALIASES_FILENAME} sidecar at compile time) — binding them partially "
+                    f"would leave the compiled model dereferencing unset constants."
+                )
+            compiled_model.load_constants(
+                constant_map, check_full_update=check_full_update and not missing, user_managed=True
+            )
+            self.loaded_weights.set(weights)
+        return compiled_model(*args, **kwargs)
+
+    spaces_aoti.LazyAOTIModel.__call__ = patched_call
+    spaces_aoti.LazyAOTIModel._constant_binding_patched = True
+    spaces_aoti.LazyAOTIModel._original_call = original_call