Localize the H3 conditioner

README.md

@@ -1,5 +1,5 @@
 ---
-title: MiniMax H3 Ultra
+title: MiniMax-H3 Ultra Fast
 emoji: ⚡
 colorFrom: purple
 colorTo: indigo
@@ -7,11 +7,11 @@ sdk: gradio
 sdk_version: 6.20.0
 app_file: app.py
 pinned: true
-short_description: Blackwell-native NVFP4 video + synchronized audio generation
+short_description: Ultra-fast local NVFP4 video + synchronized audio generation
 suggested_hardware: zero-a10g
 ---
 
-# MiniMax-H3 Ultra — pruned NVFP4 on Blackwell
+# MiniMax-H3 Ultra Fast — local conditioner + pruned NVFP4 on Blackwell
 
 Joint video and synchronized sound from MiniMax-H3, with the repeatedly executed transformer rebuilt around the
 Blackwell-native ComfyUI optimization path:
@@ -22,6 +22,8 @@ Blackwell-native ComfyUI optimization path:
 - One fused QKV projection per attention layer.
 - One fused in-place Q/K RMSNorm + partial split-half RoPE kernel.
 - Native CUDA 13 NVFP4 tensor-core GEMMs through `comfy-kitchen`.
+- A local 15.7 GB Qwen3-VL NVFP4-AWQ conditioner replaces the normal cross-Space API call. It contains exactly the
+  first 50 language layers H3 reads, with the unused 14-layer tail and vocabulary head removed.
 - A conservative EasyCache-style trajectory estimator skips complete 50-layer denoiser evaluations when successive
   latent changes are predictable; it is request-local and is not prompt-to-video output caching.
 - Segment-wise in-place AdaLN modulation and gated residual accumulation.
@@ -67,20 +69,23 @@ Blackwell at 864×480, 39 frames. Those numbers are useful implementation eviden
 H3 attention grows quadratically with packed sequence length, so resolution, duration and keyframe vision tokens
 still dominate large requests.
 
-## Split deployment
+## One-Space deployment
 
-The full checkpoint cannot fit under a single Space's 150 GB storage ceiling. This Space remains the denoising half:
+The original BF16 deployment had to be split because its 62.14 GiB conditioner plus 61.7 GiB transformer could not
+fit comfortably under one Space's storage and runtime limits. The pruned formats change that calculation:
 
 | component | where it runs | precision / format |
 |---|---|---|
-| Qwen3-VL layer-50 conditioner | [`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner) | BF16 |
+| Qwen3-VL layer-50 conditioner | this Space | truncated NVFP4-AWQ weights / BF16 GEMMs + BF16 vision tower |
 | H3 transformer | this Space | pruned NVFP4 + higher-precision islands |
 | video VAE | this Space | full precision checkpoint policy |
 | audio VAE | this Space | FP32 |
 
-The wire format is `prompt_embeds` plus `text_token_tags` in a safetensors file. The conditioner also returns the
-resolved canvas, aligned frame count and prompt plan. Keyframes are encoded again by this Space's video VAE so the
-conditioning latents exactly match the pixels seen by the conditioner.
+The embeddings stay on the same GPU worker and flow directly into H3: there is no Gradio round trip, second queue,
+safetensors serialization, or user-quota handoff during normal generation. Prompt upsampling still uses the remote
+BF16 conditioner because rewriting requires the language-model head and the 14 decoder layers deliberately removed
+from the local inference-only checkpoint. The remote service also remains an automatic fallback if local loading
+fails.
 
 ## Kernel path
 
@@ -105,7 +110,8 @@ not the default.
 
 ## Quality trade-off
 
-NVFP4 and adaptive step reuse are approximate. The checkpoint author reports that 4-bit weights can show more
+The transformer, local conditioner, and adaptive step reuse are approximate. The checkpoint author reports that
+4-bit weights can show more
 mid-motion artifacts and weaker shape retention than the larger INT8 ConvRot checkpoint on difficult 15-second
 clips. The comparison was not fully controlled, so treat it as a real caution rather than a quantified quality
 score. Adaptive reuse adds another speed/quality trade-off; lower `H3_EASYCACHE_THRESHOLD` for motion-sensitive
@@ -122,7 +128,11 @@ exact original denoiser, set `H3_ENGINE=bf16`; this restores the 61.7 GiB unquan
 | `H3_NVFP4_REPO` | `lilcheaty/MiniMax-H3-NVFP4` | Repository containing the pruned Comfy-format transformer. |
 | `H3_NVFP4_FILE` | `minimax_h3_fl2va_pruned_nvfp4.safetensors` | FL2VA/T2VA transformer file. |
 | `H3_MODEL_REPO` | `MiniMaxAI/MiniMax-H3` | Canonical schedulers and VAE checkpoint. |
-| `H3_CONDITIONER` | `multimodalart/qwen3vl-conditioner` | Remote layer-50 conditioner Space. |
+| `H3_CONDITIONER_MODE` | `local` | Use the local truncated conditioner; `remote` restores the split deployment. |
+| `H3_LOCAL_CONDITIONER_REPO` | `Comfy-Org/MiniMax-H3` | Repository containing the truncated conditioner. |
+| `H3_LOCAL_CONDITIONER_FILE` | `text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` | Local layer-50 conditioner checkpoint. |
+| `H3_CONDITIONER_NATIVE_NVFP4` | `0` | Use native W4A4 conditioner GEMMs for maximum speed; default BF16 GEMMs add less activation error. |
+| `H3_CONDITIONER` | `multimodalart/qwen3vl-conditioner` | Prompt-upsampling and local-load fallback service. |
 | `H3_PLACEMENT` | `lazy` (`nvfp4`) | Move the compact transformer and VAEs on the first GPU call, then keep them resident. |
 | `H3_ATTENTION` | `_native_cudnn` | Attention backend for both the main stack and text refiner. |
 | `H3_EASYCACHE_THRESHOLD` | `0.10` | Maximum accumulated estimated change before a full denoiser evaluation; `0` disables reuse. |
@@ -139,11 +149,14 @@ exact original denoiser, set `H3_ENGINE=bf16`; this restores the 61.7 GiB unquan
 - `comfy-kitchen==0.2.26` for the native layouts and fused Q/K kernel.
 - The pinned MiniMax-H3 diffusers pull request for the modular schedulers, packing and VAE decode path.
 
-No secret is required. All model artifacts are public, and `gradio_client` forwards the requesting user's ZeroGPU
-identity to the conditioner.
+No secret is required. All model artifacts are public. Normal requests use one local GPU booking; a prompt-upsampling
+request forwards the requesting user's ZeroGPU identity to the remote rewriter/conditioner.
 
 ## Attribution
 
+This is an optimized derivative of the original
+[`multimodalart/minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3) Space.
+
 The fused/pruned model structure follows
 [`comfy/ldm/minimax/model.py`](https://github.com/Comfy-Org/ComfyUI/blob/master/comfy/ldm/minimax/model.py) from
 ComfyUI (Apache-2.0), and the adaptive residual estimator follows ComfyUI's built-in

app.py

@@ -1,4 +1,4 @@
-"""MiniMax-H3 `t2va` / `fl2va`, split deployment — the denoising half."""
+"""MiniMax-H3 Ultra Fast: local layer-50 conditioning plus `t2va` / `fl2va` generation."""
 
 from __future__ import annotations
 
@@ -15,6 +15,7 @@ import gradio as gr
 
 MODEL_REPO = os.environ.get("H3_MODEL_REPO", "MiniMaxAI/MiniMax-H3")
 CONDITIONER_SPACE = os.environ.get("H3_CONDITIONER", "multimodalart/qwen3vl-conditioner")
+CONDITIONER_MODE = os.environ.get("H3_CONDITIONER_MODE", "local").lower()
 # `nvfp4` is the Blackwell-native ultra path; `bf16` preserves the original 33B diffusers transformer as a fallback.
 ENGINE = os.environ.get("H3_ENGINE", "nvfp4").lower()
 # `pack` places the transformer at startup, `lazy` moves everything on the first GPU call, `offload` hands placement to
@@ -73,6 +74,8 @@ def lower_duration_floor(seconds: float = MIN_UI_DURATION) -> None:
 
 PIPE = None
 MANAGER = None
+COND_PIPE = None
+COND_ERROR: str | None = None
 LOAD_ERROR: str | None = None
 LOADED_IN: float | None = None
 
@@ -82,7 +85,7 @@ def status() -> str:
         return LOAD_ERROR
     if PIPE is None:
         payload = (
-            "pruned NVFP4 transformer + full-precision VAEs (~28 GB)"
+            "pruned NVFP4 transformer + local NVFP4 conditioner + full-precision VAEs (~44 GB)"
             if ENGINE == "nvfp4"
             else "BF16 transformer + VAEs (77.3 GB)"
         )
@@ -95,21 +98,27 @@ def status() -> str:
         import h3_aoti
 
         engine_status = f"BF16, unquantized · {h3_aoti.status()}"
+    if COND_PIPE is not None:
+        import h3_local_conditioner
+
+        conditioner_status = h3_local_conditioner.status()
+    else:
+        conditioner_status = f"remote `{CONDITIONER_SPACE}`" + (" (local fallback)" if COND_ERROR else "")
     return (
         f"Ready · **{engine_status}** · VAEs full precision · placement `{PLACEMENT}` · attention `{ATTENTION}` · "
-        f"loaded in {LOADED_IN:.0f}s · conditioner `{CONDITIONER_SPACE}`"
+        f"loaded in {LOADED_IN:.0f}s · conditioner {conditioner_status}"
     )
 
 
 def load_models() -> str | None:
-    """Load the denoising half at startup.
+    """Load the compact generator and, by default, its local truncated conditioner at startup.
 
     `MiniMaxH3GeneratorBlocks` declares `transformer`, `vae`, `audio_vae`, the two schedulers and `video_processor`,
     so `load_components` fetches exactly those subfolders — `text_encoder/` and `transformer_ref/` are never touched.
     Both autoencoders carry `_keep_in_fp32_modules` over every module and stay float32: a bfloat16 audio VAE decodes
     the soundtrack roughly 20 dB too quiet.
     """
-    global PIPE, MANAGER, LOAD_ERROR, LOADED_IN
+    global PIPE, MANAGER, COND_PIPE, COND_ERROR, LOAD_ERROR, LOADED_IN
 
     if PIPE is not None or LOAD_ERROR is not None:
         return LOAD_ERROR
@@ -159,7 +168,28 @@ def load_models() -> str | None:
             manager.enable_auto_cpu_offload(device="cuda")
             _arm_decode_hooks(pipe)
 
-        PIPE, MANAGER = pipe, manager
+        cond_pipe = None
+        if CONDITIONER_MODE == "local":
+            try:
+                from h3_local_conditioner import load_local_conditioner
+                from h3_split_blocks import MiniMaxH3ConditionerBlocks
+
+                print("[cond] loading the local truncated NVFP4-AWQ conditioner ...", flush=True)
+                text_encoder, tokenizer, processor = load_local_conditioner()
+                cond_pipe = MiniMaxH3ConditionerBlocks().init_pipeline(MODEL_REPO)
+                cond_pipe.update_components(
+                    text_encoder=text_encoder,
+                    tokenizer=tokenizer,
+                    processor=processor,
+                )
+            except Exception as error:
+                traceback.print_exc()
+                COND_ERROR = f"{type(error).__name__}: {error}"
+                print(f"[cond] local load failed ({COND_ERROR}); retaining the remote fallback", flush=True)
+        elif CONDITIONER_MODE != "remote":
+            raise ValueError(f"H3_CONDITIONER_MODE must be `local` or `remote`, got {CONDITIONER_MODE!r}")
+
+        PIPE, MANAGER, COND_PIPE = pipe, manager, cond_pipe
         LOADED_IN = time.time() - started
         print(f"[gen] ready in {LOADED_IN:.0f}s", flush=True)
     except Exception as error:
@@ -225,31 +255,52 @@ _DECODE_BASE, _DECODE_PER_DEFAULT_CANVAS, _DEFAULT_CANVAS_PIXELS = 15, 15, 960 *
 _PLACEMENT_ALLOWANCE, _PAD = 12, 10
 
 
-def get_duration(prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, steps, seed, *a, **k):
+def get_duration(prompt, prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, steps, seed, *a, **k):
     height, width, num_frames, steps = int(height), int(width), int(num_frames), int(steps)
     latent_frames = (num_frames - LATENTS_PER_CHUNK) // FRAMES_PER_CHUNK * LATENTS_PER_CHUNK + 2
     patches = (height // 32) * (width // 32)
     rows = latent_frames * patches + (int(image is not None) + int(last_image is not None)) * patches
     denoise = steps * (_DUR_B * rows + _DUR_C * rows**2)
     decode = _DECODE_BASE + _DECODE_PER_DEFAULT_CANVAS * (height * width * num_frames) / _DEFAULT_CANVAS_PIXELS
-    return max(60, int(denoise + decode) + _PLACEMENT_ALLOWANCE + _PAD)
+    local_conditioning = 20 if prompt_embeds is None else 0
+    return max(60, int(denoise + decode) + local_conditioning + _PLACEMENT_ALLOWANCE + _PAD)
 
 
 @spaces.GPU(duration=get_duration, size=GPU_SIZE)
-def _generate(prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, steps, seed):
-    """The only thing on GPU time: the packed-sequence denoise loop and the two decoders.
+def _generate(prompt, prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, steps, seed):
+    """The only thing on GPU time: local conditioning, packed denoising and the two decoders.
 
-    Only the three generated outputs come back — a `@spaces.GPU` return crosses a process boundary by pickling, and
-    the full `PipelineState` still holds the packed latents, the rotary grid and the row indices on the card.
+    Only generated outputs and two timing scalars come back—a `@spaces.GPU` return crosses a process boundary by
+    pickling, and the full `PipelineState` still holds packed latents, the rotary grid and row indices on the card.
     """
     import torch
 
+    if COND_PIPE is not None and prompt_embeds is None:
+        COND_PIPE.text_encoder.to("cuda")
     if PLACEMENT == "lazy":
         PIPE.to("cuda")
     elif PLACEMENT == "pack":
         PIPE.vae.to("cuda")
         PIPE.audio_vae.to("cuda")
 
+    condition_seconds = None
+    num_text_tokens = None
+    if prompt_embeds is None:
+        if COND_PIPE is None:
+            raise RuntimeError(f"The local conditioner is unavailable: {COND_ERROR or 'disabled'}")
+        conditioned = time.time()
+        condition_state = COND_PIPE(
+            prompt=prompt,
+            image=image,
+            last_image=last_image,
+            height=int(height),
+            width=int(width),
+        )
+        prompt_embeds = condition_state.get("prompt_embeds")
+        text_token_tags = condition_state.get("text_token_tags")
+        condition_seconds = time.time() - conditioned
+        num_text_tokens = int(prompt_embeds.shape[1])
+
     begin_request = getattr(PIPE.transformer, "begin_request", None)
     end_request = getattr(PIPE.transformer, "end_request", None)
     if begin_request is not None:
@@ -270,7 +321,13 @@ def _generate(prompt_embeds, text_token_tags, image, last_image, height, width,
     finally:
         if end_request is not None:
             end_request()
-    return state.get("videos")[0], state.get("audio")[0].cpu(), state.get("sampling_rate")
+    return (
+        state.get("videos")[0],
+        state.get("audio")[0].cpu(),
+        state.get("sampling_rate"),
+        condition_seconds,
+        num_text_tokens,
+    )
 
 
 def generate(prompt, image_path=None, last_image_path=None, canvas=DEFAULT_CANVAS, duration=5, steps=28, seed=42, upsample=False, progress=gr.Progress(track_tqdm=True)):
@@ -287,28 +344,49 @@ def generate(prompt, image_path=None, last_image_path=None, canvas=DEFAULT_CANVA
     from diffusers.utils import encode_video
 
     num_frames = snap_frames(duration)
-
-    progress(0.0, desc=f"Upsampling the prompt on {CONDITIONER_SPACE} ..." if upsample else f"Conditioning on {CONDITIONER_SPACE} ...")
-    conditioned = time.time()
-    prompt_embeds, text_token_tags, metadata, plan = encode_remote(
-        prompt, image_path, last_image_path, canvas, num_frames, rewrite_prompt=upsample
-    )
-    condition_seconds = time.time() - conditioned
-    height, width, num_frames = (int(metadata[key]) for key in ("height", "width", "num_frames"))
-    refined = plan.get("refined_prompt") or ""
+    height, width = CANVASES[canvas]
 
     def keyframe(path):
-        # The conditioning latents encoded here have to be of the image the conditioner looked at, which it prepares
-        # exactly this way.
+        # Both local conditioner and denoiser receive the same upright RGB source; their resize blocks then apply the
+        # same target canvas independently.
         return ImageOps.exif_transpose(Image.open(path)).convert("RGB") if path else None
 
-    progress(0.1, desc=f"Denoising {steps} steps at {width}x{height}, {num_frames} frames ...")
+    first_frame, final_frame = keyframe(image_path), keyframe(last_image_path)
+    prompt_embeds = text_token_tags = None
+    condition_seconds = None
+    num_text_tokens = None
+    refined = ""
+
+    # Prompt rewriting needs the discarded LM head and decoder tail, so it intentionally retains the remote path.
+    # Normal generation—the default—keeps embeddings on this worker and never serializes them through another API.
+    if upsample or COND_PIPE is None:
+        progress(
+            0.0,
+            desc=f"Upsampling and conditioning on {CONDITIONER_SPACE} ..."
+            if upsample
+            else f"Local conditioner unavailable; using {CONDITIONER_SPACE} ...",
+        )
+        conditioned = time.time()
+        prompt_embeds, text_token_tags, metadata, plan = encode_remote(
+            prompt, image_path, last_image_path, canvas, num_frames, rewrite_prompt=upsample
+        )
+        condition_seconds = time.time() - conditioned
+        height, width, num_frames = (int(metadata[key]) for key in ("height", "width", "num_frames"))
+        num_text_tokens = int(plan["num_text_tokens"])
+        refined = plan.get("refined_prompt") or ""
+
+    progress(
+        0.1,
+        desc=("Local conditioning + " if prompt_embeds is None else "")
+        + f"denoising {steps} steps at {width}x{height}, {num_frames} frames ...",
+    )
     started = time.time()
-    frames, audio, sampling_rate = _generate(
+    frames, audio, sampling_rate, local_condition_seconds, local_num_text_tokens = _generate(
+        prompt,
         prompt_embeds,
         text_token_tags,
-        keyframe(image_path),
-        keyframe(last_image_path),
+        first_frame,
+        final_frame,
         height,
         width,
         num_frames,
@@ -316,6 +394,10 @@ def generate(prompt, image_path=None, last_image_path=None, canvas=DEFAULT_CANVA
         seed,
     )
     generate_seconds = time.time() - started
+    if local_condition_seconds is not None:
+        condition_seconds = local_condition_seconds
+        num_text_tokens = local_num_text_tokens
+    denoise_seconds = generate_seconds - (local_condition_seconds or 0.0)
 
     directory = os.path.join(tempfile.gettempdir(), "h3-outputs")
     os.makedirs(directory, exist_ok=True)
@@ -324,9 +406,9 @@ def generate(prompt, image_path=None, last_image_path=None, canvas=DEFAULT_CANVA
 
     report = (
         f"`{width}x{height}`, {num_frames} frames ({num_frames / FPS:.3f} s), {int(steps)} steps · "
-        f"conditioner {condition_seconds:.0f}s ({plan['num_text_tokens']} tokens"
+        f"conditioner {condition_seconds:.0f}s ({num_text_tokens} tokens"
         f"{', upsampled' if refined else ''}) · "
-        f"denoise + decode {generate_seconds:.0f}s ({generate_seconds / int(steps):.1f} s/step) · seed {int(seed)}"
+        f"denoise + decode {denoise_seconds:.0f}s ({denoise_seconds / int(steps):.1f} s/step) · seed {int(seed)}"
     )
     print(f"[gen] {report}", flush=True)
     return path, report, refined, gr.update(visible=bool(refined))
@@ -371,17 +453,19 @@ def _fit_keyframe(image_path, current_canvas):
 
 load_models()
 
-INTRO = """# MiniMax-H3 Ultra
+INTRO = """# MiniMax-H3 Ultra Fast
 
 <div align="center">
   <a href="https://huggingface.co/MiniMaxAI/MiniMax-H3" target="_blank" rel="noopener"><strong>[ model ]</strong></a> &nbsp;
   <a href="https://huggingface.co/lilcheaty/MiniMax-H3-NVFP4" target="_blank" rel="noopener"><strong>[ NVFP4 ]</strong></a> &nbsp;
   <a href="https://www.minimax.io/blog/minimax-h3" target="_blank" rel="noopener"><strong>[ blog ]</strong></a> &nbsp;
-  <a href="https://huggingface.co/spaces/multimodalart/minimax-h3-reference" target="_blank" rel="noopener"><strong>[ reference to video ]</strong></a>
+  <a href="https://huggingface.co/spaces/multimodalart/minimax-h3" target="_blank" rel="noopener"><strong>[ original Space ]</strong></a>
 </div>
 
-**MiniMax-H3 Ultra** runs the pruned Blackwell-native NVFP4 transformer with fused QKV, fused Q/K norm + RoPE,
-full-precision video/audio decoders, and the original synchronized soundtrack generation.
+**MiniMax-H3 Ultra Fast** runs a local truncated Qwen3-VL conditioner and the pruned Blackwell-native NVFP4
+transformer with fused QKV, fused Q/K norm + RoPE, full-precision video/audio decoders, and synchronized sound.
+It is optimized from the original [`multimodalart/minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3)
+Space.
 """
 
 CSS = """
@@ -389,7 +473,7 @@ CSS = """
 .dark .gradio-container { color: var(--body-text-color); }
 """
 
-with gr.Blocks(title="MiniMax-H3") as demo:
+with gr.Blocks(title="MiniMax-H3 Ultra Fast") as demo:
     gr.Markdown(INTRO)
 
     with gr.Row():

h3_local_conditioner.py

@@ -0,0 +1,174 @@
+"""Local, truncated Qwen3-VL conditioner for MiniMax-H3.
+
+The canonical diffusers checkpoint stores all 64 language layers plus the LM head in BF16 (66.7 GB), although H3
+only reads the unnormalized state after layer 50.  ComfyUI's Apache-2.0 conversion removes the unused tail and head,
+keeps the vision tower in BF16, and stores the 50 language layers as NVFP4-AWQ.  This adapter loads that single
+15.7 GB file directly into Transformers' Qwen3-VL architecture and exposes the tiny contract used by diffusers.
+
+No ComfyUI application or server is launched. Preprocessing remains Transformers' canonical Qwen3-VL processor.
+By default the checkpoint's quality-oriented weight-only policy is honored: compact NVFP4-AWQ weights are
+dequantized one layer at a time for BF16 GEMMs. Native W4A4 is available as an aggressive opt-in.
+"""
+
+from __future__ import annotations
+
+import copy
+import os
+from types import SimpleNamespace
+
+import torch
+import torch.nn as nn
+
+from h3_nvfp4 import H3Linear
+
+
+CONDITIONER_REPO = os.environ.get("H3_LOCAL_CONDITIONER_REPO", "Comfy-Org/MiniMax-H3")
+CONDITIONER_FILE = os.environ.get(
+    "H3_LOCAL_CONDITIONER_FILE",
+    "text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
+)
+SOURCE_REPO = os.environ.get("H3_MODEL_REPO", "MiniMaxAI/MiniMax-H3")
+LAYERS = 50
+NATIVE_NVFP4 = os.environ.get("H3_CONDITIONER_NATIVE_NVFP4", "0") == "1"
+
+
+class QuantizedEmbedding(nn.Module):
+    """Row-wise INT8 token lookup without dequantizing the 1.56 GB BF16 vocabulary table."""
+
+    def __init__(self, handle, prefix: str):
+        super().__init__()
+        self.register_buffer("weight", handle.get_tensor(f"{prefix}.weight"))
+        self.register_buffer("scale", handle.get_tensor(f"{prefix}.weight_scale").float())
+
+    def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
+        flat = input_ids.reshape(-1)
+        values = self.weight.index_select(0, flat).reshape(*input_ids.shape, self.weight.shape[1])
+        scales = self.scale.index_select(0, flat).reshape(*input_ids.shape, 1)
+        return values.to(torch.bfloat16).mul_(scales.to(torch.bfloat16))
+
+
+class Layer50Backbone(nn.Module):
+    """Avoid retaining 50 intermediate tensors merely to satisfy diffusers' hidden-state indexing API."""
+
+    def __init__(self, core: nn.Module):
+        super().__init__()
+        self.core = core
+
+    def forward(self, *args, **kwargs):
+        kwargs.pop("output_hidden_states", None)
+        kwargs.pop("return_dict", None)
+        kwargs["use_cache"] = False
+        output = self.core(*args, **kwargs)
+        # get_qwen3vl_prompt_embeds asks for hidden_states[50]. The first 50 entries need not be materialized.
+        return SimpleNamespace(hidden_states=(None,) * LAYERS + (output.last_hidden_state,))
+
+
+class LocalH3Conditioner(nn.Module):
+    """The subset of Qwen3VLForConditionalGeneration that MiniMax-H3 actually calls."""
+
+    def __init__(self, core: nn.Module, source_config):
+        super().__init__()
+        public_config = copy.deepcopy(source_config)
+        # Diffusers rejects a nominally 50-layer model because a normal last_hidden_state is post-norm. This adapter
+        # removes the final norm and returns the raw 50th-layer state, so advertise index 50 as available explicitly.
+        public_config.text_config.num_hidden_layers = LAYERS + 1
+        self.config = public_config
+        self.model = Layer50Backbone(core)
+
+    @property
+    def dtype(self) -> torch.dtype:
+        return torch.bfloat16
+
+    @property
+    def device(self) -> torch.device:
+        return self.model.core.visual.patch_embed.proj.weight.device
+
+
+def _target_name(checkpoint_name: str) -> str:
+    if checkpoint_name.startswith("model.layers."):
+        return "language_model.layers." + checkpoint_name.removeprefix("model.layers.")
+    if checkpoint_name.startswith("visual."):
+        return checkpoint_name
+    raise KeyError(checkpoint_name)
+
+
+def _build_core(handle):
+    from accelerate import init_empty_weights
+    from transformers import Qwen3VLConfig
+    from transformers.models.qwen3_vl.modeling_qwen3_vl import Qwen3VLModel
+
+    config = Qwen3VLConfig.from_pretrained(SOURCE_REPO, subfolder="text_encoder")
+    config.text_config.num_hidden_layers = LAYERS
+    config.text_config.use_cache = False
+    config.text_config._attn_implementation = "sdpa"
+    config.vision_config._attn_implementation = "sdpa"
+
+    with init_empty_weights(include_buffers=False):
+        core = Qwen3VLModel(config)
+
+    keys = set(handle.keys())
+    embedding_prefix = "model.embed_tokens"
+    core.language_model.embed_tokens = QuantizedEmbedding(handle, embedding_prefix)
+    consumed = {
+        key for key in keys if key == f"{embedding_prefix}.comfy_quant" or key.startswith(f"{embedding_prefix}.weight")
+    }
+
+    quantized_prefixes = sorted(
+        key.removesuffix(".comfy_quant")
+        for key in keys
+        if key.startswith("model.layers.") and key.endswith(".comfy_quant")
+    )
+    if len(quantized_prefixes) != LAYERS * 7:
+        raise RuntimeError(f"Expected {LAYERS * 7} quantized language linears, found {len(quantized_prefixes)}.")
+
+    for source_prefix in quantized_prefixes:
+        target_prefix = _target_name(source_prefix)
+        parent_name, child_name = target_prefix.rsplit(".", 1)
+        parent = core.get_submodule(parent_name)
+        original = getattr(parent, child_name)
+        linear = H3Linear(original.in_features, original.out_features, bias=original.bias is not None)
+        linear.load(handle, source_prefix)
+        if NATIVE_NVFP4:
+            linear.full_precision_mm = False
+        setattr(parent, child_name, linear)
+        consumed.update(key for key in keys if key.startswith(f"{source_prefix}."))
+
+    # MiniMax-H3 consumes the raw output of layer 49. The released Comfy checkpoint intentionally has no final norm.
+    core.language_model.norm = nn.Identity()
+
+    plain_state = {}
+    for source_name in sorted(keys - consumed):
+        if source_name.startswith("visual.") or source_name.startswith("model.layers."):
+            plain_state[_target_name(source_name)] = handle.get_tensor(source_name)
+            consumed.add(source_name)
+
+    unknown = keys - consumed
+    if unknown:
+        raise RuntimeError(f"Unhandled local-conditioner tensors: {sorted(unknown)[:12]}")
+
+    core.load_state_dict(plain_state, strict=False, assign=True)
+    meta = [name for name, value in core.named_parameters() if value.is_meta]
+    if meta:
+        raise RuntimeError(f"Local conditioner still has uninitialized parameters: {meta[:12]}")
+    core.eval()
+    return core, config
+
+
+def load_local_conditioner():
+    from huggingface_hub import hf_hub_download
+    from safetensors import safe_open
+    from transformers import Qwen3VLProcessor
+
+    path = hf_hub_download(CONDITIONER_REPO, CONDITIONER_FILE)
+    with safe_open(path, framework="pt", device="cpu") as handle:
+        core, config = _build_core(handle)
+
+    processor = Qwen3VLProcessor.from_pretrained(SOURCE_REPO, subfolder="text_encoder")
+    model = LocalH3Conditioner(core, config).eval()
+    print(f"[h3-cond] loaded local layer-50 conditioner {CONDITIONER_REPO}/{CONDITIONER_FILE}", flush=True)
+    return model, processor.tokenizer, processor
+
+
+def status() -> str:
+    compute = "native W4A4" if NATIVE_NVFP4 else "BF16 GEMM"
+    return f"local layer-50 Qwen3-VL NVFP4-AWQ weights / {compute} · `{CONDITIONER_REPO}`"

h3_nvfp4.py

@@ -172,6 +172,7 @@ class H3Linear(nn.Module):
         self.register_buffer("input_scale", None)
         self.register_buffer("pre_quant_scale", None)
         self.quantized = False
+        self.full_precision_mm = False
 
     def load(self, handle, prefix: str) -> None:
         config = _quant_config(handle, prefix)
@@ -197,6 +198,7 @@ class H3Linear(nn.Module):
             quantized = QuantizedTensor(weight.to(torch.uint8), "TensorCoreNVFP4Layout", params)
             self.weight = nn.Parameter(quantized, requires_grad=False)
             self.quantized = True
+            self.full_precision_mm = bool(config.get("full_precision_matrix_mult", False))
             for name in ("input_scale", "pre_quant_scale"):
                 key = f"{prefix}.{name}"
                 if key in handle.keys():
@@ -224,6 +226,12 @@ class H3Linear(nn.Module):
                 self.bias,
             )
 
+        if self.full_precision_mm:
+            # Some AWQ checkpoints use NVFP4 as a compact weight format but deliberately retain BF16 activations and
+            # GEMMs. Dequantization is layer-local, so residency stays compact without adding activation error.
+            weight = self.weight.dequantize().to(hidden_states.dtype)
+            return F.linear(hidden_states, weight, None if self.bias is None else self.bias.to(hidden_states.dtype))
+
         from comfy_kitchen.tensor import QuantizedTensor
 
         shape = hidden_states.shape

h3_split_blocks.py

@@ -1,8 +1,8 @@
-"""The halves of a **split** MiniMax-H3 deployment, for both of its checkpoint partitions.
+"""Composable conditioner and generator halves of MiniMax-H3, for both checkpoint partitions.
 
-MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so `MiniMaxH3Blocks` is cut
-at its `text_encoder` step: the 62.14 GiB Qwen3-VL runs in the conditioner Space, everything else in a generator
-Space, and `prompt_embeds` + `text_token_tags` is the whole wire format between them.
+The blocks cut `MiniMaxH3Blocks` at its `text_encoder` step. They can run in separate Spaces with `prompt_embeds` and
+`text_token_tags` as a wire format, or sequentially in one GPU worker when a compact local conditioner fits beside
+the generator.
 
 `resize` / `setup` run on **both** sides: they own no pretrained component, and each half needs the canvas and the
 prepared keyframes or normalized references. Both conditioner halves also return the resolved `height` / `width` /
@@ -84,7 +84,7 @@ class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
         return (
             "The denoising half of a split MiniMax-H3 deployment: the `t2va` / `fl2va` branch of `MiniMaxH3Blocks` "
             "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
-            "62.14 GiB Qwen3-VL conditioner is never loaded here."
+            "conditioner is supplied by the caller or by the preceding local conditioner half."
         )
 
     @property
@@ -139,7 +139,7 @@ class MiniMaxH3Ref2VAGeneratorBlocks(SequentialPipelineBlocks):
         return (
             "The denoising half of a split MiniMax-H3 `ref2va` deployment: the `ref2va` branch of `MiniMaxH3Blocks` "
             "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
-            "62.14 GiB Qwen3-VL conditioner is never loaded here. The transformer is the `transformer_ref` partition."
+            "conditioner is supplied by the caller or preceding local half. The transformer is the `transformer_ref` partition."
         )
 
     @property