Add showcase cases, auto-captioning, and drop no-translation pose

- Reorder examples: iStock rice-field motorbike first; add two OmniWorld
character scenes (misty forest, horse trail) in a separate "with characters"
group, per the release curation.
- Move the examples above the Advanced settings so they are visible and
clickable without scrolling.
- Auto-generate a scene prompt with Qwen2-VL-2B when the description is left
blank, instead of erroring, so an image-only request still gets a caption.
The model is held on CPU and only lifted onto the GPU while captioning.
- Remove the pan_right preset (pure in-place rotation, zero translation), the
only preset with no visible camera translation.

Co-authored-by: Cursor <cursoragent@cursor.com>

.gitattributes

@@ -38,3 +38,6 @@ diffsynth/tokenizer_configs/kolors/tokenizer/vocab.txt filter=lfs diff=lfs merge
 examples/ai-airmountains.jpg filter=lfs diff=lfs merge=lfs -text
 examples/ai-middleages.jpg filter=lfs diff=lfs merge=lfs -text
 examples/ai-valley.jpg filter=lfs diff=lfs merge=lfs -text
+examples/istock-motorbike-rice-field.jpg filter=lfs diff=lfs merge=lfs -text
+examples/omni-misty-forest.jpg filter=lfs diff=lfs merge=lfs -text
+examples/omni-horse-trail.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -25,8 +25,9 @@ tags:
 [TencentARC/SCoPE](https://huggingface.co/TencentARC/SCoPE) retrofits **Wan2.2-I2V-A14B** with
 *Sightline-Coordinate Positional Encoding*: Plücker camera rays are normalised, gated and injected
 directly into the DiT's self-attention queries and keys, so an explicit 3D camera path steers the
-generated shot. Give it a first frame, a caption and a camera move, and it returns an 81-frame,
-832×480, 16 fps clip.
+generated shot. Give it a first frame and a camera move, and it returns an 81-frame,
+832×480, 16 fps clip. A caption is used for the scene content; if you leave it blank, a small
+vision-language model (`Qwen/Qwen2-VL-2B-Instruct`) writes one from your first frame automatically.
 
 * Paper / project page: <https://visual-ai.github.io/scope/>
 * Reference code: <https://github.com/TencentARC/SCoPE>
@@ -40,9 +41,10 @@ height=480, width=832, num_frames=81, sigma_shift=5.0, switch_DiT_boundary=0.9,
 camera_cfg_scale=1.0, lock_first_frame=False, tiled=False)`.
 
 Camera paths are OpenCV **camera-to-world** matrices of shape `[81, 3, 4]`, expressed relative to
-the first frame, which is the format SCoPE was trained on. The 17 presets in `trajectories/` are
+the first frame, which is the format SCoPE was trained on. The 16 presets in `trajectories/` are
 taken from the release's own `examples/` trajectory set (re-expressed relative to frame 0), and the
-"camera motion scale" slider rescales only their translation component.
+"camera motion scale" slider rescales only their translation component. Every preset carries a real
+translation; pure in-place rotations were dropped because their effect is hard to see.
 
 ## Deviations from the reference configuration
 
@@ -63,6 +65,8 @@ control itself is architectural and unaffected by the shorter schedule.
 
 ## Credits
 
-Model, code and example images © the SCoPE authors (Apache-2.0). The three showcase images in
-`examples/` are the AI-generated cases shipped in the SCoPE repository's `examples/` directory,
-together with their original captions and `x_fov`.
+Model, code and example images © the SCoPE authors (Apache-2.0). The showcase images in
+`examples/` are drawn from the SCoPE release: three AI-generated scenes, one iStock aerial clip
+(rice-field motorbike), and two OmniWorld character scenes (kept in their own "with characters"
+group). Automatic prompting uses [`Qwen/Qwen2-VL-2B-Instruct`](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct)
+(Apache-2.0).

app.py

@@ -43,6 +43,7 @@ from torchao.quantization import (  # noqa: E402
     Int8WeightOnlyConfig,
     quantize_,
 )
+from transformers import AutoProcessor, Qwen2VLForConditionalGeneration  # noqa: E402
 
 from diffsynth.data.video import save_video  # noqa: E402
 from diffsynth.models import ModelManager  # noqa: E402
@@ -60,6 +61,14 @@ HERE = Path(__file__).resolve().parent
 REPO_ID = "TencentARC/SCoPE"
 LORA_REPO = "lightx2v/Wan2.2-Lightning"
 LORA_SUBDIR = "Wan2.2-I2V-A14B-4steps-lora-rank64-Seko-V1"
+# A small vision-language model used to auto-write a caption when the user leaves the
+# scene description blank — SCoPE always needs a caption for the content it should draw.
+CAPTIONER_REPO = "Qwen/Qwen2-VL-2B-Instruct"
+CAPTION_INSTRUCTION = (
+    "Write one vivid paragraph describing this image for a video generation model. "
+    "Cover the main subjects, the setting, materials and colours, and the lighting. "
+    "Do not mention the camera, camera motion, shot type, or the word 'image'."
+)
 WORK_DIR = Path(os.environ.get("SCOPE_WEIGHT_DIR", "/tmp/scope-weights"))
 DEVICE = "cuda"
 
@@ -79,7 +88,6 @@ PRESETS: list[tuple[str, str]] = [
     ("Dolly out — pull straight back", "dolly_out"),
     ("Truck left — slide sideways to the left", "truck_left"),
     ("Truck right — slide sideways to the right", "truck_right"),
-    ("Pan right — rotate in place, no translation", "pan_right"),
     ("Orbit left — arc around the subject", "orbit_left"),
     ("Crane up + forward — rise while pushing in", "crane_up_fwd"),
     ("Snake forward — weaving push-in", "snake_fwd"),
@@ -299,6 +307,57 @@ def build_pipeline() -> SCoPEPipeline:
 PIPE = build_pipeline()
 
 
+# --------------------------------------------------------------------------------------
+# Optional prompt writer — Qwen2-VL captions the first frame when no prompt is given.
+# Downloaded and held on CPU at build time; only lifted onto the GPU for the few seconds
+# it is actually needed, then evicted so it never competes with the two 14B experts.
+# --------------------------------------------------------------------------------------
+
+
+def build_captioner() -> tuple[Qwen2VLForConditionalGeneration, AutoProcessor]:
+    started = time.time()
+    processor = AutoProcessor.from_pretrained(
+        CAPTIONER_REPO, min_pixels=256 * 28 * 28, max_pixels=768 * 28 * 28
+    )
+    model = Qwen2VLForConditionalGeneration.from_pretrained(
+        CAPTIONER_REPO, torch_dtype=torch.bfloat16
+    )
+    model.requires_grad_(False)
+    model.eval()
+    print(f"[SCoPE] caption model ready in {time.time() - started:.0f}s", flush=True)
+    return model, processor
+
+
+CAPTIONER, CAPTION_PROCESSOR = build_captioner()
+
+
+def autocaption(image: Image.Image) -> str:
+    """Describe `image` with Qwen2-VL so a promptless request still has scene content."""
+    messages = [
+        {
+            "role": "user",
+            "content": [{"type": "image"}, {"type": "text", "text": CAPTION_INSTRUCTION}],
+        }
+    ]
+    text = CAPTION_PROCESSOR.apply_chat_template(
+        messages, tokenize=False, add_generation_prompt=True
+    )
+    inputs = CAPTION_PROCESSOR(text=[text], images=[image], return_tensors="pt")
+    CAPTIONER.to(DEVICE)
+    try:
+        inputs = inputs.to(DEVICE)
+        with torch.inference_mode():
+            generated = CAPTIONER.generate(**inputs, max_new_tokens=220, do_sample=False)
+        trimmed = generated[:, inputs["input_ids"].shape[1] :]
+        caption = CAPTION_PROCESSOR.batch_decode(
+            trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=True
+        )[0].strip()
+    finally:
+        CAPTIONER.to("cpu")
+        torch.cuda.empty_cache()
+    return caption
+
+
 # --------------------------------------------------------------------------------------
 # Inference
 # --------------------------------------------------------------------------------------
@@ -338,7 +397,11 @@ def estimate_duration(
     # Measured on ZeroGPU (fp8 experts, 832x480x81): 4 steps -> 67.5s, 8 steps -> 124s,
     # i.e. ~14.1s per sampling step over ~11s of fixed text-encode/VAE cost. Keep a ~15%
     # margin and nothing more, so a default 4-step run stays inside the free 120s quota.
-    return int(math.ceil(1.15 * (11.0 + 14.1 * int(steps))))
+    budget = 11.0 + 14.1 * int(steps)
+    if not (prompt or "").strip():
+        # Lifting Qwen2-VL onto the GPU and captioning the frame adds a fixed overhead.
+        budget += 12.0
+    return int(math.ceil(1.15 * budget))
 
 
 @spaces.GPU(duration=estimate_duration)
@@ -355,8 +418,14 @@ def generate(
 ):
     first_frame = prepare_image(image)
     prompt = (prompt or "").strip()
+    autocaptioned = False
     if not prompt:
-        raise gr.Error("Please describe the scene — SCoPE needs a caption for the content.")
+        # SCoPE needs a caption for the content, so write one from the frame instead of
+        # failing on an empty prompt.
+        prompt = autocaption(first_frame)
+        autocaptioned = True
+        if not prompt:
+            raise gr.Error("Could not caption the image automatically — please add a description.")
 
     used_seed = random.randint(0, MAX_SEED) if randomize_seed else int(seed)
     poses = load_trajectory(trajectory, motion_scale)
@@ -398,6 +467,8 @@ def generate(
         f"{int(steps)} steps · seed {used_seed} · {PRESET_LABELS.get(trajectory, trajectory)} "
         f"· motion x{motion_scale:g} · {elapsed:.0f}s"
     )
+    if autocaptioned:
+        status += f"\n\n*Auto-generated prompt (Qwen2-VL):* {prompt}"
     return output.name, used_seed, status
 
 
@@ -458,6 +529,16 @@ def preview_path(trajectory: str, motion_scale: float) -> Image.Image:
 # --------------------------------------------------------------------------------------
 
 EXAMPLES = [
+    [
+        str(HERE / "examples" / "istock-motorbike-rice-field.jpg"),
+        "A serene rural landscape in soft early-morning light with long shadows. A dirt road "
+        "winds through a lush green area; on the left, dense clusters of tall trees including palms "
+        "and a few small houses with red roofs, and on the right a vast expanse of green rice "
+        "paddies neatly divided into sections. A motorcycle travels along the dirt road, flanked by "
+        "utility poles with wires running along them. The overall atmosphere is peaceful and "
+        "idyllic, with greenery and natural elements dominating the scene.",
+        "orbit_left",
+    ],
     [
         str(HERE / "examples" / "ai-airmountains.jpg"),
         "A vast sky filled with multiple floating islands of different sizes, suspended above a "
@@ -490,6 +571,27 @@ EXAMPLES = [
     ],
 ]
 
+# Scenes that contain a person / character. Kept in their own group so the showcase above
+# stays character-free, per the release curation.
+CHARACTER_EXAMPLES = [
+    [
+        str(HERE / "examples" / "omni-misty-forest.jpg"),
+        "A character in red armor and a straw hat progresses along a forest path, their steps "
+        "deliberate as they navigate over stones and through patches of grass. The environment is "
+        "a misty forest with ancient stone structures on the left and moss-covered cliffs on the "
+        "right. The dense fog and surrounding greenery contribute to a mysterious ambiance.",
+        "orbit_left",
+    ],
+    [
+        str(HERE / "examples" / "omni-horse-trail.jpg"),
+        "A character dressed in dark attire rides a white horse steadily along a rugged dirt path "
+        "that meanders through rocky terrain interspersed with patches of grass and shrubs. To the "
+        "left, a wooden fence lines the trail. In the distance, the landscape opens up to reveal "
+        "rolling hills covered in vegetation and distant mountains under a bright sky.",
+        "crane_up_fwd",
+    ],
+]
+
 CSS = """
 #col-container { margin: 0 auto; max-width: 1180px; }
 .dark .gradio-container { color: var(--body-text-color); }
@@ -505,8 +607,9 @@ with gr.Blocks() as demo:
             rays are normalised, gated and injected straight into the DiT's self-attention
             queries/keys, so a real 3D camera path drives the generated shot.
 
-            Drop in an image, describe the scene, and pick a camera move — you get an
-            81-frame, 832x480, 16 fps clip that follows that trajectory.
+            Drop in an image, pick a camera move, and (optionally) describe the scene — you get
+            an 81-frame, 832x480, 16 fps clip that follows that trajectory. Leave the description
+            blank and a small vision-language model writes one from your image automatically.
             """
         )
         with gr.Row():
@@ -516,8 +619,9 @@ with gr.Blocks() as demo:
                 )
                 prompt_input = gr.Textbox(
                     label="Scene description",
-                    placeholder="Describe what is in the image and what should happen…",
+                    placeholder="Describe what is in the image… (leave empty to auto-caption it)",
                     lines=4,
+                    info="Optional — if left blank, Qwen2-VL writes a caption from your image.",
                 )
                 trajectory_input = gr.Dropdown(
                     label="Camera move",
@@ -536,6 +640,27 @@ with gr.Blocks() as demo:
                 )
                 status_output = gr.Markdown()
 
+        gr.Examples(
+            examples=EXAMPLES,
+            inputs=[image_input, prompt_input, trajectory_input],
+            outputs=[video_output, seed_input, status_output],
+            fn=generate,
+            cache_examples=True,
+            cache_mode="lazy",
+            label="Examples — click a row to load its image, prompt and camera move",
+            examples_per_page=8,
+        )
+        gr.Examples(
+            examples=CHARACTER_EXAMPLES,
+            inputs=[image_input, prompt_input, trajectory_input],
+            outputs=[video_output, seed_input, status_output],
+            fn=generate,
+            cache_examples=True,
+            cache_mode="lazy",
+            label="Examples with characters",
+            examples_per_page=8,
+        )
+
         with gr.Accordion("Advanced settings", open=False):
             with gr.Row():
                 steps_input = gr.Slider(
@@ -571,16 +696,6 @@ with gr.Blocks() as demo:
                 )
                 randomize_input = gr.Checkbox(label="Randomize seed", value=True)
 
-        gr.Examples(
-            examples=EXAMPLES,
-            inputs=[image_input, prompt_input, trajectory_input],
-            outputs=[video_output, seed_input, status_output],
-            fn=generate,
-            cache_examples=True,
-            cache_mode="lazy",
-            label="Examples (AI-generated showcase scenes from the SCoPE release)",
-        )
-
         gr.Markdown(
             """
             **Notes** · Camera paths are OpenCV camera-to-world matrices `[81, 3, 4]` relative to