Mike0021 commited on
Commit
38d06ea
·
verified ·
1 Parent(s): d09f368

Launch WeMM Semantic Universe

Browse files
.gitattributes CHANGED
@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/doc1.jpg filter=lfs diff=lfs merge=lfs -text
37
+ assets/doc2.jpg filter=lfs diff=lfs merge=lfs -text
38
+ assets/doc3.jpg filter=lfs diff=lfs merge=lfs -text
39
+ assets/doc4.jpg filter=lfs diff=lfs merge=lfs -text
40
+ assets/llama4_hgf.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/mapo_tofu.mp4 filter=lfs diff=lfs merge=lfs -text
42
+ assets/qwen2.5omni_hgf.png filter=lfs diff=lfs merge=lfs -text
43
+ assets/zhajiang_noodle.mp4 filter=lfs diff=lfs merge=lfs -text
.gitignore ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ __pycache__/
2
+ *.py[cod]
3
+ .venv/
4
+ .env
5
+ .DS_Store
6
+ assets/.cache/
7
+ gradio_cached_examples/
README.md CHANGED
@@ -1,13 +1,67 @@
1
  ---
2
- title: WeMM Embedding 9B
3
- emoji: 🌖
4
- colorFrom: purple
5
- colorTo: blue
6
  sdk: gradio
7
  sdk_version: 6.26.0
8
- python_version: '3.12'
9
  app_file: app.py
10
  pinned: false
 
 
 
 
 
 
 
 
 
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: WeMM Semantic Universe
3
+ emoji: 🧭
4
+ colorFrom: indigo
5
+ colorTo: yellow
6
  sdk: gradio
7
  sdk_version: 6.26.0
8
+ python_version: 3.12
9
  app_file: app.py
10
  pinned: false
11
+ license: apache-2.0
12
+ short_description: Search meaning across text, images, video, and documents
13
+ startup_duration_timeout: 1h
14
+ suggested_hardware: zero-a10g
15
+ models:
16
+ - tencent/WeMM-Embedding-9B
17
+ datasets:
18
+ - sentence-transformers/example-documents
19
+ preload_from_hub:
20
+ - tencent/WeMM-Embedding-9B
21
  ---
22
 
23
+ # WeMM Semantic Universe
24
+
25
+ An immersive, retrieval-first showcase for [Tencent WeMM-Embedding-9B](https://huggingface.co/tencent/WeMM-Embedding-9B), a universal multimodal embedding model built on Qwen3.5.
26
+
27
+ The Space demonstrates the model as a shared semantic geometry rather than a single similarity score:
28
+
29
+ - Search text, images, video, figures, and visual documents together.
30
+ - Combine an image or video with text to form a multimodal query.
31
+ - Add a custom mixed-media candidate collection alongside the curated universe.
32
+ - Inspect rankings at every native Matryoshka size: 64, 128, 256, 512, 1,024, 2,048, and 4,096 dimensions.
33
+ - Compare any two supported inputs in the Vector Microscope.
34
+ - Explore compressed vector fingerprints and a structured API payload.
35
+
36
+ ## Runtime design
37
+
38
+ The model loads once on CPU at startup. A ZeroGPU allocation moves it to GPU only for inference, then returns it to CPU. The built-in candidate universe is embedded on the first search and cached as normalized 4,096-dimensional CPU tensors; all later dimension choices use lossless prefix truncation followed by re-normalization, without re-encoding the corpus.
39
+
40
+ The first curated search is therefore slower than warm searches. The five `gr.Examples` are cached lazily so opening the Space does not consume GPU quota.
41
+
42
+ This repository targets `zero-a10g` (the current 48 GB ZeroGPU allocation). `suggested_hardware` is advisory metadata: the Space owner must still select ZeroGPU in the Space settings. `GRADIO_SSR_MODE=false` is set before Gradio imports for the fastest startup path.
43
+
44
+ ## API
45
+
46
+ Once deployed, open **Use via API** in the Gradio footer to inspect the generated client signatures.
47
+
48
+ - `/search` ranks a mixed candidate collection from a text/image/video query.
49
+ - `/compare` compares a query and candidate across every native Matryoshka dimension.
50
+
51
+ Always call `Client.view_api()` before invoking either endpoint so the client uses the deployed schema.
52
+
53
+ ## Score semantics
54
+
55
+ Every output is an L2-normalized embedding. The displayed dot products are therefore cosine similarities. They are ranking signals within a candidate set—not calibrated probabilities or universal relevance grades.
56
+
57
+ Audio is not supported by WeMM-Embedding-9B.
58
+
59
+ ## Demo media
60
+
61
+ The curated visual and video inputs come from [`sentence-transformers/example-documents`](https://huggingface.co/datasets/sentence-transformers/example-documents), the same example repository referenced in the WeMM model card. They are bundled locally so examples remain deterministic and do not depend on third-party URLs at runtime.
62
+
63
+ ## References
64
+
65
+ - [Model card](https://huggingface.co/tencent/WeMM-Embedding-9B)
66
+ - [WeMM-Embedding technical report](https://arxiv.org/abs/2608.24053)
67
+ - [Sentence Transformers usage](https://www.sbert.net/)
app.py ADDED
@@ -0,0 +1,1060 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+
3
+ # ZeroGPU and library caches must be configured before importing spaces/torch.
4
+ EXAMPLE_CACHE_VERSION = "2026-08-26-a"
5
+ os.environ.setdefault("HF_HOME", os.path.expanduser("~/.cache/huggingface"))
6
+ os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
7
+ os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
8
+ os.environ.setdefault("GRADIO_EXAMPLES_CACHE", f"/tmp/gradio_cached_examples/{EXAMPLE_CACHE_VERSION}")
9
+ os.environ.setdefault("GRADIO_ANALYTICS_ENABLED", "False")
10
+ os.environ.setdefault("GRADIO_SSR_MODE", "false")
11
+ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
12
+
13
+ import spaces
14
+
15
+ import html
16
+ import logging
17
+ import math
18
+ import threading
19
+ import time
20
+ from dataclasses import dataclass
21
+ from pathlib import Path
22
+ from typing import Any
23
+
24
+ import gradio as gr
25
+ import torch
26
+ import torch.nn.functional as F
27
+ from sentence_transformers import SentenceTransformer
28
+
29
+
30
+ MODEL_ID = "tencent/WeMM-Embedding-9B"
31
+ ROOT = Path(__file__).resolve().parent
32
+ ASSET_DIR = ROOT / "assets"
33
+ MATRYOSHKA_DIMS = (64, 128, 256, 512, 1024, 2048, 4096)
34
+ MAX_CUSTOM_TEXTS = 6
35
+ MAX_CUSTOM_MEDIA = 6
36
+
37
+ logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
38
+ LOGGER = logging.getLogger("wemm-space")
39
+
40
+
41
+ @dataclass(frozen=True)
42
+ class Candidate:
43
+ key: str
44
+ title: str
45
+ kind: str
46
+ description: str
47
+ payload: Any
48
+ media_path: str | None = None
49
+
50
+
51
+ SHOWCASE: tuple[Candidate, ...] = (
52
+ Candidate(
53
+ "llama4",
54
+ "Llama 4 model card",
55
+ "visual document",
56
+ "A dense model-card screenshot describing the Scout and Maverick variants.",
57
+ str(ASSET_DIR / "llama4_hgf.png"),
58
+ str(ASSET_DIR / "llama4_hgf.png"),
59
+ ),
60
+ Candidate(
61
+ "qwen-omni",
62
+ "Qwen2.5-Omni overview",
63
+ "visual document",
64
+ "A model page covering omni-modal perception, speech, and video capabilities.",
65
+ str(ASSET_DIR / "qwen2.5omni_hgf.png"),
66
+ str(ASSET_DIR / "qwen2.5omni_hgf.png"),
67
+ ),
68
+ Candidate(
69
+ "likelihood-contour",
70
+ "Scientific contour plot",
71
+ "figure",
72
+ "An orange likelihood contour plotted against gamma and log-scaled tau over mass.",
73
+ str(ASSET_DIR / "doc1.jpg"),
74
+ str(ASSET_DIR / "doc1.jpg"),
75
+ ),
76
+ Candidate(
77
+ "budget-1971",
78
+ "1971 budget infographic",
79
+ "visual document",
80
+ "A historical chart comparing US outlays, including natural resources spending.",
81
+ str(ASSET_DIR / "doc2.jpg"),
82
+ str(ASSET_DIR / "doc2.jpg"),
83
+ ),
84
+ Candidate(
85
+ "scoring-rules",
86
+ "Proper scoring rules paper",
87
+ "visual document",
88
+ "An academic page with lemmas, an algorithm, equations, and references.",
89
+ str(ASSET_DIR / "doc3.jpg"),
90
+ str(ASSET_DIR / "doc3.jpg"),
91
+ ),
92
+ Candidate(
93
+ "road-safety",
94
+ "Road-safety assessment",
95
+ "visual document",
96
+ "An environmental assessment page about driver training, signs, and road closures.",
97
+ str(ASSET_DIR / "doc4.jpg"),
98
+ str(ASSET_DIR / "doc4.jpg"),
99
+ ),
100
+ Candidate(
101
+ "mapo-tofu",
102
+ "Mapo tofu in motion",
103
+ "video",
104
+ "A short cooking clip showing the preparation of the Sichuan tofu dish.",
105
+ str(ASSET_DIR / "mapo_tofu.mp4"),
106
+ str(ASSET_DIR / "mapo_tofu.mp4"),
107
+ ),
108
+ Candidate(
109
+ "zhajiang-noodles",
110
+ "Zhajiang noodles in motion",
111
+ "video",
112
+ "A short cooking clip showing noodles with a savory fermented-bean sauce.",
113
+ str(ASSET_DIR / "zhajiang_noodle.mp4"),
114
+ str(ASSET_DIR / "zhajiang_noodle.mp4"),
115
+ ),
116
+ Candidate(
117
+ "vector-search",
118
+ "How vector search works",
119
+ "text",
120
+ "Dense retrieval maps queries and documents into one normalized vector space, then ranks candidates by cosine similarity.",
121
+ "Dense retrieval maps queries and documents into one normalized vector space, then ranks candidates by cosine similarity.",
122
+ ),
123
+ Candidate(
124
+ "night-train",
125
+ "A quiet journey",
126
+ "text",
127
+ "夜行列车穿过雨中的城市,车窗映出霓虹灯和安静的乘客。",
128
+ "夜行列车穿过雨中的城市,车窗映出霓虹灯和安静的乘客。",
129
+ ),
130
+ )
131
+
132
+
133
+ LOGGER.info("Loading %s on CPU", MODEL_ID)
134
+ MODEL = SentenceTransformer(
135
+ MODEL_ID,
136
+ trust_remote_code=True,
137
+ device="cpu",
138
+ model_kwargs={"dtype": torch.bfloat16, "low_cpu_mem_usage": True},
139
+ )
140
+ MODEL.eval()
141
+ torch.set_grad_enabled(False)
142
+ LOGGER.info("Model loaded; waiting for a ZeroGPU allocation")
143
+
144
+ _CACHE_LOCK = threading.Lock()
145
+ _INFERENCE_LOCK = threading.Lock()
146
+ _SHOWCASE_EMBEDDINGS: torch.Tensor | None = None
147
+
148
+
149
+ def _file_path(value: Any) -> str | None:
150
+ """Normalize Gradio file values across UI and API representations."""
151
+ if value is None:
152
+ return None
153
+ if isinstance(value, (str, Path)):
154
+ return str(value)
155
+ if isinstance(value, dict):
156
+ path = value.get("path") or value.get("name")
157
+ return str(path) if path else None
158
+ path = getattr(value, "path", None) or getattr(value, "name", None)
159
+ return str(path) if path else None
160
+
161
+
162
+ def _is_video(path: str) -> bool:
163
+ return Path(path.split("?", 1)[0]).suffix.lower() in {
164
+ ".mp4",
165
+ ".webm",
166
+ ".mov",
167
+ ".mkv",
168
+ ".avi",
169
+ ".mpeg",
170
+ ".mpg",
171
+ }
172
+
173
+
174
+ def _multimodal_payload(text: str | None, image: Any, video: Any) -> tuple[Any, str]:
175
+ text = (text or "").strip()
176
+ image_path = _file_path(image)
177
+ video_path = _file_path(video)
178
+ if image_path and video_path:
179
+ raise gr.Error("Choose one visual query: an image or a video, not both.")
180
+ if image_path:
181
+ if text:
182
+ return {"image": image_path, "text": text}, "image + text"
183
+ return image_path, "image"
184
+ if video_path:
185
+ if text:
186
+ return {"video": video_path, "text": text}, "video + text"
187
+ return video_path, "video"
188
+ if text:
189
+ return text, "text"
190
+ raise gr.Error("Add a text, image, or video query to begin.")
191
+
192
+
193
+ def _parse_text_candidates(raw: str | None) -> list[Candidate]:
194
+ candidates: list[Candidate] = []
195
+ lines = [line.strip() for line in (raw or "").splitlines() if line.strip()]
196
+ if len(lines) > MAX_CUSTOM_TEXTS:
197
+ raise gr.Error(f"Use at most {MAX_CUSTOM_TEXTS} custom text candidates.")
198
+ for index, line in enumerate(lines, start=1):
199
+ if "::" in line:
200
+ title, body = (part.strip() for part in line.split("::", 1))
201
+ title = title or f"Custom text {index}"
202
+ body = body or title
203
+ else:
204
+ title = f"Custom text {index}"
205
+ body = line
206
+ candidates.append(
207
+ Candidate(
208
+ f"custom-text-{index}",
209
+ title[:80],
210
+ "text",
211
+ body[:240],
212
+ body,
213
+ )
214
+ )
215
+ return candidates
216
+
217
+
218
+ def _parse_media_candidates(raw: Any) -> list[Candidate]:
219
+ candidates: list[Candidate] = []
220
+ items = raw or []
221
+ if len(items) > MAX_CUSTOM_MEDIA:
222
+ raise gr.Error(f"Upload at most {MAX_CUSTOM_MEDIA} candidate media files.")
223
+ for index, item in enumerate(items, start=1):
224
+ media = item[0] if isinstance(item, (tuple, list)) else item
225
+ caption = item[1] if isinstance(item, (tuple, list)) and len(item) > 1 else None
226
+ path = _file_path(media)
227
+ if not path:
228
+ continue
229
+ kind = "video" if _is_video(path) else "image"
230
+ title = (caption or f"Uploaded {kind} {index}").strip()
231
+ candidates.append(
232
+ Candidate(
233
+ f"custom-media-{index}",
234
+ title[:80],
235
+ kind,
236
+ f"User-supplied {kind} candidate.",
237
+ path,
238
+ path,
239
+ )
240
+ )
241
+ return candidates
242
+
243
+
244
+ def _encode(items: list[Any], *, query: bool) -> torch.Tensor:
245
+ method = MODEL.encode_query if query else MODEL.encode_document
246
+ with torch.inference_mode():
247
+ embeddings = method(
248
+ items,
249
+ batch_size=1,
250
+ convert_to_tensor=True,
251
+ normalize_embeddings=True,
252
+ show_progress_bar=False,
253
+ )
254
+ if embeddings.ndim == 1:
255
+ embeddings = embeddings.unsqueeze(0)
256
+ return embeddings.float().cpu()
257
+
258
+
259
+ def _truncate_normalize(embeddings: torch.Tensor, dimension: int) -> torch.Tensor:
260
+ return F.normalize(embeddings[..., :dimension], p=2, dim=-1)
261
+
262
+
263
+ def _dimension_scores(query: torch.Tensor, documents: torch.Tensor) -> dict[int, list[float]]:
264
+ scores: dict[int, list[float]] = {}
265
+ for dimension in MATRYOSHKA_DIMS:
266
+ query_d = _truncate_normalize(query, dimension)
267
+ docs_d = _truncate_normalize(documents, dimension)
268
+ scores[dimension] = (query_d @ docs_d.T).squeeze(0).tolist()
269
+ return scores
270
+
271
+
272
+ def _search_duration(*args: Any, **kwargs: Any) -> int:
273
+ """Budget more time for the cold corpus pass and video queries."""
274
+ query_payload = args[0] if args else None
275
+ custom_candidates = args[2] if len(args) > 2 else []
276
+ include_showcase = bool(args[3]) if len(args) > 3 else True
277
+ query_has_video = (
278
+ isinstance(query_payload, dict) and bool(query_payload.get("video"))
279
+ ) or (isinstance(query_payload, str) and _is_video(query_payload))
280
+ has_uploaded_video = any(item.kind == "video" for item in custom_candidates or [])
281
+ if include_showcase and _SHOWCASE_EMBEDDINGS is None:
282
+ return 240
283
+ return 120 if query_has_video or has_uploaded_video else 90
284
+
285
+
286
+ @spaces.GPU(duration=_search_duration)
287
+ def _run_search(
288
+ query_payload: Any,
289
+ query_kind: str,
290
+ custom_candidates: list[Candidate],
291
+ include_showcase: bool,
292
+ dimension: int,
293
+ progress: gr.Progress,
294
+ ) -> tuple[torch.Tensor, torch.Tensor, list[Candidate], float, bool]:
295
+ global _SHOWCASE_EMBEDDINGS
296
+
297
+ started = time.perf_counter()
298
+ was_cold = include_showcase and _SHOWCASE_EMBEDDINGS is None
299
+ progress(0.04, desc="Allocating the 9B model on GPU")
300
+ MODEL.to("cuda")
301
+ try:
302
+ progress(0.16, desc=f"Encoding the {query_kind} query")
303
+ query_embedding = _encode([query_payload], query=True)
304
+
305
+ document_blocks: list[torch.Tensor] = []
306
+ candidates: list[Candidate] = []
307
+ if include_showcase:
308
+ with _CACHE_LOCK:
309
+ cached = _SHOWCASE_EMBEDDINGS
310
+ if cached is None:
311
+ progress(0.30, desc="Mapping the curated multimodal universe")
312
+ encoded = _encode([item.payload for item in SHOWCASE], query=False)
313
+ with _CACHE_LOCK:
314
+ if _SHOWCASE_EMBEDDINGS is None:
315
+ _SHOWCASE_EMBEDDINGS = encoded
316
+ cached = _SHOWCASE_EMBEDDINGS
317
+ document_blocks.append(cached)
318
+ candidates.extend(SHOWCASE)
319
+
320
+ if custom_candidates:
321
+ progress(0.72, desc="Encoding your candidate collection")
322
+ custom_embeddings = _encode([item.payload for item in custom_candidates], query=False)
323
+ document_blocks.append(custom_embeddings)
324
+ candidates.extend(custom_candidates)
325
+
326
+ if not document_blocks:
327
+ raise gr.Error("Include the showcase universe or add at least one candidate.")
328
+
329
+ documents = torch.cat(document_blocks, dim=0)
330
+ progress(0.92, desc=f"Ranking in {dimension:,} dimensions")
331
+ return query_embedding, documents, candidates, time.perf_counter() - started, was_cold
332
+ finally:
333
+ MODEL.to("cpu")
334
+ if torch.cuda.is_available():
335
+ torch.cuda.empty_cache()
336
+
337
+
338
+ def _score_tone(score: float) -> str:
339
+ if score >= 0.70:
340
+ return "high"
341
+ if score >= 0.40:
342
+ return "mid"
343
+ return "low"
344
+
345
+
346
+ def _render_summary(
347
+ query_kind: str,
348
+ dimension: int,
349
+ candidate_count: int,
350
+ elapsed: float,
351
+ was_cold: bool,
352
+ top_title: str,
353
+ top_score: float,
354
+ ) -> str:
355
+ cache_note = "cold corpus map" if was_cold else "warm corpus cache"
356
+ return f"""
357
+ <section class="run-summary">
358
+ <div class="run-kicker"><span class="live-dot"></span> semantic field resolved</div>
359
+ <div class="run-main">
360
+ <div><span class="run-label">Top match</span><strong>{html.escape(top_title)}</strong></div>
361
+ <div class="hero-score"><span>{top_score:+.3f}</span><small>cosine</small></div>
362
+ </div>
363
+ <div class="run-meta">
364
+ <span>{html.escape(query_kind)}</span><i></i>
365
+ <span>{dimension:,}D</span><i></i>
366
+ <span>{candidate_count} candidates</span><i></i>
367
+ <span>{elapsed:.1f}s</span><i></i>
368
+ <span>{cache_note}</span>
369
+ </div>
370
+ </section>
371
+ """
372
+
373
+
374
+ def _render_rankings(ranked: list[tuple[Candidate, float]]) -> str:
375
+ rows: list[str] = []
376
+ for rank, (candidate, score) in enumerate(ranked, start=1):
377
+ fill = min(100.0, max(2.0, max(0.0, score) * 100.0))
378
+ rows.append(
379
+ f"""
380
+ <article class="rank-row {'winner' if rank == 1 else ''}">
381
+ <div class="rank-number">{rank:02d}</div>
382
+ <div class="rank-copy">
383
+ <div class="rank-title-line">
384
+ <strong>{html.escape(candidate.title)}</strong>
385
+ <span class="kind-pill">{html.escape(candidate.kind)}</span>
386
+ </div>
387
+ <p>{html.escape(candidate.description)}</p>
388
+ <div class="score-track"><span style="width:{fill:.1f}%"></span></div>
389
+ </div>
390
+ <div class="rank-score { _score_tone(score) }">{score:+.3f}<small>cos</small></div>
391
+ </article>
392
+ """
393
+ )
394
+ return '<div class="ranking-stack">' + "".join(rows) + "</div>"
395
+
396
+
397
+ def _render_curve(series: dict[str, list[float]], title: str) -> str:
398
+ width, height = 820, 300
399
+ left, right, top, bottom = 58, 20, 28, 48
400
+ plot_w, plot_h = width - left - right, height - top - bottom
401
+ all_values = [value for values in series.values() for value in values]
402
+ low = max(-1.0, min(all_values) - 0.08)
403
+ high = min(1.0, max(all_values) + 0.08)
404
+ if high - low < 0.2:
405
+ midpoint = (high + low) / 2
406
+ low, high = max(-1.0, midpoint - 0.1), min(1.0, midpoint + 0.1)
407
+
408
+ def x_at(index: int) -> float:
409
+ return left + index * plot_w / (len(MATRYOSHKA_DIMS) - 1)
410
+
411
+ def y_at(value: float) -> float:
412
+ return top + (high - value) * plot_h / max(1e-8, high - low)
413
+
414
+ grid: list[str] = []
415
+ for tick in range(5):
416
+ value = high - tick * (high - low) / 4
417
+ y = y_at(value)
418
+ grid.append(
419
+ f'<line x1="{left}" y1="{y:.1f}" x2="{width-right}" y2="{y:.1f}" class="chart-grid" />'
420
+ f'<text x="{left-10}" y="{y+4:.1f}" text-anchor="end" class="chart-axis">{value:+.2f}</text>'
421
+ )
422
+ for index, dimension in enumerate(MATRYOSHKA_DIMS):
423
+ x = x_at(index)
424
+ grid.append(f'<text x="{x:.1f}" y="{height-18}" text-anchor="middle" class="chart-axis">{dimension}</text>')
425
+
426
+ colors = ("#ffb86b", "#78e8df", "#a994ff", "#ff7aa2", "#9ad45b")
427
+ paths: list[str] = []
428
+ legend: list[str] = []
429
+ for series_index, (name, values) in enumerate(series.items()):
430
+ color = colors[series_index % len(colors)]
431
+ points = " ".join(f"{x_at(i):.1f},{y_at(value):.1f}" for i, value in enumerate(values))
432
+ circles = "".join(
433
+ f'<circle cx="{x_at(i):.1f}" cy="{y_at(value):.1f}" r="3.5" fill="{color}" />'
434
+ for i, value in enumerate(values)
435
+ )
436
+ paths.append(f'<polyline points="{points}" fill="none" stroke="{color}" stroke-width="3" />{circles}')
437
+ legend.append(
438
+ f'<span><b style="background:{color}"></b>{html.escape(name[:38])}</span>'
439
+ )
440
+
441
+ return f"""
442
+ <section class="viz-card">
443
+ <div class="viz-heading"><div><small>MATRYOSHKA SCOPE</small><h3>{html.escape(title)}</h3></div><span>64 → 4096 dimensions</span></div>
444
+ <svg class="dimension-chart" viewBox="0 0 {width} {height}" role="img" aria-label="Similarity by embedding dimension">
445
+ {''.join(grid)}{''.join(paths)}
446
+ </svg>
447
+ <div class="chart-legend">{''.join(legend)}</div>
448
+ <p class="viz-note">Cosine similarity at every native truncation size. Compare trends, not universal thresholds.</p>
449
+ </section>
450
+ """
451
+
452
+
453
+ def _fingerprint_svg(vector: torch.Tensor, label: str) -> str:
454
+ values = vector.detach().float().flatten()
455
+ bar_count = min(96, values.numel())
456
+ chunks = torch.tensor_split(values, bar_count)
457
+ samples = [float(chunk.mean()) for chunk in chunks]
458
+ scale = max(max(abs(value) for value in samples), 1e-6)
459
+ width, height = 800, 210
460
+ center = 104
461
+ bar_w = (width - 24) / bar_count
462
+ bars: list[str] = []
463
+ for index, value in enumerate(samples):
464
+ magnitude = min(84.0, abs(value) / scale * 84.0)
465
+ x = 12 + index * bar_w
466
+ y = center - magnitude if value >= 0 else center
467
+ color = "#70e4da" if value >= 0 else "#ff9d57"
468
+ bars.append(
469
+ f'<rect x="{x:.1f}" y="{y:.1f}" width="{max(1.2, bar_w-1.5):.1f}" height="{magnitude:.1f}" rx="1.5" fill="{color}" opacity=".9" />'
470
+ )
471
+ return f"""
472
+ <section class="viz-card fingerprint-card">
473
+ <div class="viz-heading"><div><small>VECTOR FINGERPRINT</small><h3>{html.escape(label)}</h3></div><span>{values.numel():,} values</span></div>
474
+ <svg class="fingerprint" viewBox="0 0 {width} {height}" role="img" aria-label="Compressed signed embedding fingerprint">
475
+ <line x1="12" y1="{center}" x2="{width-12}" y2="{center}" class="zero-line" />
476
+ {''.join(bars)}
477
+ </svg>
478
+ <div class="fingerprint-key"><span><b class="positive"></b>positive</span><span><b class="negative"></b>negative</span><em>96 pooled slices · shape, not magnitude</em></div>
479
+ </section>
480
+ """
481
+
482
+
483
+ def search_experience(
484
+ query_text: str,
485
+ query_image: Any,
486
+ query_video: Any,
487
+ custom_texts: str,
488
+ candidate_media: Any,
489
+ include_showcase: bool,
490
+ dimension: int,
491
+ progress: gr.Progress = gr.Progress(track_tqdm=True),
492
+ ) -> tuple[str, str, list[tuple[str, str]], str, str, dict[str, Any]]:
493
+ """Search a mixed text/image/video collection with a multimodal query."""
494
+ dimension = int(dimension)
495
+ if dimension not in MATRYOSHKA_DIMS:
496
+ raise gr.Error("Choose one of the model's native Matryoshka dimensions.")
497
+ query_payload, query_kind = _multimodal_payload(query_text, query_image, query_video)
498
+ custom_candidates = _parse_text_candidates(custom_texts) + _parse_media_candidates(candidate_media)
499
+
500
+ with _INFERENCE_LOCK:
501
+ query, documents, candidates, elapsed, was_cold = _run_search(
502
+ query_payload,
503
+ query_kind,
504
+ custom_candidates,
505
+ bool(include_showcase),
506
+ dimension,
507
+ progress,
508
+ )
509
+ dimension_map = _dimension_scores(query, documents)
510
+ chosen_scores = dimension_map[dimension]
511
+ order = sorted(range(len(candidates)), key=lambda index: chosen_scores[index], reverse=True)
512
+ ranked = [(candidates[index], float(chosen_scores[index])) for index in order]
513
+
514
+ top_candidate, top_score = ranked[0]
515
+ summary = _render_summary(
516
+ query_kind,
517
+ dimension,
518
+ len(candidates),
519
+ elapsed,
520
+ was_cold,
521
+ top_candidate.title,
522
+ top_score,
523
+ )
524
+ rankings = _render_rankings(ranked[:8])
525
+ gallery = [
526
+ (candidate.media_path, f"#{rank} · {candidate.title} · cosine {score:+.3f}")
527
+ for rank, (candidate, score) in enumerate(ranked, start=1)
528
+ if candidate.media_path
529
+ ][:8]
530
+
531
+ curve_series: dict[str, list[float]] = {}
532
+ for index in order[:4]:
533
+ curve_series[candidates[index].title] = [dimension_map[dim][index] for dim in MATRYOSHKA_DIMS]
534
+ curve = _render_curve(curve_series, "Does the ranking survive compression?")
535
+ fingerprint = _fingerprint_svg(query[0, :dimension], f"Query · {query_kind} · {dimension:,}D")
536
+
537
+ diagnostics = {
538
+ "model": MODEL_ID,
539
+ "query_modality": query_kind,
540
+ "selected_dimension": dimension,
541
+ "full_embedding_dimension": int(query.shape[-1]),
542
+ "l2_norm_after_truncation": round(float(_truncate_normalize(query, dimension).norm()), 6),
543
+ "candidate_count": len(candidates),
544
+ "gpu_pass_seconds": round(elapsed, 3),
545
+ "showcase_cache": "created" if was_cold else "reused" if include_showcase else "not_requested",
546
+ "top_matches": [
547
+ {"rank": rank, "title": item.title, "modality": item.kind, "cosine": round(score, 6)}
548
+ for rank, (item, score) in enumerate(ranked[:5], start=1)
549
+ ],
550
+ "query_vector_preview": [round(float(value), 6) for value in query[0, :12]],
551
+ "note": "Cosine similarity is a ranking signal, not a calibrated probability.",
552
+ }
553
+ return summary, rankings, gallery, curve, fingerprint, diagnostics
554
+
555
+
556
+ def _pair_duration(*args: Any, **kwargs: Any) -> int:
557
+ payloads = args[:2]
558
+ has_video = any(
559
+ (isinstance(value, dict) and bool(value.get("video")))
560
+ or (isinstance(value, str) and _is_video(value))
561
+ for value in payloads
562
+ )
563
+ return 150 if has_video else 90
564
+
565
+
566
+ @spaces.GPU(duration=_pair_duration)
567
+ def _run_pair(
568
+ query_payload: Any,
569
+ candidate_payload: Any,
570
+ progress: gr.Progress,
571
+ ) -> tuple[torch.Tensor, torch.Tensor, float]:
572
+ started = time.perf_counter()
573
+ progress(0.08, desc="Allocating the model on GPU")
574
+ MODEL.to("cuda")
575
+ try:
576
+ progress(0.35, desc="Encoding the query")
577
+ query = _encode([query_payload], query=True)
578
+ progress(0.68, desc="Encoding the candidate")
579
+ candidate = _encode([candidate_payload], query=False)
580
+ return query, candidate, time.perf_counter() - started
581
+ finally:
582
+ MODEL.to("cpu")
583
+ if torch.cuda.is_available():
584
+ torch.cuda.empty_cache()
585
+
586
+
587
+ def _interpret_score(score: float) -> tuple[str, str]:
588
+ if score >= 0.75:
589
+ return "high alignment", "These inputs occupy a very similar region for this retrieval model."
590
+ if score >= 0.50:
591
+ return "meaningful alignment", "The model sees a substantial semantic relationship."
592
+ if score >= 0.25:
593
+ return "weak alignment", "There is some overlap, but stronger candidates may rank above it."
594
+ return "low alignment", "The model places these inputs relatively far apart."
595
+
596
+
597
+ def _render_pair_score(score: float, dimension: int, query_kind: str, candidate_kind: str, elapsed: float) -> str:
598
+ label, explanation = _interpret_score(score)
599
+ ring = min(100.0, max(0.0, (score + 1.0) * 50.0))
600
+ return f"""
601
+ <section class="pair-score-card">
602
+ <div class="score-orbit" style="--score:{ring:.2f}">
603
+ <div><strong>{score:+.3f}</strong><span>cosine</span></div>
604
+ </div>
605
+ <div class="pair-score-copy">
606
+ <small>PAIRWISE READOUT</small>
607
+ <h2>{html.escape(label)}</h2>
608
+ <p>{html.escape(explanation)}</p>
609
+ <div class="run-meta"><span>{html.escape(query_kind)}</span><i></i><span>{html.escape(candidate_kind)}</span><i></i><span>{dimension:,}D</span><i></i><span>{elapsed:.1f}s</span></div>
610
+ </div>
611
+ </section>
612
+ """
613
+
614
+
615
+ def compare_experience(
616
+ query_text: str,
617
+ query_image: Any,
618
+ query_video: Any,
619
+ candidate_text: str,
620
+ candidate_image: Any,
621
+ candidate_video: Any,
622
+ dimension: int,
623
+ progress: gr.Progress = gr.Progress(track_tqdm=True),
624
+ ) -> tuple[str, str, str, dict[str, Any]]:
625
+ """Compare any two supported inputs across all Matryoshka dimensions."""
626
+ dimension = int(dimension)
627
+ query_payload, query_kind = _multimodal_payload(query_text, query_image, query_video)
628
+ candidate_payload, candidate_kind = _multimodal_payload(candidate_text, candidate_image, candidate_video)
629
+ with _INFERENCE_LOCK:
630
+ query, candidate, elapsed = _run_pair(query_payload, candidate_payload, progress)
631
+ scores_by_dimension = {
632
+ dim: float((_truncate_normalize(query, dim) @ _truncate_normalize(candidate, dim).T).item())
633
+ for dim in MATRYOSHKA_DIMS
634
+ }
635
+ selected_score = scores_by_dimension[dimension]
636
+ score_card = _render_pair_score(selected_score, dimension, query_kind, candidate_kind, elapsed)
637
+ curve = _render_curve(
638
+ {f"{query_kind} → {candidate_kind}": [scores_by_dimension[dim] for dim in MATRYOSHKA_DIMS]},
639
+ "Semantic alignment under compression",
640
+ )
641
+ fingerprints = (
642
+ '<div class="fingerprint-pair">'
643
+ + _fingerprint_svg(query[0, :dimension], f"Query · {query_kind}")
644
+ + _fingerprint_svg(candidate[0, :dimension], f"Candidate · {candidate_kind}")
645
+ + "</div>"
646
+ )
647
+ diagnostics = {
648
+ "model": MODEL_ID,
649
+ "query_modality": query_kind,
650
+ "candidate_modality": candidate_kind,
651
+ "selected_dimension": dimension,
652
+ "selected_cosine": round(selected_score, 6),
653
+ "cosine_by_dimension": {str(dim): round(score, 6) for dim, score in scores_by_dimension.items()},
654
+ "gpu_pass_seconds": round(elapsed, 3),
655
+ "note": "Interpret thresholds relative to a task-specific candidate set.",
656
+ }
657
+ return score_card, curve, fingerprints, diagnostics
658
+
659
+
660
+ CSS = """
661
+ :root {
662
+ --ink: #f6f3ec;
663
+ --muted: #a9abb5;
664
+ --panel: rgba(17, 20, 26, .82);
665
+ --line: rgba(255, 255, 255, .10);
666
+ --warm: #ffad66;
667
+ --cool: #71e2da;
668
+ --violet: #a994ff;
669
+ }
670
+
671
+ body, .gradio-container {
672
+ background:
673
+ radial-gradient(circle at 13% 0%, rgba(255, 143, 68, .16), transparent 30rem),
674
+ radial-gradient(circle at 92% 13%, rgba(88, 218, 211, .11), transparent 34rem),
675
+ #090b10 !important;
676
+ color: var(--ink) !important;
677
+ }
678
+ .gradio-container { max-width: 1380px !important; padding: 0 28px 60px !important; }
679
+ .gradio-container * { box-sizing: border-box; }
680
+ .gradio-container .prose { color: var(--ink); }
681
+ .gradio-container label, .gradio-container .label-wrap { color: #d7d7dc !important; }
682
+ .gradio-container input, .gradio-container textarea {
683
+ background: rgba(7, 9, 13, .72) !important;
684
+ border-color: rgba(255,255,255,.12) !important;
685
+ color: #f8f6f1 !important;
686
+ }
687
+ .gradio-container .block, .gradio-container .form {
688
+ border-color: var(--line) !important;
689
+ }
690
+
691
+ #hero { padding: 76px 4px 38px; }
692
+ .hero-shell { position: relative; overflow: hidden; border-bottom: 1px solid var(--line); padding-bottom: 44px; }
693
+ .eyebrow { display:flex; align-items:center; gap:10px; color:var(--cool); font-size:12px; font-weight:700; letter-spacing:.18em; text-transform:uppercase; }
694
+ .eyebrow:before { content:""; width:26px; height:1px; background:var(--cool); box-shadow:0 0 14px var(--cool); }
695
+ .hero-title { margin: 18px 0 8px; font-size: clamp(52px, 8vw, 112px); line-height:.88; letter-spacing:-.075em; font-weight:760; }
696
+ .hero-title .accent { color:transparent; -webkit-text-stroke:1px rgba(255,255,255,.62); }
697
+ .hero-title .dot { color:var(--warm); text-shadow:0 0 42px rgba(255,173,102,.6); }
698
+ .hero-sub { max-width:760px; margin:24px 0 0; font-size:clamp(17px,2vw,23px); line-height:1.55; color:#c0c1c8; }
699
+ .hero-grid { display:grid; grid-template-columns:1fr auto; align-items:end; gap:30px; }
700
+ .hero-stats { display:grid; grid-template-columns:repeat(2,minmax(110px,1fr)); gap:1px; background:var(--line); border:1px solid var(--line); min-width:350px; }
701
+ .hero-stats div { background:rgba(9,11,16,.88); padding:18px 20px; }
702
+ .hero-stats strong { display:block; font-size:28px; line-height:1; color:#fff; letter-spacing:-.04em; }
703
+ .hero-stats span { display:block; margin-top:8px; color:var(--muted); font-size:11px; text-transform:uppercase; letter-spacing:.12em; }
704
+ .capability-rail { display:flex; gap:8px; flex-wrap:wrap; margin-top:26px; }
705
+ .capability-rail span { border:1px solid var(--line); border-radius:99px; padding:7px 12px; color:#c9c9cf; font-size:12px; background:rgba(255,255,255,.025); }
706
+
707
+ .section-intro { margin:34px 0 18px; }
708
+ .section-intro small, .viz-heading small, .pair-score-copy small { color:var(--warm); letter-spacing:.16em; font-weight:750; font-size:11px; }
709
+ .section-intro h2 { font-size:30px; letter-spacing:-.035em; margin:6px 0; }
710
+ .section-intro p { color:var(--muted); margin:0; max-width:760px; }
711
+
712
+ .input-panel, .output-panel { background:linear-gradient(145deg,rgba(22,25,32,.9),rgba(12,14,19,.86)) !important; border:1px solid var(--line) !important; border-radius:18px !important; padding:18px !important; box-shadow:0 22px 70px rgba(0,0,0,.24); }
713
+ .primary-action { min-height:52px !important; border:0 !important; color:#16110d !important; font-weight:800 !important; letter-spacing:.01em; background:linear-gradient(105deg,#ff8e54,#ffd187) !important; box-shadow:0 10px 30px rgba(255,142,84,.2) !important; }
714
+ .primary-action:hover { transform:translateY(-1px); filter:brightness(1.04); }
715
+
716
+ .run-summary { border:1px solid rgba(112,228,218,.22); border-radius:18px; padding:22px 24px; background:linear-gradient(120deg,rgba(31,48,48,.54),rgba(17,19,25,.94)); margin-bottom:16px; }
717
+ .run-kicker { color:var(--cool); font-size:11px; text-transform:uppercase; letter-spacing:.16em; font-weight:750; }
718
+ .live-dot { display:inline-block; width:7px; height:7px; border-radius:99px; background:var(--cool); box-shadow:0 0 15px var(--cool); margin-right:8px; }
719
+ .run-main { display:flex; align-items:flex-end; justify-content:space-between; gap:20px; margin:14px 0 18px; }
720
+ .run-label { display:block; color:var(--muted); font-size:12px; margin-bottom:5px; }
721
+ .run-main strong { font-size:clamp(23px,3vw,38px); letter-spacing:-.04em; }
722
+ .hero-score { text-align:right; }
723
+ .hero-score span { display:block; font-size:40px; color:var(--cool); font-variant-numeric:tabular-nums; letter-spacing:-.05em; }
724
+ .hero-score small { color:var(--muted); text-transform:uppercase; letter-spacing:.14em; }
725
+ .run-meta { display:flex; align-items:center; gap:10px; flex-wrap:wrap; color:#aeb0b8; font-size:12px; }
726
+ .run-meta i { display:block; width:3px; height:3px; border-radius:99px; background:#555963; }
727
+
728
+ .ranking-stack { display:grid; gap:9px; }
729
+ .rank-row { display:grid; grid-template-columns:48px 1fr 80px; gap:16px; align-items:center; padding:15px 17px; border:1px solid var(--line); border-radius:14px; background:rgba(255,255,255,.025); transition:.2s ease; }
730
+ .rank-row:hover { transform:translateX(3px); border-color:rgba(255,173,102,.28); background:rgba(255,255,255,.04); }
731
+ .rank-row.winner { border-color:rgba(255,173,102,.35); background:linear-gradient(100deg,rgba(255,150,85,.11),rgba(255,255,255,.025)); }
732
+ .rank-number { color:#6d7079; font-size:14px; font-variant-numeric:tabular-nums; }
733
+ .rank-title-line { display:flex; gap:9px; align-items:center; flex-wrap:wrap; }
734
+ .rank-title-line strong { font-size:16px; color:#f7f3ec; }
735
+ .kind-pill { font-size:9px; letter-spacing:.09em; text-transform:uppercase; color:#bfc1c8; border:1px solid var(--line); border-radius:99px; padding:4px 7px; }
736
+ .rank-copy p { margin:4px 0 9px; color:#91949e; font-size:12px; line-height:1.45; }
737
+ .score-track { height:2px; background:rgba(255,255,255,.06); overflow:hidden; }
738
+ .score-track span { display:block; height:100%; background:linear-gradient(90deg,var(--warm),var(--cool)); }
739
+ .rank-score { text-align:right; font-size:18px; font-variant-numeric:tabular-nums; color:#b9bbc2; }
740
+ .rank-score.high { color:var(--cool); }.rank-score.mid { color:var(--warm); }
741
+ .rank-score small { display:block; font-size:8px; color:#747780; letter-spacing:.14em; margin-top:3px; text-transform:uppercase; }
742
+
743
+ .viz-card { border:1px solid var(--line); border-radius:18px; padding:20px; background:rgba(15,17,23,.78); overflow:hidden; }
744
+ .viz-heading { display:flex; align-items:flex-start; justify-content:space-between; gap:16px; }
745
+ .viz-heading h3 { margin:5px 0 0; font-size:20px; letter-spacing:-.025em; }
746
+ .viz-heading > span { color:#8f929d; font-size:11px; border:1px solid var(--line); border-radius:99px; padding:6px 9px; }
747
+ .dimension-chart, .fingerprint { width:100%; height:auto; overflow:visible; }
748
+ .chart-grid { stroke:rgba(255,255,255,.07); stroke-width:1; }
749
+ .chart-axis { fill:#777b85; font-size:10px; font-family:ui-monospace,SFMono-Regular,Menlo,monospace; }
750
+ .chart-legend { display:flex; flex-wrap:wrap; gap:8px 16px; }
751
+ .chart-legend span { color:#aeb0b8; font-size:11px; }
752
+ .chart-legend b { display:inline-block; width:7px; height:7px; border-radius:99px; margin-right:6px; }
753
+ .viz-note { margin:14px 0 0; color:#6f727c; font-size:11px; }
754
+ .zero-line { stroke:rgba(255,255,255,.18); stroke-width:1; }
755
+ .fingerprint-key { display:flex; gap:14px; align-items:center; flex-wrap:wrap; color:#848791; font-size:10px; text-transform:uppercase; letter-spacing:.08em; }
756
+ .fingerprint-key b { display:inline-block; width:7px; height:7px; border-radius:2px; margin-right:5px; }.fingerprint-key .positive{background:var(--cool)}.fingerprint-key .negative{background:var(--warm)}
757
+ .fingerprint-key em { margin-left:auto; text-transform:none; letter-spacing:0; color:#6e717a; }
758
+ .fingerprint-pair { display:grid; grid-template-columns:1fr 1fr; gap:12px; }
759
+
760
+ .pair-score-card { display:grid; grid-template-columns:190px 1fr; gap:30px; align-items:center; border:1px solid rgba(169,148,255,.24); border-radius:20px; padding:26px; background:linear-gradient(125deg,rgba(50,40,80,.35),rgba(14,17,23,.92)); }
761
+ .score-orbit { width:170px; aspect-ratio:1; border-radius:50%; padding:2px; background:conic-gradient(var(--violet) calc(var(--score)*1%),rgba(255,255,255,.07) 0); box-shadow:0 0 55px rgba(169,148,255,.12); }
762
+ .score-orbit > div { width:100%; height:100%; border-radius:50%; background:#0d0f15; display:flex; flex-direction:column; align-items:center; justify-content:center; }
763
+ .score-orbit strong { font-size:38px; letter-spacing:-.05em; font-variant-numeric:tabular-nums; }.score-orbit span{color:#777b85;font-size:10px;text-transform:uppercase;letter-spacing:.14em}
764
+ .pair-score-copy h2 { margin:6px 0 8px; font-size:34px; letter-spacing:-.045em; }.pair-score-copy p{color:#aeb0b8;max-width:580px}
765
+
766
+ .model-note { margin:28px 0 0; padding:18px 20px; border-left:2px solid var(--warm); background:rgba(255,173,102,.05); color:#9ea0aa; font-size:12px; line-height:1.6; }
767
+ .space-footer { display:flex; justify-content:space-between; align-items:center; gap:20px; flex-wrap:wrap; border-top:1px solid var(--line); padding:28px 4px 0; margin-top:44px; color:#767984; font-size:12px; }
768
+ .space-footer a { color:#c4c6cd !important; text-decoration:none; }.space-footer a:hover{color:var(--warm)!important}
769
+
770
+ @media (max-width: 900px) {
771
+ .gradio-container { padding:0 14px 40px !important; }
772
+ #hero { padding-top:44px; }
773
+ .hero-grid { grid-template-columns:1fr; }
774
+ .hero-stats { min-width:0; width:100%; }
775
+ .fingerprint-pair { grid-template-columns:1fr; }
776
+ }
777
+ @media (max-width: 620px) {
778
+ .hero-title { font-size:50px; }
779
+ .hero-stats { grid-template-columns:1fr 1fr; }
780
+ .rank-row { grid-template-columns:34px 1fr 64px; gap:8px; padding:12px 10px; }
781
+ .rank-copy p { display:none; }
782
+ .pair-score-card { grid-template-columns:1fr; text-align:center; }
783
+ .score-orbit { margin:auto; }.pair-score-copy .run-meta{justify-content:center}
784
+ }
785
+ """
786
+
787
+
788
+ HERO = """
789
+ <div id="hero" class="hero-shell">
790
+ <div class="eyebrow">Tencent WeMM · multimodal embedding</div>
791
+ <div class="hero-grid">
792
+ <div>
793
+ <h1 class="hero-title">One space<span class="accent">.<br>Every medium</span><span class="dot">.</span></h1>
794
+ <p class="hero-sub">Search meaning—not file types—across text, images, video, charts, and visual documents in one shared semantic geometry.</p>
795
+ <div class="capability-rail"><span>text ↔ image</span><span>text ↔ video</span><span>visual documents</span><span>interleaved inputs</span><span>Matryoshka embeddings</span></div>
796
+ </div>
797
+ <div class="hero-stats">
798
+ <div><strong>4096</strong><span>native dimensions</span></div>
799
+ <div><strong>9B</strong><span>parameters</span></div>
800
+ <div><strong>80.6</strong><span>MMEB-v2 avg</span></div>
801
+ <div><strong>190</strong><span>MMEB-v3 tasks</span></div>
802
+ </div>
803
+ </div>
804
+ </div>
805
+ """
806
+
807
+
808
+ INTRO_SEARCH = """
809
+ <div class="section-intro">
810
+ <small>01 / RETRIEVAL UNIVERSE</small>
811
+ <h2>Ask in one modality. Discover in another.</h2>
812
+ <p>Search the built-in field of screenshots, figures, dense documents, video, and multilingual text—or bring your own candidates.</p>
813
+ </div>
814
+ """
815
+
816
+
817
+ INTRO_PAIR = """
818
+ <div class="section-intro">
819
+ <small>02 / VECTOR MICROSCOPE</small>
820
+ <h2>Put any two ideas under the lens.</h2>
821
+ <p>Use a query and candidate as text, image, video, or visual-plus-text. Then watch their alignment change as the embedding compresses.</p>
822
+ </div>
823
+ """
824
+
825
+
826
+ INITIAL_SUMMARY = """
827
+ <section class="run-summary">
828
+ <div class="run-kicker"><span class="live-dot"></span> model ready</div>
829
+ <div class="run-main"><div><span class="run-label">Awaiting a query</span><strong>Search across media</strong></div><div class="hero-score"><span>—</span><small>cosine</small></div></div>
830
+ <div class="run-meta"><span>text</span><i></i><span>image</span><i></i><span>video</span><i></i><span>visual documents</span></div>
831
+ </section>
832
+ """
833
+
834
+
835
+ INITIAL_RANKING = """
836
+ <div class="model-note">Choose a curated example below or compose a multimodal query. The first run maps the showcase corpus once; later searches reuse its CPU-cached 4,096D vectors.</div>
837
+ """
838
+
839
+
840
+ with gr.Blocks(css=CSS, title="WeMM · Multimodal Embedding Universe", fill_width=True) as demo:
841
+ gr.HTML(HERO)
842
+
843
+ with gr.Tabs():
844
+ with gr.Tab("Search the universe", id="search"):
845
+ gr.HTML(INTRO_SEARCH)
846
+ with gr.Row(equal_height=False):
847
+ with gr.Column(scale=5, elem_classes="input-panel"):
848
+ query_text = gr.Textbox(
849
+ label="Query · text",
850
+ placeholder="Try: Which document explains temporary road closures?",
851
+ lines=3,
852
+ max_lines=7,
853
+ )
854
+ with gr.Row():
855
+ query_image = gr.Image(
856
+ label="Query · image (optional)",
857
+ type="filepath",
858
+ sources=["upload", "clipboard", "webcam"],
859
+ height=220,
860
+ )
861
+ query_video = gr.Video(
862
+ label="Query · video (optional)",
863
+ format="mp4",
864
+ height=220,
865
+ )
866
+ gr.Markdown("Add text to an image or video to create a joint multimodal query.")
867
+ with gr.Accordion("Build your own candidate collection", open=False):
868
+ custom_texts = gr.Textbox(
869
+ label="Text candidates",
870
+ placeholder="Title :: Candidate text\nAnother title :: Another candidate",
871
+ lines=5,
872
+ info=f"One candidate per line, up to {MAX_CUSTOM_TEXTS}.",
873
+ )
874
+ candidate_media = gr.Gallery(
875
+ label="Image + video candidates",
876
+ type="filepath",
877
+ file_types=["image", "video"],
878
+ sources=["upload"],
879
+ columns=3,
880
+ height=250,
881
+ )
882
+ include_showcase = gr.Checkbox(
883
+ value=True,
884
+ label="Include the curated multimodal universe",
885
+ info="10 candidates spanning text, images, video, figures, and visual documents.",
886
+ )
887
+ dimension = gr.Radio(
888
+ choices=list(MATRYOSHKA_DIMS),
889
+ value=1024,
890
+ label="Embedding budget",
891
+ info="Native Matryoshka dimensions; smaller vectors trade storage for fidelity.",
892
+ )
893
+ search_button = gr.Button("Map the semantic field →", variant="primary", elem_classes="primary-action")
894
+
895
+ with gr.Column(scale=7, elem_classes="output-panel"):
896
+ search_summary = gr.HTML(INITIAL_SUMMARY)
897
+ ranking_output = gr.HTML(INITIAL_RANKING)
898
+
899
+ with gr.Row(equal_height=False):
900
+ result_gallery = gr.Gallery(
901
+ value=[
902
+ (str(ASSET_DIR / "llama4_hgf.png"), "Visual document · Llama 4 model card"),
903
+ (str(ASSET_DIR / "doc2.jpg"), "Visual document · 1971 budget infographic"),
904
+ (str(ASSET_DIR / "mapo_tofu.mp4"), "Video · Mapo tofu in motion"),
905
+ (str(ASSET_DIR / "doc4.jpg"), "Visual document · road-safety assessment"),
906
+ ],
907
+ label="Ranked visual field",
908
+ columns=4,
909
+ rows=2,
910
+ height=430,
911
+ object_fit="contain",
912
+ interactive=False,
913
+ buttons=["fullscreen", "download_all"],
914
+ )
915
+ with gr.Row(equal_height=False):
916
+ dimension_output = gr.HTML('<div class="viz-card"><div class="viz-heading"><div><small>MATRYOSHKA SCOPE</small><h3>Dimension stability appears here</h3></div></div></div>')
917
+ fingerprint_output = gr.HTML('<div class="viz-card"><div class="viz-heading"><div><small>VECTOR FINGERPRINT</small><h3>Your query vector appears here</h3></div></div></div>')
918
+ with gr.Accordion("Embedding telemetry · inspect the API payload", open=False):
919
+ diagnostics_output = gr.JSON(label="Diagnostics")
920
+
921
+ example_inputs = [
922
+ query_text,
923
+ query_image,
924
+ query_video,
925
+ custom_texts,
926
+ candidate_media,
927
+ include_showcase,
928
+ dimension,
929
+ ]
930
+ example_outputs = [
931
+ search_summary,
932
+ ranking_output,
933
+ result_gallery,
934
+ dimension_output,
935
+ fingerprint_output,
936
+ diagnostics_output,
937
+ ]
938
+ gr.Examples(
939
+ examples=[
940
+ ["Which Llama 4 model variants are available?", None, None, "", None, True, 512],
941
+ ["How is mapo tofu prepared?", None, None, "", None, True, 1024],
942
+ ["Find the environmental assessment page about driver training and temporary road closures.", None, None, "", None, True, 256],
943
+ [
944
+ "Match this screenshot to the most relevant description.",
945
+ str(ASSET_DIR / "llama4_hgf.png"),
946
+ None,
947
+ "Llama family :: Scout and Maverick are multimodal mixture-of-experts model variants.\nRecipe :: Soft tofu simmered in spicy chili-bean sauce.",
948
+ None,
949
+ False,
950
+ 256,
951
+ ],
952
+ [
953
+ "What dish is being prepared in this clip?",
954
+ None,
955
+ str(ASSET_DIR / "mapo_tofu.mp4"),
956
+ "Sichuan classic :: Mapo tofu combines soft tofu with a spicy, numbing bean-paste sauce.\nSpaceflight :: A launch vehicle carries a satellite into orbit.",
957
+ None,
958
+ False,
959
+ 512,
960
+ ],
961
+ ],
962
+ inputs=example_inputs,
963
+ outputs=example_outputs,
964
+ fn=search_experience,
965
+ cache_examples=True,
966
+ cache_mode="lazy",
967
+ label="Curated expeditions",
968
+ example_labels=[
969
+ "Find Llama 4 across a screenshot",
970
+ "Search a cooking video with text",
971
+ "Retrieve a dense safety document",
972
+ "Match an image to text candidates",
973
+ "Match a video to text candidates",
974
+ ],
975
+ )
976
+
977
+ search_event = search_button.click(
978
+ fn=search_experience,
979
+ inputs=example_inputs,
980
+ outputs=example_outputs,
981
+ api_name="search",
982
+ api_description="Rank a mixed text/image/video collection using WeMM-Embedding-9B.",
983
+ concurrency_limit=1,
984
+ concurrency_id="wemm_gpu",
985
+ time_limit=300,
986
+ scroll_to_output=True,
987
+ )
988
+ query_text.submit(
989
+ fn=search_experience,
990
+ inputs=example_inputs,
991
+ outputs=example_outputs,
992
+ api_name=None,
993
+ api_visibility="private",
994
+ concurrency_limit=1,
995
+ concurrency_id="wemm_gpu",
996
+ time_limit=300,
997
+ scroll_to_output=True,
998
+ )
999
+
1000
+ with gr.Tab("Compare two ideas", id="compare"):
1001
+ gr.HTML(INTRO_PAIR)
1002
+ with gr.Row(equal_height=False):
1003
+ with gr.Column(elem_classes="input-panel"):
1004
+ gr.Markdown("### A · Query")
1005
+ pair_query_text = gr.Textbox(label="Text", placeholder="Describe or contextualize the query", lines=3)
1006
+ with gr.Row():
1007
+ pair_query_image = gr.Image(label="Image", type="filepath", height=210)
1008
+ pair_query_video = gr.Video(label="Video", format="mp4", height=210)
1009
+ with gr.Column(elem_classes="input-panel"):
1010
+ gr.Markdown("### B · Candidate")
1011
+ pair_candidate_text = gr.Textbox(label="Text", placeholder="Describe or contextualize the candidate", lines=3)
1012
+ with gr.Row():
1013
+ pair_candidate_image = gr.Image(label="Image", type="filepath", height=210)
1014
+ pair_candidate_video = gr.Video(label="Video", format="mp4", height=210)
1015
+ pair_dimension = gr.Radio(
1016
+ choices=list(MATRYOSHKA_DIMS),
1017
+ value=1024,
1018
+ label="Embedding budget",
1019
+ )
1020
+ compare_button = gr.Button("Measure semantic alignment →", variant="primary", elem_classes="primary-action")
1021
+ pair_score_output = gr.HTML('<div class="model-note">Add one modality on each side. You may pair visual media with text context.</div>')
1022
+ pair_curve_output = gr.HTML('<div class="viz-card"><div class="viz-heading"><div><small>MATRYOSHKA SCOPE</small><h3>Alignment by dimension appears here</h3></div></div></div>')
1023
+ pair_fingerprints_output = gr.HTML()
1024
+ with gr.Accordion("Pairwise telemetry", open=False):
1025
+ pair_diagnostics_output = gr.JSON(label="Diagnostics")
1026
+
1027
+ pair_inputs = [
1028
+ pair_query_text,
1029
+ pair_query_image,
1030
+ pair_query_video,
1031
+ pair_candidate_text,
1032
+ pair_candidate_image,
1033
+ pair_candidate_video,
1034
+ pair_dimension,
1035
+ ]
1036
+ compare_button.click(
1037
+ fn=compare_experience,
1038
+ inputs=pair_inputs,
1039
+ outputs=[pair_score_output, pair_curve_output, pair_fingerprints_output, pair_diagnostics_output],
1040
+ api_name="compare",
1041
+ api_description="Compare a multimodal query and candidate across every native Matryoshka dimension.",
1042
+ concurrency_limit=1,
1043
+ concurrency_id="wemm_gpu",
1044
+ time_limit=240,
1045
+ scroll_to_output=True,
1046
+ )
1047
+
1048
+ gr.HTML(
1049
+ """
1050
+ <div class="model-note"><strong>Read scores comparatively.</strong> Cosine similarity is useful for ranking candidates within a collection; it is not a calibrated confidence or a universal relevance grade. Audio is not supported by WeMM-Embedding-9B.</div>
1051
+ <footer class="space-footer"><span>WeMM-Embedding-9B · Apache-2.0 · built on Qwen3.5</span><span><a href="https://huggingface.co/tencent/WeMM-Embedding-9B" target="_blank">Model card ↗</a>&nbsp;&nbsp;&nbsp;<a href="https://arxiv.org/abs/2608.24053" target="_blank">Technical report ↗</a></span></footer>
1052
+ """
1053
+ )
1054
+
1055
+
1056
+ if __name__ == "__main__":
1057
+ demo.queue(default_concurrency_limit=1, max_size=24).launch(
1058
+ allowed_paths=[str(ASSET_DIR)],
1059
+ show_error=True,
1060
+ )
assets/doc1.jpg ADDED

Git LFS Details

  • SHA256: 064a48b12f3c87294a26967763864d093235efe7ae9c46a3e5bf2fc2ec7a005d
  • Pointer size: 131 Bytes
  • Size of remote file: 126 kB
assets/doc2.jpg ADDED

Git LFS Details

  • SHA256: 1d3df793c847aa7e600c535a78a022c82514f5dfb13cd00ba6fb6cede47b8576
  • Pointer size: 131 Bytes
  • Size of remote file: 922 kB
assets/doc3.jpg ADDED

Git LFS Details

  • SHA256: 6989fe6df558ba71de4a66d60ef66ddefc4fa9f39ccba8621410850c2c4474c8
  • Pointer size: 131 Bytes
  • Size of remote file: 534 kB
assets/doc4.jpg ADDED

Git LFS Details

  • SHA256: e4daa1f009e168a55e1237e6d198b694570f641f3132784b7da3f207f304316c
  • Pointer size: 131 Bytes
  • Size of remote file: 509 kB
assets/llama4_hgf.png ADDED

Git LFS Details

  • SHA256: bd2e827ddb93616659da2214267b6c9461aff7bd2f38a802c852cf2f4be3cef0
  • Pointer size: 131 Bytes
  • Size of remote file: 180 kB
assets/mapo_tofu.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:23bd7a2a9a554bc09084cb74e584ca6129292073efcd2350f180e81975f96ec5
3
+ size 5250889
assets/qwen2.5omni_hgf.png ADDED

Git LFS Details

  • SHA256: 46df97974679c23d566142fe0055ef148c79d629713d1c0b8775027109bdd700
  • Pointer size: 131 Bytes
  • Size of remote file: 194 kB
assets/zhajiang_noodle.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:640b98fde893982dc0e866e72d537d9798a5a8432a6a7abc8f76d630c897a1b1
3
+ size 3571831
requirements.txt ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ gradio==6.26.0
2
+ spaces==0.51.1
3
+ transformers==5.2.0
4
+ sentence-transformers==5.7.0
5
+ accelerate==1.14.0
6
+ qwen-vl-utils[decord]==0.0.14