Spaces:
Paused
Paused
Launch WeMM Semantic Universe
Browse files- .gitattributes +8 -0
- .gitignore +7 -0
- README.md +60 -6
- app.py +1060 -0
- assets/doc1.jpg +3 -0
- assets/doc2.jpg +3 -0
- assets/doc3.jpg +3 -0
- assets/doc4.jpg +3 -0
- assets/llama4_hgf.png +3 -0
- assets/mapo_tofu.mp4 +3 -0
- assets/qwen2.5omni_hgf.png +3 -0
- assets/zhajiang_noodle.mp4 +3 -0
- requirements.txt +6 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/doc1.jpg filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
assets/doc2.jpg filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
assets/doc3.jpg filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
assets/doc4.jpg filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
assets/llama4_hgf.png filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
assets/mapo_tofu.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
assets/qwen2.5omni_hgf.png filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
assets/zhajiang_noodle.mp4 filter=lfs diff=lfs merge=lfs -text
|
.gitignore
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
__pycache__/
|
| 2 |
+
*.py[cod]
|
| 3 |
+
.venv/
|
| 4 |
+
.env
|
| 5 |
+
.DS_Store
|
| 6 |
+
assets/.cache/
|
| 7 |
+
gradio_cached_examples/
|
README.md
CHANGED
|
@@ -1,13 +1,67 @@
|
|
| 1 |
---
|
| 2 |
-
title: WeMM
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.26.0
|
| 8 |
-
python_version:
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: WeMM Semantic Universe
|
| 3 |
+
emoji: 🧭
|
| 4 |
+
colorFrom: indigo
|
| 5 |
+
colorTo: yellow
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.26.0
|
| 8 |
+
python_version: 3.12
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
+
license: apache-2.0
|
| 12 |
+
short_description: Search meaning across text, images, video, and documents
|
| 13 |
+
startup_duration_timeout: 1h
|
| 14 |
+
suggested_hardware: zero-a10g
|
| 15 |
+
models:
|
| 16 |
+
- tencent/WeMM-Embedding-9B
|
| 17 |
+
datasets:
|
| 18 |
+
- sentence-transformers/example-documents
|
| 19 |
+
preload_from_hub:
|
| 20 |
+
- tencent/WeMM-Embedding-9B
|
| 21 |
---
|
| 22 |
|
| 23 |
+
# WeMM Semantic Universe
|
| 24 |
+
|
| 25 |
+
An immersive, retrieval-first showcase for [Tencent WeMM-Embedding-9B](https://huggingface.co/tencent/WeMM-Embedding-9B), a universal multimodal embedding model built on Qwen3.5.
|
| 26 |
+
|
| 27 |
+
The Space demonstrates the model as a shared semantic geometry rather than a single similarity score:
|
| 28 |
+
|
| 29 |
+
- Search text, images, video, figures, and visual documents together.
|
| 30 |
+
- Combine an image or video with text to form a multimodal query.
|
| 31 |
+
- Add a custom mixed-media candidate collection alongside the curated universe.
|
| 32 |
+
- Inspect rankings at every native Matryoshka size: 64, 128, 256, 512, 1,024, 2,048, and 4,096 dimensions.
|
| 33 |
+
- Compare any two supported inputs in the Vector Microscope.
|
| 34 |
+
- Explore compressed vector fingerprints and a structured API payload.
|
| 35 |
+
|
| 36 |
+
## Runtime design
|
| 37 |
+
|
| 38 |
+
The model loads once on CPU at startup. A ZeroGPU allocation moves it to GPU only for inference, then returns it to CPU. The built-in candidate universe is embedded on the first search and cached as normalized 4,096-dimensional CPU tensors; all later dimension choices use lossless prefix truncation followed by re-normalization, without re-encoding the corpus.
|
| 39 |
+
|
| 40 |
+
The first curated search is therefore slower than warm searches. The five `gr.Examples` are cached lazily so opening the Space does not consume GPU quota.
|
| 41 |
+
|
| 42 |
+
This repository targets `zero-a10g` (the current 48 GB ZeroGPU allocation). `suggested_hardware` is advisory metadata: the Space owner must still select ZeroGPU in the Space settings. `GRADIO_SSR_MODE=false` is set before Gradio imports for the fastest startup path.
|
| 43 |
+
|
| 44 |
+
## API
|
| 45 |
+
|
| 46 |
+
Once deployed, open **Use via API** in the Gradio footer to inspect the generated client signatures.
|
| 47 |
+
|
| 48 |
+
- `/search` ranks a mixed candidate collection from a text/image/video query.
|
| 49 |
+
- `/compare` compares a query and candidate across every native Matryoshka dimension.
|
| 50 |
+
|
| 51 |
+
Always call `Client.view_api()` before invoking either endpoint so the client uses the deployed schema.
|
| 52 |
+
|
| 53 |
+
## Score semantics
|
| 54 |
+
|
| 55 |
+
Every output is an L2-normalized embedding. The displayed dot products are therefore cosine similarities. They are ranking signals within a candidate set—not calibrated probabilities or universal relevance grades.
|
| 56 |
+
|
| 57 |
+
Audio is not supported by WeMM-Embedding-9B.
|
| 58 |
+
|
| 59 |
+
## Demo media
|
| 60 |
+
|
| 61 |
+
The curated visual and video inputs come from [`sentence-transformers/example-documents`](https://huggingface.co/datasets/sentence-transformers/example-documents), the same example repository referenced in the WeMM model card. They are bundled locally so examples remain deterministic and do not depend on third-party URLs at runtime.
|
| 62 |
+
|
| 63 |
+
## References
|
| 64 |
+
|
| 65 |
+
- [Model card](https://huggingface.co/tencent/WeMM-Embedding-9B)
|
| 66 |
+
- [WeMM-Embedding technical report](https://arxiv.org/abs/2608.24053)
|
| 67 |
+
- [Sentence Transformers usage](https://www.sbert.net/)
|
app.py
ADDED
|
@@ -0,0 +1,1060 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
|
| 3 |
+
# ZeroGPU and library caches must be configured before importing spaces/torch.
|
| 4 |
+
EXAMPLE_CACHE_VERSION = "2026-08-26-a"
|
| 5 |
+
os.environ.setdefault("HF_HOME", os.path.expanduser("~/.cache/huggingface"))
|
| 6 |
+
os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
|
| 7 |
+
os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
|
| 8 |
+
os.environ.setdefault("GRADIO_EXAMPLES_CACHE", f"/tmp/gradio_cached_examples/{EXAMPLE_CACHE_VERSION}")
|
| 9 |
+
os.environ.setdefault("GRADIO_ANALYTICS_ENABLED", "False")
|
| 10 |
+
os.environ.setdefault("GRADIO_SSR_MODE", "false")
|
| 11 |
+
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
|
| 12 |
+
|
| 13 |
+
import spaces
|
| 14 |
+
|
| 15 |
+
import html
|
| 16 |
+
import logging
|
| 17 |
+
import math
|
| 18 |
+
import threading
|
| 19 |
+
import time
|
| 20 |
+
from dataclasses import dataclass
|
| 21 |
+
from pathlib import Path
|
| 22 |
+
from typing import Any
|
| 23 |
+
|
| 24 |
+
import gradio as gr
|
| 25 |
+
import torch
|
| 26 |
+
import torch.nn.functional as F
|
| 27 |
+
from sentence_transformers import SentenceTransformer
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
MODEL_ID = "tencent/WeMM-Embedding-9B"
|
| 31 |
+
ROOT = Path(__file__).resolve().parent
|
| 32 |
+
ASSET_DIR = ROOT / "assets"
|
| 33 |
+
MATRYOSHKA_DIMS = (64, 128, 256, 512, 1024, 2048, 4096)
|
| 34 |
+
MAX_CUSTOM_TEXTS = 6
|
| 35 |
+
MAX_CUSTOM_MEDIA = 6
|
| 36 |
+
|
| 37 |
+
logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
|
| 38 |
+
LOGGER = logging.getLogger("wemm-space")
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
@dataclass(frozen=True)
|
| 42 |
+
class Candidate:
|
| 43 |
+
key: str
|
| 44 |
+
title: str
|
| 45 |
+
kind: str
|
| 46 |
+
description: str
|
| 47 |
+
payload: Any
|
| 48 |
+
media_path: str | None = None
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
SHOWCASE: tuple[Candidate, ...] = (
|
| 52 |
+
Candidate(
|
| 53 |
+
"llama4",
|
| 54 |
+
"Llama 4 model card",
|
| 55 |
+
"visual document",
|
| 56 |
+
"A dense model-card screenshot describing the Scout and Maverick variants.",
|
| 57 |
+
str(ASSET_DIR / "llama4_hgf.png"),
|
| 58 |
+
str(ASSET_DIR / "llama4_hgf.png"),
|
| 59 |
+
),
|
| 60 |
+
Candidate(
|
| 61 |
+
"qwen-omni",
|
| 62 |
+
"Qwen2.5-Omni overview",
|
| 63 |
+
"visual document",
|
| 64 |
+
"A model page covering omni-modal perception, speech, and video capabilities.",
|
| 65 |
+
str(ASSET_DIR / "qwen2.5omni_hgf.png"),
|
| 66 |
+
str(ASSET_DIR / "qwen2.5omni_hgf.png"),
|
| 67 |
+
),
|
| 68 |
+
Candidate(
|
| 69 |
+
"likelihood-contour",
|
| 70 |
+
"Scientific contour plot",
|
| 71 |
+
"figure",
|
| 72 |
+
"An orange likelihood contour plotted against gamma and log-scaled tau over mass.",
|
| 73 |
+
str(ASSET_DIR / "doc1.jpg"),
|
| 74 |
+
str(ASSET_DIR / "doc1.jpg"),
|
| 75 |
+
),
|
| 76 |
+
Candidate(
|
| 77 |
+
"budget-1971",
|
| 78 |
+
"1971 budget infographic",
|
| 79 |
+
"visual document",
|
| 80 |
+
"A historical chart comparing US outlays, including natural resources spending.",
|
| 81 |
+
str(ASSET_DIR / "doc2.jpg"),
|
| 82 |
+
str(ASSET_DIR / "doc2.jpg"),
|
| 83 |
+
),
|
| 84 |
+
Candidate(
|
| 85 |
+
"scoring-rules",
|
| 86 |
+
"Proper scoring rules paper",
|
| 87 |
+
"visual document",
|
| 88 |
+
"An academic page with lemmas, an algorithm, equations, and references.",
|
| 89 |
+
str(ASSET_DIR / "doc3.jpg"),
|
| 90 |
+
str(ASSET_DIR / "doc3.jpg"),
|
| 91 |
+
),
|
| 92 |
+
Candidate(
|
| 93 |
+
"road-safety",
|
| 94 |
+
"Road-safety assessment",
|
| 95 |
+
"visual document",
|
| 96 |
+
"An environmental assessment page about driver training, signs, and road closures.",
|
| 97 |
+
str(ASSET_DIR / "doc4.jpg"),
|
| 98 |
+
str(ASSET_DIR / "doc4.jpg"),
|
| 99 |
+
),
|
| 100 |
+
Candidate(
|
| 101 |
+
"mapo-tofu",
|
| 102 |
+
"Mapo tofu in motion",
|
| 103 |
+
"video",
|
| 104 |
+
"A short cooking clip showing the preparation of the Sichuan tofu dish.",
|
| 105 |
+
str(ASSET_DIR / "mapo_tofu.mp4"),
|
| 106 |
+
str(ASSET_DIR / "mapo_tofu.mp4"),
|
| 107 |
+
),
|
| 108 |
+
Candidate(
|
| 109 |
+
"zhajiang-noodles",
|
| 110 |
+
"Zhajiang noodles in motion",
|
| 111 |
+
"video",
|
| 112 |
+
"A short cooking clip showing noodles with a savory fermented-bean sauce.",
|
| 113 |
+
str(ASSET_DIR / "zhajiang_noodle.mp4"),
|
| 114 |
+
str(ASSET_DIR / "zhajiang_noodle.mp4"),
|
| 115 |
+
),
|
| 116 |
+
Candidate(
|
| 117 |
+
"vector-search",
|
| 118 |
+
"How vector search works",
|
| 119 |
+
"text",
|
| 120 |
+
"Dense retrieval maps queries and documents into one normalized vector space, then ranks candidates by cosine similarity.",
|
| 121 |
+
"Dense retrieval maps queries and documents into one normalized vector space, then ranks candidates by cosine similarity.",
|
| 122 |
+
),
|
| 123 |
+
Candidate(
|
| 124 |
+
"night-train",
|
| 125 |
+
"A quiet journey",
|
| 126 |
+
"text",
|
| 127 |
+
"夜行列车穿过雨中的城市,车窗映出霓虹灯和安静的乘客。",
|
| 128 |
+
"夜行列车穿过雨中的城市,车窗映出霓虹灯和安静的乘客。",
|
| 129 |
+
),
|
| 130 |
+
)
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
LOGGER.info("Loading %s on CPU", MODEL_ID)
|
| 134 |
+
MODEL = SentenceTransformer(
|
| 135 |
+
MODEL_ID,
|
| 136 |
+
trust_remote_code=True,
|
| 137 |
+
device="cpu",
|
| 138 |
+
model_kwargs={"dtype": torch.bfloat16, "low_cpu_mem_usage": True},
|
| 139 |
+
)
|
| 140 |
+
MODEL.eval()
|
| 141 |
+
torch.set_grad_enabled(False)
|
| 142 |
+
LOGGER.info("Model loaded; waiting for a ZeroGPU allocation")
|
| 143 |
+
|
| 144 |
+
_CACHE_LOCK = threading.Lock()
|
| 145 |
+
_INFERENCE_LOCK = threading.Lock()
|
| 146 |
+
_SHOWCASE_EMBEDDINGS: torch.Tensor | None = None
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def _file_path(value: Any) -> str | None:
|
| 150 |
+
"""Normalize Gradio file values across UI and API representations."""
|
| 151 |
+
if value is None:
|
| 152 |
+
return None
|
| 153 |
+
if isinstance(value, (str, Path)):
|
| 154 |
+
return str(value)
|
| 155 |
+
if isinstance(value, dict):
|
| 156 |
+
path = value.get("path") or value.get("name")
|
| 157 |
+
return str(path) if path else None
|
| 158 |
+
path = getattr(value, "path", None) or getattr(value, "name", None)
|
| 159 |
+
return str(path) if path else None
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def _is_video(path: str) -> bool:
|
| 163 |
+
return Path(path.split("?", 1)[0]).suffix.lower() in {
|
| 164 |
+
".mp4",
|
| 165 |
+
".webm",
|
| 166 |
+
".mov",
|
| 167 |
+
".mkv",
|
| 168 |
+
".avi",
|
| 169 |
+
".mpeg",
|
| 170 |
+
".mpg",
|
| 171 |
+
}
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
def _multimodal_payload(text: str | None, image: Any, video: Any) -> tuple[Any, str]:
|
| 175 |
+
text = (text or "").strip()
|
| 176 |
+
image_path = _file_path(image)
|
| 177 |
+
video_path = _file_path(video)
|
| 178 |
+
if image_path and video_path:
|
| 179 |
+
raise gr.Error("Choose one visual query: an image or a video, not both.")
|
| 180 |
+
if image_path:
|
| 181 |
+
if text:
|
| 182 |
+
return {"image": image_path, "text": text}, "image + text"
|
| 183 |
+
return image_path, "image"
|
| 184 |
+
if video_path:
|
| 185 |
+
if text:
|
| 186 |
+
return {"video": video_path, "text": text}, "video + text"
|
| 187 |
+
return video_path, "video"
|
| 188 |
+
if text:
|
| 189 |
+
return text, "text"
|
| 190 |
+
raise gr.Error("Add a text, image, or video query to begin.")
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
def _parse_text_candidates(raw: str | None) -> list[Candidate]:
|
| 194 |
+
candidates: list[Candidate] = []
|
| 195 |
+
lines = [line.strip() for line in (raw or "").splitlines() if line.strip()]
|
| 196 |
+
if len(lines) > MAX_CUSTOM_TEXTS:
|
| 197 |
+
raise gr.Error(f"Use at most {MAX_CUSTOM_TEXTS} custom text candidates.")
|
| 198 |
+
for index, line in enumerate(lines, start=1):
|
| 199 |
+
if "::" in line:
|
| 200 |
+
title, body = (part.strip() for part in line.split("::", 1))
|
| 201 |
+
title = title or f"Custom text {index}"
|
| 202 |
+
body = body or title
|
| 203 |
+
else:
|
| 204 |
+
title = f"Custom text {index}"
|
| 205 |
+
body = line
|
| 206 |
+
candidates.append(
|
| 207 |
+
Candidate(
|
| 208 |
+
f"custom-text-{index}",
|
| 209 |
+
title[:80],
|
| 210 |
+
"text",
|
| 211 |
+
body[:240],
|
| 212 |
+
body,
|
| 213 |
+
)
|
| 214 |
+
)
|
| 215 |
+
return candidates
|
| 216 |
+
|
| 217 |
+
|
| 218 |
+
def _parse_media_candidates(raw: Any) -> list[Candidate]:
|
| 219 |
+
candidates: list[Candidate] = []
|
| 220 |
+
items = raw or []
|
| 221 |
+
if len(items) > MAX_CUSTOM_MEDIA:
|
| 222 |
+
raise gr.Error(f"Upload at most {MAX_CUSTOM_MEDIA} candidate media files.")
|
| 223 |
+
for index, item in enumerate(items, start=1):
|
| 224 |
+
media = item[0] if isinstance(item, (tuple, list)) else item
|
| 225 |
+
caption = item[1] if isinstance(item, (tuple, list)) and len(item) > 1 else None
|
| 226 |
+
path = _file_path(media)
|
| 227 |
+
if not path:
|
| 228 |
+
continue
|
| 229 |
+
kind = "video" if _is_video(path) else "image"
|
| 230 |
+
title = (caption or f"Uploaded {kind} {index}").strip()
|
| 231 |
+
candidates.append(
|
| 232 |
+
Candidate(
|
| 233 |
+
f"custom-media-{index}",
|
| 234 |
+
title[:80],
|
| 235 |
+
kind,
|
| 236 |
+
f"User-supplied {kind} candidate.",
|
| 237 |
+
path,
|
| 238 |
+
path,
|
| 239 |
+
)
|
| 240 |
+
)
|
| 241 |
+
return candidates
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def _encode(items: list[Any], *, query: bool) -> torch.Tensor:
|
| 245 |
+
method = MODEL.encode_query if query else MODEL.encode_document
|
| 246 |
+
with torch.inference_mode():
|
| 247 |
+
embeddings = method(
|
| 248 |
+
items,
|
| 249 |
+
batch_size=1,
|
| 250 |
+
convert_to_tensor=True,
|
| 251 |
+
normalize_embeddings=True,
|
| 252 |
+
show_progress_bar=False,
|
| 253 |
+
)
|
| 254 |
+
if embeddings.ndim == 1:
|
| 255 |
+
embeddings = embeddings.unsqueeze(0)
|
| 256 |
+
return embeddings.float().cpu()
|
| 257 |
+
|
| 258 |
+
|
| 259 |
+
def _truncate_normalize(embeddings: torch.Tensor, dimension: int) -> torch.Tensor:
|
| 260 |
+
return F.normalize(embeddings[..., :dimension], p=2, dim=-1)
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
def _dimension_scores(query: torch.Tensor, documents: torch.Tensor) -> dict[int, list[float]]:
|
| 264 |
+
scores: dict[int, list[float]] = {}
|
| 265 |
+
for dimension in MATRYOSHKA_DIMS:
|
| 266 |
+
query_d = _truncate_normalize(query, dimension)
|
| 267 |
+
docs_d = _truncate_normalize(documents, dimension)
|
| 268 |
+
scores[dimension] = (query_d @ docs_d.T).squeeze(0).tolist()
|
| 269 |
+
return scores
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
def _search_duration(*args: Any, **kwargs: Any) -> int:
|
| 273 |
+
"""Budget more time for the cold corpus pass and video queries."""
|
| 274 |
+
query_payload = args[0] if args else None
|
| 275 |
+
custom_candidates = args[2] if len(args) > 2 else []
|
| 276 |
+
include_showcase = bool(args[3]) if len(args) > 3 else True
|
| 277 |
+
query_has_video = (
|
| 278 |
+
isinstance(query_payload, dict) and bool(query_payload.get("video"))
|
| 279 |
+
) or (isinstance(query_payload, str) and _is_video(query_payload))
|
| 280 |
+
has_uploaded_video = any(item.kind == "video" for item in custom_candidates or [])
|
| 281 |
+
if include_showcase and _SHOWCASE_EMBEDDINGS is None:
|
| 282 |
+
return 240
|
| 283 |
+
return 120 if query_has_video or has_uploaded_video else 90
|
| 284 |
+
|
| 285 |
+
|
| 286 |
+
@spaces.GPU(duration=_search_duration)
|
| 287 |
+
def _run_search(
|
| 288 |
+
query_payload: Any,
|
| 289 |
+
query_kind: str,
|
| 290 |
+
custom_candidates: list[Candidate],
|
| 291 |
+
include_showcase: bool,
|
| 292 |
+
dimension: int,
|
| 293 |
+
progress: gr.Progress,
|
| 294 |
+
) -> tuple[torch.Tensor, torch.Tensor, list[Candidate], float, bool]:
|
| 295 |
+
global _SHOWCASE_EMBEDDINGS
|
| 296 |
+
|
| 297 |
+
started = time.perf_counter()
|
| 298 |
+
was_cold = include_showcase and _SHOWCASE_EMBEDDINGS is None
|
| 299 |
+
progress(0.04, desc="Allocating the 9B model on GPU")
|
| 300 |
+
MODEL.to("cuda")
|
| 301 |
+
try:
|
| 302 |
+
progress(0.16, desc=f"Encoding the {query_kind} query")
|
| 303 |
+
query_embedding = _encode([query_payload], query=True)
|
| 304 |
+
|
| 305 |
+
document_blocks: list[torch.Tensor] = []
|
| 306 |
+
candidates: list[Candidate] = []
|
| 307 |
+
if include_showcase:
|
| 308 |
+
with _CACHE_LOCK:
|
| 309 |
+
cached = _SHOWCASE_EMBEDDINGS
|
| 310 |
+
if cached is None:
|
| 311 |
+
progress(0.30, desc="Mapping the curated multimodal universe")
|
| 312 |
+
encoded = _encode([item.payload for item in SHOWCASE], query=False)
|
| 313 |
+
with _CACHE_LOCK:
|
| 314 |
+
if _SHOWCASE_EMBEDDINGS is None:
|
| 315 |
+
_SHOWCASE_EMBEDDINGS = encoded
|
| 316 |
+
cached = _SHOWCASE_EMBEDDINGS
|
| 317 |
+
document_blocks.append(cached)
|
| 318 |
+
candidates.extend(SHOWCASE)
|
| 319 |
+
|
| 320 |
+
if custom_candidates:
|
| 321 |
+
progress(0.72, desc="Encoding your candidate collection")
|
| 322 |
+
custom_embeddings = _encode([item.payload for item in custom_candidates], query=False)
|
| 323 |
+
document_blocks.append(custom_embeddings)
|
| 324 |
+
candidates.extend(custom_candidates)
|
| 325 |
+
|
| 326 |
+
if not document_blocks:
|
| 327 |
+
raise gr.Error("Include the showcase universe or add at least one candidate.")
|
| 328 |
+
|
| 329 |
+
documents = torch.cat(document_blocks, dim=0)
|
| 330 |
+
progress(0.92, desc=f"Ranking in {dimension:,} dimensions")
|
| 331 |
+
return query_embedding, documents, candidates, time.perf_counter() - started, was_cold
|
| 332 |
+
finally:
|
| 333 |
+
MODEL.to("cpu")
|
| 334 |
+
if torch.cuda.is_available():
|
| 335 |
+
torch.cuda.empty_cache()
|
| 336 |
+
|
| 337 |
+
|
| 338 |
+
def _score_tone(score: float) -> str:
|
| 339 |
+
if score >= 0.70:
|
| 340 |
+
return "high"
|
| 341 |
+
if score >= 0.40:
|
| 342 |
+
return "mid"
|
| 343 |
+
return "low"
|
| 344 |
+
|
| 345 |
+
|
| 346 |
+
def _render_summary(
|
| 347 |
+
query_kind: str,
|
| 348 |
+
dimension: int,
|
| 349 |
+
candidate_count: int,
|
| 350 |
+
elapsed: float,
|
| 351 |
+
was_cold: bool,
|
| 352 |
+
top_title: str,
|
| 353 |
+
top_score: float,
|
| 354 |
+
) -> str:
|
| 355 |
+
cache_note = "cold corpus map" if was_cold else "warm corpus cache"
|
| 356 |
+
return f"""
|
| 357 |
+
<section class="run-summary">
|
| 358 |
+
<div class="run-kicker"><span class="live-dot"></span> semantic field resolved</div>
|
| 359 |
+
<div class="run-main">
|
| 360 |
+
<div><span class="run-label">Top match</span><strong>{html.escape(top_title)}</strong></div>
|
| 361 |
+
<div class="hero-score"><span>{top_score:+.3f}</span><small>cosine</small></div>
|
| 362 |
+
</div>
|
| 363 |
+
<div class="run-meta">
|
| 364 |
+
<span>{html.escape(query_kind)}</span><i></i>
|
| 365 |
+
<span>{dimension:,}D</span><i></i>
|
| 366 |
+
<span>{candidate_count} candidates</span><i></i>
|
| 367 |
+
<span>{elapsed:.1f}s</span><i></i>
|
| 368 |
+
<span>{cache_note}</span>
|
| 369 |
+
</div>
|
| 370 |
+
</section>
|
| 371 |
+
"""
|
| 372 |
+
|
| 373 |
+
|
| 374 |
+
def _render_rankings(ranked: list[tuple[Candidate, float]]) -> str:
|
| 375 |
+
rows: list[str] = []
|
| 376 |
+
for rank, (candidate, score) in enumerate(ranked, start=1):
|
| 377 |
+
fill = min(100.0, max(2.0, max(0.0, score) * 100.0))
|
| 378 |
+
rows.append(
|
| 379 |
+
f"""
|
| 380 |
+
<article class="rank-row {'winner' if rank == 1 else ''}">
|
| 381 |
+
<div class="rank-number">{rank:02d}</div>
|
| 382 |
+
<div class="rank-copy">
|
| 383 |
+
<div class="rank-title-line">
|
| 384 |
+
<strong>{html.escape(candidate.title)}</strong>
|
| 385 |
+
<span class="kind-pill">{html.escape(candidate.kind)}</span>
|
| 386 |
+
</div>
|
| 387 |
+
<p>{html.escape(candidate.description)}</p>
|
| 388 |
+
<div class="score-track"><span style="width:{fill:.1f}%"></span></div>
|
| 389 |
+
</div>
|
| 390 |
+
<div class="rank-score { _score_tone(score) }">{score:+.3f}<small>cos</small></div>
|
| 391 |
+
</article>
|
| 392 |
+
"""
|
| 393 |
+
)
|
| 394 |
+
return '<div class="ranking-stack">' + "".join(rows) + "</div>"
|
| 395 |
+
|
| 396 |
+
|
| 397 |
+
def _render_curve(series: dict[str, list[float]], title: str) -> str:
|
| 398 |
+
width, height = 820, 300
|
| 399 |
+
left, right, top, bottom = 58, 20, 28, 48
|
| 400 |
+
plot_w, plot_h = width - left - right, height - top - bottom
|
| 401 |
+
all_values = [value for values in series.values() for value in values]
|
| 402 |
+
low = max(-1.0, min(all_values) - 0.08)
|
| 403 |
+
high = min(1.0, max(all_values) + 0.08)
|
| 404 |
+
if high - low < 0.2:
|
| 405 |
+
midpoint = (high + low) / 2
|
| 406 |
+
low, high = max(-1.0, midpoint - 0.1), min(1.0, midpoint + 0.1)
|
| 407 |
+
|
| 408 |
+
def x_at(index: int) -> float:
|
| 409 |
+
return left + index * plot_w / (len(MATRYOSHKA_DIMS) - 1)
|
| 410 |
+
|
| 411 |
+
def y_at(value: float) -> float:
|
| 412 |
+
return top + (high - value) * plot_h / max(1e-8, high - low)
|
| 413 |
+
|
| 414 |
+
grid: list[str] = []
|
| 415 |
+
for tick in range(5):
|
| 416 |
+
value = high - tick * (high - low) / 4
|
| 417 |
+
y = y_at(value)
|
| 418 |
+
grid.append(
|
| 419 |
+
f'<line x1="{left}" y1="{y:.1f}" x2="{width-right}" y2="{y:.1f}" class="chart-grid" />'
|
| 420 |
+
f'<text x="{left-10}" y="{y+4:.1f}" text-anchor="end" class="chart-axis">{value:+.2f}</text>'
|
| 421 |
+
)
|
| 422 |
+
for index, dimension in enumerate(MATRYOSHKA_DIMS):
|
| 423 |
+
x = x_at(index)
|
| 424 |
+
grid.append(f'<text x="{x:.1f}" y="{height-18}" text-anchor="middle" class="chart-axis">{dimension}</text>')
|
| 425 |
+
|
| 426 |
+
colors = ("#ffb86b", "#78e8df", "#a994ff", "#ff7aa2", "#9ad45b")
|
| 427 |
+
paths: list[str] = []
|
| 428 |
+
legend: list[str] = []
|
| 429 |
+
for series_index, (name, values) in enumerate(series.items()):
|
| 430 |
+
color = colors[series_index % len(colors)]
|
| 431 |
+
points = " ".join(f"{x_at(i):.1f},{y_at(value):.1f}" for i, value in enumerate(values))
|
| 432 |
+
circles = "".join(
|
| 433 |
+
f'<circle cx="{x_at(i):.1f}" cy="{y_at(value):.1f}" r="3.5" fill="{color}" />'
|
| 434 |
+
for i, value in enumerate(values)
|
| 435 |
+
)
|
| 436 |
+
paths.append(f'<polyline points="{points}" fill="none" stroke="{color}" stroke-width="3" />{circles}')
|
| 437 |
+
legend.append(
|
| 438 |
+
f'<span><b style="background:{color}"></b>{html.escape(name[:38])}</span>'
|
| 439 |
+
)
|
| 440 |
+
|
| 441 |
+
return f"""
|
| 442 |
+
<section class="viz-card">
|
| 443 |
+
<div class="viz-heading"><div><small>MATRYOSHKA SCOPE</small><h3>{html.escape(title)}</h3></div><span>64 → 4096 dimensions</span></div>
|
| 444 |
+
<svg class="dimension-chart" viewBox="0 0 {width} {height}" role="img" aria-label="Similarity by embedding dimension">
|
| 445 |
+
{''.join(grid)}{''.join(paths)}
|
| 446 |
+
</svg>
|
| 447 |
+
<div class="chart-legend">{''.join(legend)}</div>
|
| 448 |
+
<p class="viz-note">Cosine similarity at every native truncation size. Compare trends, not universal thresholds.</p>
|
| 449 |
+
</section>
|
| 450 |
+
"""
|
| 451 |
+
|
| 452 |
+
|
| 453 |
+
def _fingerprint_svg(vector: torch.Tensor, label: str) -> str:
|
| 454 |
+
values = vector.detach().float().flatten()
|
| 455 |
+
bar_count = min(96, values.numel())
|
| 456 |
+
chunks = torch.tensor_split(values, bar_count)
|
| 457 |
+
samples = [float(chunk.mean()) for chunk in chunks]
|
| 458 |
+
scale = max(max(abs(value) for value in samples), 1e-6)
|
| 459 |
+
width, height = 800, 210
|
| 460 |
+
center = 104
|
| 461 |
+
bar_w = (width - 24) / bar_count
|
| 462 |
+
bars: list[str] = []
|
| 463 |
+
for index, value in enumerate(samples):
|
| 464 |
+
magnitude = min(84.0, abs(value) / scale * 84.0)
|
| 465 |
+
x = 12 + index * bar_w
|
| 466 |
+
y = center - magnitude if value >= 0 else center
|
| 467 |
+
color = "#70e4da" if value >= 0 else "#ff9d57"
|
| 468 |
+
bars.append(
|
| 469 |
+
f'<rect x="{x:.1f}" y="{y:.1f}" width="{max(1.2, bar_w-1.5):.1f}" height="{magnitude:.1f}" rx="1.5" fill="{color}" opacity=".9" />'
|
| 470 |
+
)
|
| 471 |
+
return f"""
|
| 472 |
+
<section class="viz-card fingerprint-card">
|
| 473 |
+
<div class="viz-heading"><div><small>VECTOR FINGERPRINT</small><h3>{html.escape(label)}</h3></div><span>{values.numel():,} values</span></div>
|
| 474 |
+
<svg class="fingerprint" viewBox="0 0 {width} {height}" role="img" aria-label="Compressed signed embedding fingerprint">
|
| 475 |
+
<line x1="12" y1="{center}" x2="{width-12}" y2="{center}" class="zero-line" />
|
| 476 |
+
{''.join(bars)}
|
| 477 |
+
</svg>
|
| 478 |
+
<div class="fingerprint-key"><span><b class="positive"></b>positive</span><span><b class="negative"></b>negative</span><em>96 pooled slices · shape, not magnitude</em></div>
|
| 479 |
+
</section>
|
| 480 |
+
"""
|
| 481 |
+
|
| 482 |
+
|
| 483 |
+
def search_experience(
|
| 484 |
+
query_text: str,
|
| 485 |
+
query_image: Any,
|
| 486 |
+
query_video: Any,
|
| 487 |
+
custom_texts: str,
|
| 488 |
+
candidate_media: Any,
|
| 489 |
+
include_showcase: bool,
|
| 490 |
+
dimension: int,
|
| 491 |
+
progress: gr.Progress = gr.Progress(track_tqdm=True),
|
| 492 |
+
) -> tuple[str, str, list[tuple[str, str]], str, str, dict[str, Any]]:
|
| 493 |
+
"""Search a mixed text/image/video collection with a multimodal query."""
|
| 494 |
+
dimension = int(dimension)
|
| 495 |
+
if dimension not in MATRYOSHKA_DIMS:
|
| 496 |
+
raise gr.Error("Choose one of the model's native Matryoshka dimensions.")
|
| 497 |
+
query_payload, query_kind = _multimodal_payload(query_text, query_image, query_video)
|
| 498 |
+
custom_candidates = _parse_text_candidates(custom_texts) + _parse_media_candidates(candidate_media)
|
| 499 |
+
|
| 500 |
+
with _INFERENCE_LOCK:
|
| 501 |
+
query, documents, candidates, elapsed, was_cold = _run_search(
|
| 502 |
+
query_payload,
|
| 503 |
+
query_kind,
|
| 504 |
+
custom_candidates,
|
| 505 |
+
bool(include_showcase),
|
| 506 |
+
dimension,
|
| 507 |
+
progress,
|
| 508 |
+
)
|
| 509 |
+
dimension_map = _dimension_scores(query, documents)
|
| 510 |
+
chosen_scores = dimension_map[dimension]
|
| 511 |
+
order = sorted(range(len(candidates)), key=lambda index: chosen_scores[index], reverse=True)
|
| 512 |
+
ranked = [(candidates[index], float(chosen_scores[index])) for index in order]
|
| 513 |
+
|
| 514 |
+
top_candidate, top_score = ranked[0]
|
| 515 |
+
summary = _render_summary(
|
| 516 |
+
query_kind,
|
| 517 |
+
dimension,
|
| 518 |
+
len(candidates),
|
| 519 |
+
elapsed,
|
| 520 |
+
was_cold,
|
| 521 |
+
top_candidate.title,
|
| 522 |
+
top_score,
|
| 523 |
+
)
|
| 524 |
+
rankings = _render_rankings(ranked[:8])
|
| 525 |
+
gallery = [
|
| 526 |
+
(candidate.media_path, f"#{rank} · {candidate.title} · cosine {score:+.3f}")
|
| 527 |
+
for rank, (candidate, score) in enumerate(ranked, start=1)
|
| 528 |
+
if candidate.media_path
|
| 529 |
+
][:8]
|
| 530 |
+
|
| 531 |
+
curve_series: dict[str, list[float]] = {}
|
| 532 |
+
for index in order[:4]:
|
| 533 |
+
curve_series[candidates[index].title] = [dimension_map[dim][index] for dim in MATRYOSHKA_DIMS]
|
| 534 |
+
curve = _render_curve(curve_series, "Does the ranking survive compression?")
|
| 535 |
+
fingerprint = _fingerprint_svg(query[0, :dimension], f"Query · {query_kind} · {dimension:,}D")
|
| 536 |
+
|
| 537 |
+
diagnostics = {
|
| 538 |
+
"model": MODEL_ID,
|
| 539 |
+
"query_modality": query_kind,
|
| 540 |
+
"selected_dimension": dimension,
|
| 541 |
+
"full_embedding_dimension": int(query.shape[-1]),
|
| 542 |
+
"l2_norm_after_truncation": round(float(_truncate_normalize(query, dimension).norm()), 6),
|
| 543 |
+
"candidate_count": len(candidates),
|
| 544 |
+
"gpu_pass_seconds": round(elapsed, 3),
|
| 545 |
+
"showcase_cache": "created" if was_cold else "reused" if include_showcase else "not_requested",
|
| 546 |
+
"top_matches": [
|
| 547 |
+
{"rank": rank, "title": item.title, "modality": item.kind, "cosine": round(score, 6)}
|
| 548 |
+
for rank, (item, score) in enumerate(ranked[:5], start=1)
|
| 549 |
+
],
|
| 550 |
+
"query_vector_preview": [round(float(value), 6) for value in query[0, :12]],
|
| 551 |
+
"note": "Cosine similarity is a ranking signal, not a calibrated probability.",
|
| 552 |
+
}
|
| 553 |
+
return summary, rankings, gallery, curve, fingerprint, diagnostics
|
| 554 |
+
|
| 555 |
+
|
| 556 |
+
def _pair_duration(*args: Any, **kwargs: Any) -> int:
|
| 557 |
+
payloads = args[:2]
|
| 558 |
+
has_video = any(
|
| 559 |
+
(isinstance(value, dict) and bool(value.get("video")))
|
| 560 |
+
or (isinstance(value, str) and _is_video(value))
|
| 561 |
+
for value in payloads
|
| 562 |
+
)
|
| 563 |
+
return 150 if has_video else 90
|
| 564 |
+
|
| 565 |
+
|
| 566 |
+
@spaces.GPU(duration=_pair_duration)
|
| 567 |
+
def _run_pair(
|
| 568 |
+
query_payload: Any,
|
| 569 |
+
candidate_payload: Any,
|
| 570 |
+
progress: gr.Progress,
|
| 571 |
+
) -> tuple[torch.Tensor, torch.Tensor, float]:
|
| 572 |
+
started = time.perf_counter()
|
| 573 |
+
progress(0.08, desc="Allocating the model on GPU")
|
| 574 |
+
MODEL.to("cuda")
|
| 575 |
+
try:
|
| 576 |
+
progress(0.35, desc="Encoding the query")
|
| 577 |
+
query = _encode([query_payload], query=True)
|
| 578 |
+
progress(0.68, desc="Encoding the candidate")
|
| 579 |
+
candidate = _encode([candidate_payload], query=False)
|
| 580 |
+
return query, candidate, time.perf_counter() - started
|
| 581 |
+
finally:
|
| 582 |
+
MODEL.to("cpu")
|
| 583 |
+
if torch.cuda.is_available():
|
| 584 |
+
torch.cuda.empty_cache()
|
| 585 |
+
|
| 586 |
+
|
| 587 |
+
def _interpret_score(score: float) -> tuple[str, str]:
|
| 588 |
+
if score >= 0.75:
|
| 589 |
+
return "high alignment", "These inputs occupy a very similar region for this retrieval model."
|
| 590 |
+
if score >= 0.50:
|
| 591 |
+
return "meaningful alignment", "The model sees a substantial semantic relationship."
|
| 592 |
+
if score >= 0.25:
|
| 593 |
+
return "weak alignment", "There is some overlap, but stronger candidates may rank above it."
|
| 594 |
+
return "low alignment", "The model places these inputs relatively far apart."
|
| 595 |
+
|
| 596 |
+
|
| 597 |
+
def _render_pair_score(score: float, dimension: int, query_kind: str, candidate_kind: str, elapsed: float) -> str:
|
| 598 |
+
label, explanation = _interpret_score(score)
|
| 599 |
+
ring = min(100.0, max(0.0, (score + 1.0) * 50.0))
|
| 600 |
+
return f"""
|
| 601 |
+
<section class="pair-score-card">
|
| 602 |
+
<div class="score-orbit" style="--score:{ring:.2f}">
|
| 603 |
+
<div><strong>{score:+.3f}</strong><span>cosine</span></div>
|
| 604 |
+
</div>
|
| 605 |
+
<div class="pair-score-copy">
|
| 606 |
+
<small>PAIRWISE READOUT</small>
|
| 607 |
+
<h2>{html.escape(label)}</h2>
|
| 608 |
+
<p>{html.escape(explanation)}</p>
|
| 609 |
+
<div class="run-meta"><span>{html.escape(query_kind)}</span><i></i><span>{html.escape(candidate_kind)}</span><i></i><span>{dimension:,}D</span><i></i><span>{elapsed:.1f}s</span></div>
|
| 610 |
+
</div>
|
| 611 |
+
</section>
|
| 612 |
+
"""
|
| 613 |
+
|
| 614 |
+
|
| 615 |
+
def compare_experience(
|
| 616 |
+
query_text: str,
|
| 617 |
+
query_image: Any,
|
| 618 |
+
query_video: Any,
|
| 619 |
+
candidate_text: str,
|
| 620 |
+
candidate_image: Any,
|
| 621 |
+
candidate_video: Any,
|
| 622 |
+
dimension: int,
|
| 623 |
+
progress: gr.Progress = gr.Progress(track_tqdm=True),
|
| 624 |
+
) -> tuple[str, str, str, dict[str, Any]]:
|
| 625 |
+
"""Compare any two supported inputs across all Matryoshka dimensions."""
|
| 626 |
+
dimension = int(dimension)
|
| 627 |
+
query_payload, query_kind = _multimodal_payload(query_text, query_image, query_video)
|
| 628 |
+
candidate_payload, candidate_kind = _multimodal_payload(candidate_text, candidate_image, candidate_video)
|
| 629 |
+
with _INFERENCE_LOCK:
|
| 630 |
+
query, candidate, elapsed = _run_pair(query_payload, candidate_payload, progress)
|
| 631 |
+
scores_by_dimension = {
|
| 632 |
+
dim: float((_truncate_normalize(query, dim) @ _truncate_normalize(candidate, dim).T).item())
|
| 633 |
+
for dim in MATRYOSHKA_DIMS
|
| 634 |
+
}
|
| 635 |
+
selected_score = scores_by_dimension[dimension]
|
| 636 |
+
score_card = _render_pair_score(selected_score, dimension, query_kind, candidate_kind, elapsed)
|
| 637 |
+
curve = _render_curve(
|
| 638 |
+
{f"{query_kind} → {candidate_kind}": [scores_by_dimension[dim] for dim in MATRYOSHKA_DIMS]},
|
| 639 |
+
"Semantic alignment under compression",
|
| 640 |
+
)
|
| 641 |
+
fingerprints = (
|
| 642 |
+
'<div class="fingerprint-pair">'
|
| 643 |
+
+ _fingerprint_svg(query[0, :dimension], f"Query · {query_kind}")
|
| 644 |
+
+ _fingerprint_svg(candidate[0, :dimension], f"Candidate · {candidate_kind}")
|
| 645 |
+
+ "</div>"
|
| 646 |
+
)
|
| 647 |
+
diagnostics = {
|
| 648 |
+
"model": MODEL_ID,
|
| 649 |
+
"query_modality": query_kind,
|
| 650 |
+
"candidate_modality": candidate_kind,
|
| 651 |
+
"selected_dimension": dimension,
|
| 652 |
+
"selected_cosine": round(selected_score, 6),
|
| 653 |
+
"cosine_by_dimension": {str(dim): round(score, 6) for dim, score in scores_by_dimension.items()},
|
| 654 |
+
"gpu_pass_seconds": round(elapsed, 3),
|
| 655 |
+
"note": "Interpret thresholds relative to a task-specific candidate set.",
|
| 656 |
+
}
|
| 657 |
+
return score_card, curve, fingerprints, diagnostics
|
| 658 |
+
|
| 659 |
+
|
| 660 |
+
CSS = """
|
| 661 |
+
:root {
|
| 662 |
+
--ink: #f6f3ec;
|
| 663 |
+
--muted: #a9abb5;
|
| 664 |
+
--panel: rgba(17, 20, 26, .82);
|
| 665 |
+
--line: rgba(255, 255, 255, .10);
|
| 666 |
+
--warm: #ffad66;
|
| 667 |
+
--cool: #71e2da;
|
| 668 |
+
--violet: #a994ff;
|
| 669 |
+
}
|
| 670 |
+
|
| 671 |
+
body, .gradio-container {
|
| 672 |
+
background:
|
| 673 |
+
radial-gradient(circle at 13% 0%, rgba(255, 143, 68, .16), transparent 30rem),
|
| 674 |
+
radial-gradient(circle at 92% 13%, rgba(88, 218, 211, .11), transparent 34rem),
|
| 675 |
+
#090b10 !important;
|
| 676 |
+
color: var(--ink) !important;
|
| 677 |
+
}
|
| 678 |
+
.gradio-container { max-width: 1380px !important; padding: 0 28px 60px !important; }
|
| 679 |
+
.gradio-container * { box-sizing: border-box; }
|
| 680 |
+
.gradio-container .prose { color: var(--ink); }
|
| 681 |
+
.gradio-container label, .gradio-container .label-wrap { color: #d7d7dc !important; }
|
| 682 |
+
.gradio-container input, .gradio-container textarea {
|
| 683 |
+
background: rgba(7, 9, 13, .72) !important;
|
| 684 |
+
border-color: rgba(255,255,255,.12) !important;
|
| 685 |
+
color: #f8f6f1 !important;
|
| 686 |
+
}
|
| 687 |
+
.gradio-container .block, .gradio-container .form {
|
| 688 |
+
border-color: var(--line) !important;
|
| 689 |
+
}
|
| 690 |
+
|
| 691 |
+
#hero { padding: 76px 4px 38px; }
|
| 692 |
+
.hero-shell { position: relative; overflow: hidden; border-bottom: 1px solid var(--line); padding-bottom: 44px; }
|
| 693 |
+
.eyebrow { display:flex; align-items:center; gap:10px; color:var(--cool); font-size:12px; font-weight:700; letter-spacing:.18em; text-transform:uppercase; }
|
| 694 |
+
.eyebrow:before { content:""; width:26px; height:1px; background:var(--cool); box-shadow:0 0 14px var(--cool); }
|
| 695 |
+
.hero-title { margin: 18px 0 8px; font-size: clamp(52px, 8vw, 112px); line-height:.88; letter-spacing:-.075em; font-weight:760; }
|
| 696 |
+
.hero-title .accent { color:transparent; -webkit-text-stroke:1px rgba(255,255,255,.62); }
|
| 697 |
+
.hero-title .dot { color:var(--warm); text-shadow:0 0 42px rgba(255,173,102,.6); }
|
| 698 |
+
.hero-sub { max-width:760px; margin:24px 0 0; font-size:clamp(17px,2vw,23px); line-height:1.55; color:#c0c1c8; }
|
| 699 |
+
.hero-grid { display:grid; grid-template-columns:1fr auto; align-items:end; gap:30px; }
|
| 700 |
+
.hero-stats { display:grid; grid-template-columns:repeat(2,minmax(110px,1fr)); gap:1px; background:var(--line); border:1px solid var(--line); min-width:350px; }
|
| 701 |
+
.hero-stats div { background:rgba(9,11,16,.88); padding:18px 20px; }
|
| 702 |
+
.hero-stats strong { display:block; font-size:28px; line-height:1; color:#fff; letter-spacing:-.04em; }
|
| 703 |
+
.hero-stats span { display:block; margin-top:8px; color:var(--muted); font-size:11px; text-transform:uppercase; letter-spacing:.12em; }
|
| 704 |
+
.capability-rail { display:flex; gap:8px; flex-wrap:wrap; margin-top:26px; }
|
| 705 |
+
.capability-rail span { border:1px solid var(--line); border-radius:99px; padding:7px 12px; color:#c9c9cf; font-size:12px; background:rgba(255,255,255,.025); }
|
| 706 |
+
|
| 707 |
+
.section-intro { margin:34px 0 18px; }
|
| 708 |
+
.section-intro small, .viz-heading small, .pair-score-copy small { color:var(--warm); letter-spacing:.16em; font-weight:750; font-size:11px; }
|
| 709 |
+
.section-intro h2 { font-size:30px; letter-spacing:-.035em; margin:6px 0; }
|
| 710 |
+
.section-intro p { color:var(--muted); margin:0; max-width:760px; }
|
| 711 |
+
|
| 712 |
+
.input-panel, .output-panel { background:linear-gradient(145deg,rgba(22,25,32,.9),rgba(12,14,19,.86)) !important; border:1px solid var(--line) !important; border-radius:18px !important; padding:18px !important; box-shadow:0 22px 70px rgba(0,0,0,.24); }
|
| 713 |
+
.primary-action { min-height:52px !important; border:0 !important; color:#16110d !important; font-weight:800 !important; letter-spacing:.01em; background:linear-gradient(105deg,#ff8e54,#ffd187) !important; box-shadow:0 10px 30px rgba(255,142,84,.2) !important; }
|
| 714 |
+
.primary-action:hover { transform:translateY(-1px); filter:brightness(1.04); }
|
| 715 |
+
|
| 716 |
+
.run-summary { border:1px solid rgba(112,228,218,.22); border-radius:18px; padding:22px 24px; background:linear-gradient(120deg,rgba(31,48,48,.54),rgba(17,19,25,.94)); margin-bottom:16px; }
|
| 717 |
+
.run-kicker { color:var(--cool); font-size:11px; text-transform:uppercase; letter-spacing:.16em; font-weight:750; }
|
| 718 |
+
.live-dot { display:inline-block; width:7px; height:7px; border-radius:99px; background:var(--cool); box-shadow:0 0 15px var(--cool); margin-right:8px; }
|
| 719 |
+
.run-main { display:flex; align-items:flex-end; justify-content:space-between; gap:20px; margin:14px 0 18px; }
|
| 720 |
+
.run-label { display:block; color:var(--muted); font-size:12px; margin-bottom:5px; }
|
| 721 |
+
.run-main strong { font-size:clamp(23px,3vw,38px); letter-spacing:-.04em; }
|
| 722 |
+
.hero-score { text-align:right; }
|
| 723 |
+
.hero-score span { display:block; font-size:40px; color:var(--cool); font-variant-numeric:tabular-nums; letter-spacing:-.05em; }
|
| 724 |
+
.hero-score small { color:var(--muted); text-transform:uppercase; letter-spacing:.14em; }
|
| 725 |
+
.run-meta { display:flex; align-items:center; gap:10px; flex-wrap:wrap; color:#aeb0b8; font-size:12px; }
|
| 726 |
+
.run-meta i { display:block; width:3px; height:3px; border-radius:99px; background:#555963; }
|
| 727 |
+
|
| 728 |
+
.ranking-stack { display:grid; gap:9px; }
|
| 729 |
+
.rank-row { display:grid; grid-template-columns:48px 1fr 80px; gap:16px; align-items:center; padding:15px 17px; border:1px solid var(--line); border-radius:14px; background:rgba(255,255,255,.025); transition:.2s ease; }
|
| 730 |
+
.rank-row:hover { transform:translateX(3px); border-color:rgba(255,173,102,.28); background:rgba(255,255,255,.04); }
|
| 731 |
+
.rank-row.winner { border-color:rgba(255,173,102,.35); background:linear-gradient(100deg,rgba(255,150,85,.11),rgba(255,255,255,.025)); }
|
| 732 |
+
.rank-number { color:#6d7079; font-size:14px; font-variant-numeric:tabular-nums; }
|
| 733 |
+
.rank-title-line { display:flex; gap:9px; align-items:center; flex-wrap:wrap; }
|
| 734 |
+
.rank-title-line strong { font-size:16px; color:#f7f3ec; }
|
| 735 |
+
.kind-pill { font-size:9px; letter-spacing:.09em; text-transform:uppercase; color:#bfc1c8; border:1px solid var(--line); border-radius:99px; padding:4px 7px; }
|
| 736 |
+
.rank-copy p { margin:4px 0 9px; color:#91949e; font-size:12px; line-height:1.45; }
|
| 737 |
+
.score-track { height:2px; background:rgba(255,255,255,.06); overflow:hidden; }
|
| 738 |
+
.score-track span { display:block; height:100%; background:linear-gradient(90deg,var(--warm),var(--cool)); }
|
| 739 |
+
.rank-score { text-align:right; font-size:18px; font-variant-numeric:tabular-nums; color:#b9bbc2; }
|
| 740 |
+
.rank-score.high { color:var(--cool); }.rank-score.mid { color:var(--warm); }
|
| 741 |
+
.rank-score small { display:block; font-size:8px; color:#747780; letter-spacing:.14em; margin-top:3px; text-transform:uppercase; }
|
| 742 |
+
|
| 743 |
+
.viz-card { border:1px solid var(--line); border-radius:18px; padding:20px; background:rgba(15,17,23,.78); overflow:hidden; }
|
| 744 |
+
.viz-heading { display:flex; align-items:flex-start; justify-content:space-between; gap:16px; }
|
| 745 |
+
.viz-heading h3 { margin:5px 0 0; font-size:20px; letter-spacing:-.025em; }
|
| 746 |
+
.viz-heading > span { color:#8f929d; font-size:11px; border:1px solid var(--line); border-radius:99px; padding:6px 9px; }
|
| 747 |
+
.dimension-chart, .fingerprint { width:100%; height:auto; overflow:visible; }
|
| 748 |
+
.chart-grid { stroke:rgba(255,255,255,.07); stroke-width:1; }
|
| 749 |
+
.chart-axis { fill:#777b85; font-size:10px; font-family:ui-monospace,SFMono-Regular,Menlo,monospace; }
|
| 750 |
+
.chart-legend { display:flex; flex-wrap:wrap; gap:8px 16px; }
|
| 751 |
+
.chart-legend span { color:#aeb0b8; font-size:11px; }
|
| 752 |
+
.chart-legend b { display:inline-block; width:7px; height:7px; border-radius:99px; margin-right:6px; }
|
| 753 |
+
.viz-note { margin:14px 0 0; color:#6f727c; font-size:11px; }
|
| 754 |
+
.zero-line { stroke:rgba(255,255,255,.18); stroke-width:1; }
|
| 755 |
+
.fingerprint-key { display:flex; gap:14px; align-items:center; flex-wrap:wrap; color:#848791; font-size:10px; text-transform:uppercase; letter-spacing:.08em; }
|
| 756 |
+
.fingerprint-key b { display:inline-block; width:7px; height:7px; border-radius:2px; margin-right:5px; }.fingerprint-key .positive{background:var(--cool)}.fingerprint-key .negative{background:var(--warm)}
|
| 757 |
+
.fingerprint-key em { margin-left:auto; text-transform:none; letter-spacing:0; color:#6e717a; }
|
| 758 |
+
.fingerprint-pair { display:grid; grid-template-columns:1fr 1fr; gap:12px; }
|
| 759 |
+
|
| 760 |
+
.pair-score-card { display:grid; grid-template-columns:190px 1fr; gap:30px; align-items:center; border:1px solid rgba(169,148,255,.24); border-radius:20px; padding:26px; background:linear-gradient(125deg,rgba(50,40,80,.35),rgba(14,17,23,.92)); }
|
| 761 |
+
.score-orbit { width:170px; aspect-ratio:1; border-radius:50%; padding:2px; background:conic-gradient(var(--violet) calc(var(--score)*1%),rgba(255,255,255,.07) 0); box-shadow:0 0 55px rgba(169,148,255,.12); }
|
| 762 |
+
.score-orbit > div { width:100%; height:100%; border-radius:50%; background:#0d0f15; display:flex; flex-direction:column; align-items:center; justify-content:center; }
|
| 763 |
+
.score-orbit strong { font-size:38px; letter-spacing:-.05em; font-variant-numeric:tabular-nums; }.score-orbit span{color:#777b85;font-size:10px;text-transform:uppercase;letter-spacing:.14em}
|
| 764 |
+
.pair-score-copy h2 { margin:6px 0 8px; font-size:34px; letter-spacing:-.045em; }.pair-score-copy p{color:#aeb0b8;max-width:580px}
|
| 765 |
+
|
| 766 |
+
.model-note { margin:28px 0 0; padding:18px 20px; border-left:2px solid var(--warm); background:rgba(255,173,102,.05); color:#9ea0aa; font-size:12px; line-height:1.6; }
|
| 767 |
+
.space-footer { display:flex; justify-content:space-between; align-items:center; gap:20px; flex-wrap:wrap; border-top:1px solid var(--line); padding:28px 4px 0; margin-top:44px; color:#767984; font-size:12px; }
|
| 768 |
+
.space-footer a { color:#c4c6cd !important; text-decoration:none; }.space-footer a:hover{color:var(--warm)!important}
|
| 769 |
+
|
| 770 |
+
@media (max-width: 900px) {
|
| 771 |
+
.gradio-container { padding:0 14px 40px !important; }
|
| 772 |
+
#hero { padding-top:44px; }
|
| 773 |
+
.hero-grid { grid-template-columns:1fr; }
|
| 774 |
+
.hero-stats { min-width:0; width:100%; }
|
| 775 |
+
.fingerprint-pair { grid-template-columns:1fr; }
|
| 776 |
+
}
|
| 777 |
+
@media (max-width: 620px) {
|
| 778 |
+
.hero-title { font-size:50px; }
|
| 779 |
+
.hero-stats { grid-template-columns:1fr 1fr; }
|
| 780 |
+
.rank-row { grid-template-columns:34px 1fr 64px; gap:8px; padding:12px 10px; }
|
| 781 |
+
.rank-copy p { display:none; }
|
| 782 |
+
.pair-score-card { grid-template-columns:1fr; text-align:center; }
|
| 783 |
+
.score-orbit { margin:auto; }.pair-score-copy .run-meta{justify-content:center}
|
| 784 |
+
}
|
| 785 |
+
"""
|
| 786 |
+
|
| 787 |
+
|
| 788 |
+
HERO = """
|
| 789 |
+
<div id="hero" class="hero-shell">
|
| 790 |
+
<div class="eyebrow">Tencent WeMM · multimodal embedding</div>
|
| 791 |
+
<div class="hero-grid">
|
| 792 |
+
<div>
|
| 793 |
+
<h1 class="hero-title">One space<span class="accent">.<br>Every medium</span><span class="dot">.</span></h1>
|
| 794 |
+
<p class="hero-sub">Search meaning—not file types—across text, images, video, charts, and visual documents in one shared semantic geometry.</p>
|
| 795 |
+
<div class="capability-rail"><span>text ↔ image</span><span>text ↔ video</span><span>visual documents</span><span>interleaved inputs</span><span>Matryoshka embeddings</span></div>
|
| 796 |
+
</div>
|
| 797 |
+
<div class="hero-stats">
|
| 798 |
+
<div><strong>4096</strong><span>native dimensions</span></div>
|
| 799 |
+
<div><strong>9B</strong><span>parameters</span></div>
|
| 800 |
+
<div><strong>80.6</strong><span>MMEB-v2 avg</span></div>
|
| 801 |
+
<div><strong>190</strong><span>MMEB-v3 tasks</span></div>
|
| 802 |
+
</div>
|
| 803 |
+
</div>
|
| 804 |
+
</div>
|
| 805 |
+
"""
|
| 806 |
+
|
| 807 |
+
|
| 808 |
+
INTRO_SEARCH = """
|
| 809 |
+
<div class="section-intro">
|
| 810 |
+
<small>01 / RETRIEVAL UNIVERSE</small>
|
| 811 |
+
<h2>Ask in one modality. Discover in another.</h2>
|
| 812 |
+
<p>Search the built-in field of screenshots, figures, dense documents, video, and multilingual text—or bring your own candidates.</p>
|
| 813 |
+
</div>
|
| 814 |
+
"""
|
| 815 |
+
|
| 816 |
+
|
| 817 |
+
INTRO_PAIR = """
|
| 818 |
+
<div class="section-intro">
|
| 819 |
+
<small>02 / VECTOR MICROSCOPE</small>
|
| 820 |
+
<h2>Put any two ideas under the lens.</h2>
|
| 821 |
+
<p>Use a query and candidate as text, image, video, or visual-plus-text. Then watch their alignment change as the embedding compresses.</p>
|
| 822 |
+
</div>
|
| 823 |
+
"""
|
| 824 |
+
|
| 825 |
+
|
| 826 |
+
INITIAL_SUMMARY = """
|
| 827 |
+
<section class="run-summary">
|
| 828 |
+
<div class="run-kicker"><span class="live-dot"></span> model ready</div>
|
| 829 |
+
<div class="run-main"><div><span class="run-label">Awaiting a query</span><strong>Search across media</strong></div><div class="hero-score"><span>—</span><small>cosine</small></div></div>
|
| 830 |
+
<div class="run-meta"><span>text</span><i></i><span>image</span><i></i><span>video</span><i></i><span>visual documents</span></div>
|
| 831 |
+
</section>
|
| 832 |
+
"""
|
| 833 |
+
|
| 834 |
+
|
| 835 |
+
INITIAL_RANKING = """
|
| 836 |
+
<div class="model-note">Choose a curated example below or compose a multimodal query. The first run maps the showcase corpus once; later searches reuse its CPU-cached 4,096D vectors.</div>
|
| 837 |
+
"""
|
| 838 |
+
|
| 839 |
+
|
| 840 |
+
with gr.Blocks(css=CSS, title="WeMM · Multimodal Embedding Universe", fill_width=True) as demo:
|
| 841 |
+
gr.HTML(HERO)
|
| 842 |
+
|
| 843 |
+
with gr.Tabs():
|
| 844 |
+
with gr.Tab("Search the universe", id="search"):
|
| 845 |
+
gr.HTML(INTRO_SEARCH)
|
| 846 |
+
with gr.Row(equal_height=False):
|
| 847 |
+
with gr.Column(scale=5, elem_classes="input-panel"):
|
| 848 |
+
query_text = gr.Textbox(
|
| 849 |
+
label="Query · text",
|
| 850 |
+
placeholder="Try: Which document explains temporary road closures?",
|
| 851 |
+
lines=3,
|
| 852 |
+
max_lines=7,
|
| 853 |
+
)
|
| 854 |
+
with gr.Row():
|
| 855 |
+
query_image = gr.Image(
|
| 856 |
+
label="Query · image (optional)",
|
| 857 |
+
type="filepath",
|
| 858 |
+
sources=["upload", "clipboard", "webcam"],
|
| 859 |
+
height=220,
|
| 860 |
+
)
|
| 861 |
+
query_video = gr.Video(
|
| 862 |
+
label="Query · video (optional)",
|
| 863 |
+
format="mp4",
|
| 864 |
+
height=220,
|
| 865 |
+
)
|
| 866 |
+
gr.Markdown("Add text to an image or video to create a joint multimodal query.")
|
| 867 |
+
with gr.Accordion("Build your own candidate collection", open=False):
|
| 868 |
+
custom_texts = gr.Textbox(
|
| 869 |
+
label="Text candidates",
|
| 870 |
+
placeholder="Title :: Candidate text\nAnother title :: Another candidate",
|
| 871 |
+
lines=5,
|
| 872 |
+
info=f"One candidate per line, up to {MAX_CUSTOM_TEXTS}.",
|
| 873 |
+
)
|
| 874 |
+
candidate_media = gr.Gallery(
|
| 875 |
+
label="Image + video candidates",
|
| 876 |
+
type="filepath",
|
| 877 |
+
file_types=["image", "video"],
|
| 878 |
+
sources=["upload"],
|
| 879 |
+
columns=3,
|
| 880 |
+
height=250,
|
| 881 |
+
)
|
| 882 |
+
include_showcase = gr.Checkbox(
|
| 883 |
+
value=True,
|
| 884 |
+
label="Include the curated multimodal universe",
|
| 885 |
+
info="10 candidates spanning text, images, video, figures, and visual documents.",
|
| 886 |
+
)
|
| 887 |
+
dimension = gr.Radio(
|
| 888 |
+
choices=list(MATRYOSHKA_DIMS),
|
| 889 |
+
value=1024,
|
| 890 |
+
label="Embedding budget",
|
| 891 |
+
info="Native Matryoshka dimensions; smaller vectors trade storage for fidelity.",
|
| 892 |
+
)
|
| 893 |
+
search_button = gr.Button("Map the semantic field →", variant="primary", elem_classes="primary-action")
|
| 894 |
+
|
| 895 |
+
with gr.Column(scale=7, elem_classes="output-panel"):
|
| 896 |
+
search_summary = gr.HTML(INITIAL_SUMMARY)
|
| 897 |
+
ranking_output = gr.HTML(INITIAL_RANKING)
|
| 898 |
+
|
| 899 |
+
with gr.Row(equal_height=False):
|
| 900 |
+
result_gallery = gr.Gallery(
|
| 901 |
+
value=[
|
| 902 |
+
(str(ASSET_DIR / "llama4_hgf.png"), "Visual document · Llama 4 model card"),
|
| 903 |
+
(str(ASSET_DIR / "doc2.jpg"), "Visual document · 1971 budget infographic"),
|
| 904 |
+
(str(ASSET_DIR / "mapo_tofu.mp4"), "Video · Mapo tofu in motion"),
|
| 905 |
+
(str(ASSET_DIR / "doc4.jpg"), "Visual document · road-safety assessment"),
|
| 906 |
+
],
|
| 907 |
+
label="Ranked visual field",
|
| 908 |
+
columns=4,
|
| 909 |
+
rows=2,
|
| 910 |
+
height=430,
|
| 911 |
+
object_fit="contain",
|
| 912 |
+
interactive=False,
|
| 913 |
+
buttons=["fullscreen", "download_all"],
|
| 914 |
+
)
|
| 915 |
+
with gr.Row(equal_height=False):
|
| 916 |
+
dimension_output = gr.HTML('<div class="viz-card"><div class="viz-heading"><div><small>MATRYOSHKA SCOPE</small><h3>Dimension stability appears here</h3></div></div></div>')
|
| 917 |
+
fingerprint_output = gr.HTML('<div class="viz-card"><div class="viz-heading"><div><small>VECTOR FINGERPRINT</small><h3>Your query vector appears here</h3></div></div></div>')
|
| 918 |
+
with gr.Accordion("Embedding telemetry · inspect the API payload", open=False):
|
| 919 |
+
diagnostics_output = gr.JSON(label="Diagnostics")
|
| 920 |
+
|
| 921 |
+
example_inputs = [
|
| 922 |
+
query_text,
|
| 923 |
+
query_image,
|
| 924 |
+
query_video,
|
| 925 |
+
custom_texts,
|
| 926 |
+
candidate_media,
|
| 927 |
+
include_showcase,
|
| 928 |
+
dimension,
|
| 929 |
+
]
|
| 930 |
+
example_outputs = [
|
| 931 |
+
search_summary,
|
| 932 |
+
ranking_output,
|
| 933 |
+
result_gallery,
|
| 934 |
+
dimension_output,
|
| 935 |
+
fingerprint_output,
|
| 936 |
+
diagnostics_output,
|
| 937 |
+
]
|
| 938 |
+
gr.Examples(
|
| 939 |
+
examples=[
|
| 940 |
+
["Which Llama 4 model variants are available?", None, None, "", None, True, 512],
|
| 941 |
+
["How is mapo tofu prepared?", None, None, "", None, True, 1024],
|
| 942 |
+
["Find the environmental assessment page about driver training and temporary road closures.", None, None, "", None, True, 256],
|
| 943 |
+
[
|
| 944 |
+
"Match this screenshot to the most relevant description.",
|
| 945 |
+
str(ASSET_DIR / "llama4_hgf.png"),
|
| 946 |
+
None,
|
| 947 |
+
"Llama family :: Scout and Maverick are multimodal mixture-of-experts model variants.\nRecipe :: Soft tofu simmered in spicy chili-bean sauce.",
|
| 948 |
+
None,
|
| 949 |
+
False,
|
| 950 |
+
256,
|
| 951 |
+
],
|
| 952 |
+
[
|
| 953 |
+
"What dish is being prepared in this clip?",
|
| 954 |
+
None,
|
| 955 |
+
str(ASSET_DIR / "mapo_tofu.mp4"),
|
| 956 |
+
"Sichuan classic :: Mapo tofu combines soft tofu with a spicy, numbing bean-paste sauce.\nSpaceflight :: A launch vehicle carries a satellite into orbit.",
|
| 957 |
+
None,
|
| 958 |
+
False,
|
| 959 |
+
512,
|
| 960 |
+
],
|
| 961 |
+
],
|
| 962 |
+
inputs=example_inputs,
|
| 963 |
+
outputs=example_outputs,
|
| 964 |
+
fn=search_experience,
|
| 965 |
+
cache_examples=True,
|
| 966 |
+
cache_mode="lazy",
|
| 967 |
+
label="Curated expeditions",
|
| 968 |
+
example_labels=[
|
| 969 |
+
"Find Llama 4 across a screenshot",
|
| 970 |
+
"Search a cooking video with text",
|
| 971 |
+
"Retrieve a dense safety document",
|
| 972 |
+
"Match an image to text candidates",
|
| 973 |
+
"Match a video to text candidates",
|
| 974 |
+
],
|
| 975 |
+
)
|
| 976 |
+
|
| 977 |
+
search_event = search_button.click(
|
| 978 |
+
fn=search_experience,
|
| 979 |
+
inputs=example_inputs,
|
| 980 |
+
outputs=example_outputs,
|
| 981 |
+
api_name="search",
|
| 982 |
+
api_description="Rank a mixed text/image/video collection using WeMM-Embedding-9B.",
|
| 983 |
+
concurrency_limit=1,
|
| 984 |
+
concurrency_id="wemm_gpu",
|
| 985 |
+
time_limit=300,
|
| 986 |
+
scroll_to_output=True,
|
| 987 |
+
)
|
| 988 |
+
query_text.submit(
|
| 989 |
+
fn=search_experience,
|
| 990 |
+
inputs=example_inputs,
|
| 991 |
+
outputs=example_outputs,
|
| 992 |
+
api_name=None,
|
| 993 |
+
api_visibility="private",
|
| 994 |
+
concurrency_limit=1,
|
| 995 |
+
concurrency_id="wemm_gpu",
|
| 996 |
+
time_limit=300,
|
| 997 |
+
scroll_to_output=True,
|
| 998 |
+
)
|
| 999 |
+
|
| 1000 |
+
with gr.Tab("Compare two ideas", id="compare"):
|
| 1001 |
+
gr.HTML(INTRO_PAIR)
|
| 1002 |
+
with gr.Row(equal_height=False):
|
| 1003 |
+
with gr.Column(elem_classes="input-panel"):
|
| 1004 |
+
gr.Markdown("### A · Query")
|
| 1005 |
+
pair_query_text = gr.Textbox(label="Text", placeholder="Describe or contextualize the query", lines=3)
|
| 1006 |
+
with gr.Row():
|
| 1007 |
+
pair_query_image = gr.Image(label="Image", type="filepath", height=210)
|
| 1008 |
+
pair_query_video = gr.Video(label="Video", format="mp4", height=210)
|
| 1009 |
+
with gr.Column(elem_classes="input-panel"):
|
| 1010 |
+
gr.Markdown("### B · Candidate")
|
| 1011 |
+
pair_candidate_text = gr.Textbox(label="Text", placeholder="Describe or contextualize the candidate", lines=3)
|
| 1012 |
+
with gr.Row():
|
| 1013 |
+
pair_candidate_image = gr.Image(label="Image", type="filepath", height=210)
|
| 1014 |
+
pair_candidate_video = gr.Video(label="Video", format="mp4", height=210)
|
| 1015 |
+
pair_dimension = gr.Radio(
|
| 1016 |
+
choices=list(MATRYOSHKA_DIMS),
|
| 1017 |
+
value=1024,
|
| 1018 |
+
label="Embedding budget",
|
| 1019 |
+
)
|
| 1020 |
+
compare_button = gr.Button("Measure semantic alignment →", variant="primary", elem_classes="primary-action")
|
| 1021 |
+
pair_score_output = gr.HTML('<div class="model-note">Add one modality on each side. You may pair visual media with text context.</div>')
|
| 1022 |
+
pair_curve_output = gr.HTML('<div class="viz-card"><div class="viz-heading"><div><small>MATRYOSHKA SCOPE</small><h3>Alignment by dimension appears here</h3></div></div></div>')
|
| 1023 |
+
pair_fingerprints_output = gr.HTML()
|
| 1024 |
+
with gr.Accordion("Pairwise telemetry", open=False):
|
| 1025 |
+
pair_diagnostics_output = gr.JSON(label="Diagnostics")
|
| 1026 |
+
|
| 1027 |
+
pair_inputs = [
|
| 1028 |
+
pair_query_text,
|
| 1029 |
+
pair_query_image,
|
| 1030 |
+
pair_query_video,
|
| 1031 |
+
pair_candidate_text,
|
| 1032 |
+
pair_candidate_image,
|
| 1033 |
+
pair_candidate_video,
|
| 1034 |
+
pair_dimension,
|
| 1035 |
+
]
|
| 1036 |
+
compare_button.click(
|
| 1037 |
+
fn=compare_experience,
|
| 1038 |
+
inputs=pair_inputs,
|
| 1039 |
+
outputs=[pair_score_output, pair_curve_output, pair_fingerprints_output, pair_diagnostics_output],
|
| 1040 |
+
api_name="compare",
|
| 1041 |
+
api_description="Compare a multimodal query and candidate across every native Matryoshka dimension.",
|
| 1042 |
+
concurrency_limit=1,
|
| 1043 |
+
concurrency_id="wemm_gpu",
|
| 1044 |
+
time_limit=240,
|
| 1045 |
+
scroll_to_output=True,
|
| 1046 |
+
)
|
| 1047 |
+
|
| 1048 |
+
gr.HTML(
|
| 1049 |
+
"""
|
| 1050 |
+
<div class="model-note"><strong>Read scores comparatively.</strong> Cosine similarity is useful for ranking candidates within a collection; it is not a calibrated confidence or a universal relevance grade. Audio is not supported by WeMM-Embedding-9B.</div>
|
| 1051 |
+
<footer class="space-footer"><span>WeMM-Embedding-9B · Apache-2.0 · built on Qwen3.5</span><span><a href="https://huggingface.co/tencent/WeMM-Embedding-9B" target="_blank">Model card ↗</a> <a href="https://arxiv.org/abs/2608.24053" target="_blank">Technical report ↗</a></span></footer>
|
| 1052 |
+
"""
|
| 1053 |
+
)
|
| 1054 |
+
|
| 1055 |
+
|
| 1056 |
+
if __name__ == "__main__":
|
| 1057 |
+
demo.queue(default_concurrency_limit=1, max_size=24).launch(
|
| 1058 |
+
allowed_paths=[str(ASSET_DIR)],
|
| 1059 |
+
show_error=True,
|
| 1060 |
+
)
|
assets/doc1.jpg
ADDED
|
Git LFS Details
|
assets/doc2.jpg
ADDED
|
Git LFS Details
|
assets/doc3.jpg
ADDED
|
Git LFS Details
|
assets/doc4.jpg
ADDED
|
Git LFS Details
|
assets/llama4_hgf.png
ADDED
|
Git LFS Details
|
assets/mapo_tofu.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:23bd7a2a9a554bc09084cb74e584ca6129292073efcd2350f180e81975f96ec5
|
| 3 |
+
size 5250889
|
assets/qwen2.5omni_hgf.png
ADDED
|
Git LFS Details
|
assets/zhajiang_noodle.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:640b98fde893982dc0e866e72d537d9798a5a8432a6a7abc8f76d630c897a1b1
|
| 3 |
+
size 3571831
|
requirements.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
gradio==6.26.0
|
| 2 |
+
spaces==0.51.1
|
| 3 |
+
transformers==5.2.0
|
| 4 |
+
sentence-transformers==5.7.0
|
| 5 |
+
accelerate==1.14.0
|
| 6 |
+
qwen-vl-utils[decord]==0.0.14
|