Guomh0707 commited on
Commit
d50c2ff
·
verified ·
1 Parent(s): 758512a

Update model card

Browse files
Files changed (1) hide show
  1. README.md +120 -0
README.md ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: video-to-video
3
+ tags:
4
+ - video
5
+ - video-reconstruction
6
+ - video-generation
7
+ - video-representation
8
+ - v-rae
9
+ - rae
10
+ - pytorch
11
+ - arxiv:2608.13556
12
+ datasets:
13
+ - quchenyuan/UCF101-ZIP
14
+ ---
15
+
16
+ <div align="center">
17
+ <h1>V-RAE: Rethinking Video Latent Spaces for Generation</h1>
18
+ <p>
19
+ <a href="https://guominghui07.github.io/">Minghui Guo</a><sup>1</sup> &nbsp;&nbsp; <a href="https://sqwu.top/">Shengqiong Wu</a><sup>2</sup> &nbsp;&nbsp; <a href="https://haofei.vip/">Hao Fei</a><sup>2</sup>
20
+ </p>
21
+ <p>
22
+ <sup>1</sup>National University of Singapore &nbsp;&nbsp; <sup>2</sup>University of Oxford
23
+ </p>
24
+ </div>
25
+
26
+ <p align="center">
27
+ <a href="https://arxiv.org/abs/2608.13556"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b" alt="Paper"></a>
28
+ <a href="https://v-rae.github.io/"><img src="https://img.shields.io/badge/Homepage-Project%20Page-blue" alt="Project Page"></a>
29
+ <a href="https://github.com/V-RAE/V-RAE"><img src="https://img.shields.io/badge/Code-GitHub-black" alt="Code"></a>
30
+ </p>
31
+
32
+ V-RAE is a video representation autoencoder that builds compact generative
33
+ latents on top of frozen vision foundation model representations. It uses a
34
+ lightweight temporal pooling module to remove temporal redundancy while
35
+ preserving semantic structure, together with a video decoder for reconstructing
36
+ continuous motion.
37
+
38
+ ![V-RAE method](https://raw.githubusercontent.com/V-RAE/V-RAE/main/assets/V-RAE.png)
39
+
40
+ ![V-RAE overview](https://raw.githubusercontent.com/V-RAE/V-RAE/main/assets/V-RAE-overall.png)
41
+
42
+ ## Released Models
43
+
44
+ This repository provides V-RAE checkpoints based on four frozen visual encoders:
45
+
46
+ | Checkpoint | Encoder |
47
+ | --- | --- |
48
+ | `vrae/vrae_dinov3.pt` | DINOv3 ViT-L/16 |
49
+ | `vrae/vrae_siglip2.pt` | SigLIP2 ViT-L/16 |
50
+ | `vrae/vrae_vjepa2.1.pt` | V-JEPA2.1 ViT-L/16 |
51
+ | `vrae/vrae_eupe.pt` | EUPE ViT-B/16 |
52
+
53
+ Class-conditional VideoDiT checkpoints and their matching latent statistics are
54
+ also provided for UCF101 and Kinetics-600:
55
+
56
+ - UCF101: V-JEPA2.1 and EUPE variants
57
+ - Kinetics-600: V-JEPA2.1 variant
58
+
59
+ ## Quick Start
60
+
61
+ Install the official implementation:
62
+
63
+ ```bash
64
+ git clone https://github.com/V-RAE/V-RAE.git
65
+ cd V-RAE
66
+
67
+ conda create -n vrae python=3.10 -y
68
+ conda activate vrae
69
+ conda install -c conda-forge ffmpeg -y
70
+
71
+ pip install uv
72
+ uv pip install -e .
73
+ ```
74
+
75
+ Download the released checkpoints:
76
+
77
+ ```bash
78
+ hf download Guomh0707/V-RAE-Models --local-dir ckpts
79
+ ```
80
+
81
+ Download the matching frozen encoder by following the
82
+ [encoder instructions](https://github.com/V-RAE/V-RAE/blob/main/third_party/README.md#download-pre-trained-encoder-weights).
83
+
84
+ Place three input videos at `assets/sample1.mp4`, `assets/sample2.mp4`, and
85
+ `assets/sample3.mp4`, then run one of:
86
+
87
+ ```bash
88
+ python sampling.py dino
89
+ python sampling.py siglip
90
+ python sampling.py vjepa
91
+ python sampling.py eupe
92
+ ```
93
+
94
+ Reconstruction results are saved under `outputs/<variant>/`.
95
+
96
+ ## Results
97
+
98
+ | Evaluation | Result |
99
+ | --- | ---: |
100
+ | Kinetics-600 reconstruction | **2.13 rFVD** |
101
+ | UCF101 class-conditional generation | **117.86 gFVD** |
102
+ | Kinetics-600 class-conditional generation | **19.16 gFVD** |
103
+
104
+
105
+ ## Notes
106
+
107
+ - Each V-RAE checkpoint requires its corresponding frozen encoder weights.
108
+ - These are custom PyTorch checkpoints and require the official V-RAE codebase.
109
+ - This repository is intended primarily for research use.
110
+
111
+ ## Citation
112
+
113
+ ```bibtex
114
+ @article{guo2026vrae,
115
+ title = {V-RAE: Rethinking Video Latent Spaces for Generation},
116
+ author = {Guo, Minghui and Wu, Shengqiong and Fei, Hao},
117
+ journal = {arXiv preprint arXiv:2608.13556},
118
+ year = {2026},
119
+ }
120
+ ```