ViMoLa — WACV 2027

ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation
Bishoy Galoaa, Sarah Ostadabbas · Northeastern University · Accepted at WACV 2027

Resource	Link
Paper (arXiv)	https://arxiv.org/abs/2610.25595 · PDF
Code	https://github.com/ostadabbas/ViMoLa
Model (this repo)	https://huggingface.co/bishoygaloaa/ViMoLa

Predict discrete motion tokens from language and inject them into a frozen Wan 2.2 TI2V backbone. No trajectories at inference.

Teaser

Same prompt and first frame: Wan stays nearly static; ViMoLa produces the uphill motion from text alone.

Prompt: “pushing a boulder up a steep hill” — (1) input → (2) predicted motion field → (3) Wan 2.2 (static) → (4) ViMoLa (text + image only).

Checkpoints

File	Description
checkpoints/tokenizer.pt	Spatial motion VQ-VAE
checkpoints/t2m.pt	Text-to-motion transformer
checkpoints/adapters.pt	Zero-gated Wan motion adapters

Quick inference

git clone https://github.com/ostadabbas/ViMoLa.git
cd ViMoLa
pip install -r requirements.txt
python scripts/download_weights.py --out checkpoints
hf download Wan-AI/Wan2.2-TI2V-5B-Diffusers --local-dir models/Wan2.2-TI2V-5B-Diffusers
bash scripts/infer.sh assets/sisyphus_input.png "pushing a boulder up a steep hill"

Recommended: --adapter_strength 0.1.

Citation

@article{galoaa2026vimola,
  title   = {ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation},
  author  = {Galoaa, Bishoy and Ostadabbas, Sarah},
  journal = {arXiv preprint arXiv:2610.25595},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.25595}
}

@inproceedings{galoaa2027vimola,
  title     = {ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation},
  author    = {Galoaa, Bishoy and Ostadabbas, Sarah},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

License

MIT for ViMoLa weights/code in this repo. Wan 2.2 and datasets retain their own licenses.