Papers
arxiv:2606.03988

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

Published on Jun 3
· Submitted by
Weikai Huang
on Jun 8
#1 Paper of the day
Authors:
,
,
,
,
,
,
,
,
,

Abstract

Imaginative Perception Tokens (IPT) enhance vision-language models' spatial reasoning by providing intermediate perceptual representations that externalize what the model would perceive from alternative viewpoints, outperforming traditional text-based reasoning methods.

Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.

Community

Paper author Paper submitter

Can models think visually in space as humans do? Introducing: Imaginative Perception Tokens by UW, OpenAI, Microsoft, and AI2.

Imaginative Perception improves spatial reasoning in multimodal language models by teaching them to imagine useful visual perspectives as images. These imagined images help the model reason beyond the original view and answer spatial questions more accurately.

📄 Paper: Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
arXiv: https://arxiv.org/abs/2606.03988

💻 Code
• Training: https://github.com/weikaih04/Imaginative-Perception-Token
• Evaluation: https://github.com/weikaih04/Imaginative-Perception-Token-Eval

🤗 Data & Benchmarks
• Datasets (MVC + PET + PT): https://huggingface.co/collections/weikaih/imaginative-perception-token-data
• Spatial Mental Modeling Benchmark (human-verified): https://huggingface.co/collections/weikaih/spatial-mental-modeling-benchmark

overview (1)

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

the flow-matching loss that forecasts imaginative latent tokens is the standout detail here. it's clever because you're training the model to simulate an unseen view in token space, effectively aligning geometry with the downstream reasoning rather than forcing everything through language. this setup seems to boost generalization across occlusions and viewpoint changes, and it even helps when you don't emit an imagined image at test time. i wonder how sensitive the gains are to the granularity of the imagined tokens—do coarser tokens hurt more than fine-grained ones, and is there a sweet spot where flow-matching buys the most headroom? btw the arxivlens breakdown helped me parse the method details, https://arxivlens.com/PaperView/Details/imaginative-perception-tokens-enhance-spatial-reasoning-in-multimodal-language-models-9820-4a5a8d52

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2606.03988
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

Datasets citing this paper 13

Browse 13 datasets citing this paper

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2606.03988 in a Space README.md to link it from this page.

Collections including this paper 2