MInfoVisQA: A Multilingual Benchmark for Information-Rich Visual Question Answering
Abstract
MInfoVisQA, a 24-language parallel benchmark, reveals substantial cross-language robustness gaps in multimodal large language models for information-rich visual question answering.
Multimodal large language models (MLLMs) excel in vision-language tasks; yet, whether they maintain consistent understanding of information-rich visuals (InfoVis) when the surface language changes remains unclear. To quantify this capability, we introduce MInfoVisQA, a parallel benchmark with 8,960 QA configurations derived from 128 seed instances across 24 languages. We construct seed-matched, language-parallel visuals and QA pairs by disentangling numerical-structural data from text and programmatically synchronizing translated labels. We evaluate 25 MLLMs under two settings: Language-Aligned QA (LQA, matching visual-text languages) and Cross-Language QA (XQA, fixed QA language with varying visual languages). The results show that aggregate accuracy masks substantial robustness gaps: two models with comparable mean LQA accuracy (80.4\%) exhibit performance spreads of 7.8 and 20.3 percentage points across languages. Moreover, all 25 models score lower with non-English visuals under fixed English QA, underscoring the need for cross-language sensitivity metrics.
Community
We built MInfoVisQA to test this across 24 languages. We render the same charts and tables in each language, preserving their numbers and structure. This lets us ask the same question in English about each version and see what happens when only the language in the image changes. Even with the question and answer language fixed to English, we still see accuracy gaps across languages.
We’d love to hear what you find when you try it with your own models! 🤗
- 🏆 Leaderboard · Hugging Face Space
- 📄 Paper · arXiv:2611.21644
- 🌐 Project homepage · MInfoVisQA
- 💻 Code · GitHub repository
- 🤗 Dataset · Hugging Face Dataset
Get this paper in your agent:
hf papers read 2611.21644 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper