arxiv:2609.07470

Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy

Published on Sep 7

· Submitted by

Ayoub Kirouane on Sep 9

KIEFER

Upvote

Authors:

Ayoub Kirouane ,

Christos Petrocheilos

Abstract

Adding Greek to a robot vision-language-action model via machine-translated instructions reveals measurement pitfalls and shows bilingual training improves performance over monolingual baselines, though it remains far below English levels.

Generated by thinkingmachines/Inkling-Small

Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist for most languages. We study the addition of Greek to an open vision-language-action stack using only machine-rephrased instructions and no architecture changes. The main challenge is measurement rather than translation. Several plausible instruments produce false conclusions: a color-histogram metric rewards noise, a single-goal benchmark scores 84.6% under correct Greek and 82.6% under deliberately wrong instructions, training loss fails to predict Greek success, and single-run comparisons are dominated by seed variation. On a discriminative ninety-task suite with three seeds per arm, a multilingual text tower without Greek demonstrations remains at its wrong-instruction floor, while Greek-only training exceeds its control by at most 2.7 points. Bilingual training yields a consistent 6.7-7.1 point margin over its control and reaches about two fifths of English performance. The policy also overfits the translator's phrasing; training on seven phrasings per task approximately halves this penalty. Warm-starting from a language-adapted world model and unfreezing the text tower both degrade performance. The results support two practical requirements for low-resource robot-policy localization: build a guaranteed null before trusting a metric, and replicate low-resource-language results across seeds.

View arXiv page View PDF Add to collection

Community

ayoubkirouane

Paper author Paper submitter 1 day ago

Hi folks, author here.👋

Almost every robot foundation model out there is trained and evaluated in English. For most of the world's languages there is no robot demonstration data at all, and nobody is going to collect it anytime soon. So we wanted to know what it actually takes to add one of those languages to an existing policy, and more importantly, how you'd even tell if it worked.

We used Greek as the test case. Not because it's special, but because it's a language we can read and check, and it has the same problem as hundreds of others: zero robot data. We took an open VLA policy (Cosmos3), gave it Greek instructions that were machine-rephrased from English, changed nothing in the architecture, and trained. Getting it to run was the easy part.

The hard part was measurement. We kept building evaluations that looked reasonable and quietly gave us wrong answers. The one that bothered us most: a single-goal benchmark scored 84.6% with correct Greek instructions and 82.6% with deliberately wrong ones. The policy was finishing the task without really reading the text, and the benchmark couldn't tell. On top of that we had a color-histogram metric that rewarded noise, training loss that said nothing about Greek success, and single-run comparisons where the random seed mattered more than anything we changed.

So most of the paper is about getting the measurement right before making any claims about transfer. The Greek numbers are in there, but the point is that none of the traps we hit were about Greek. If you're trying to add Arabic, Swahili, Vietnamese, or anything else without a demo corpus, you'll walk into the same ones, and we hope this saves you some time.