PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data
Abstract
A coding agent automates robot demonstration generation by simulating tasks, rendering photorealistic videos, and producing deployable training data at lower cost than teleoperation.
Teleoperation remains the default way to collect robot demonstrations, but every episode costs robot and operator time, and every new task starts from scratch. We ask whether a coding agent can take over the teleoperator's job. In our DataFarm, the agent receives a data request and one calibration episode of the robot in the target scene and rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge. The simulator supplies exact actions, and a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently. The output is real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and deployed on the real robot. Independent judges certify the expert and every rendered episode before admission. On five real-robot tasks, policies trained only on farm data match policies trained on as many teleoperated episodes (93 vs. 84 successes in 100 trials), and under an unseen tablecloth and coloured light they keep 87 successes where the teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs \21 of API usage on average, paid once per task, plus 5 for a render contract; every further episode costs \0.80 on average for simulation, rendering, and review, with no robot or operator. One hour of three-view data for a task costs 107 on average, these costs included, slightly below a quote of \117 for teleoperation in China and below 290 in labour alone in North America.
Community
Teleoperation remains the default way to collect robot demonstrations, but every episode costs robot and operator time, and every new task starts from scratch. We ask whether a coding agent can take over the teleoperator's job. In our DataFarm, the agent receives a data request and one calibration episode of the robot in the target scene and rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge. The simulator supplies exact actions, and a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently. The output is real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and deployed on the real robot. Independent judges certify the expert and every rendered episode before admission. On five real-robot tasks, policies trained only on farm data match policies trained on as many teleoperated episodes (93 vs. 84 successes in 100 trials), and under an unseen tablecloth and coloured light they keep 87 successes where the teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs $21 of API usage on average, paid once per task, plus $5 for a render contract; every further episode costs $0.80 on average for simulation, rendering, and review, with no robot or operator. One hour of three-view data for a task costs $107 on average, these costs included, slightly below a quote of $117 for teleoperation in China and below $290 in labour alone in North America.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents (2026)
- ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation (2026)
- Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies (2026)
- ReShoot: Generative Visual Domain Randomization of Recorded Robot Demonstrations for Visuomotor Policy Learning (2026)
- Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools (2026)
- AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images (2026)
- Encore: Few-Shot Agentic Discovery of Manipulation Strategies (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
A coding agent doing the teleoperator's job and a video model making the rollouts look real made me smile. Judges certifying every episode before it is admitted is a thoughtful touch. How often did the judges reject an episode?
Thanks for the kind words, and good question: rejection is a routine part of how the farm works, not a rare event. Every rejected attempt stays in the record and counts toward the cost we report. Rejections happen at three points.
Simulated episodes. The agent writes the task expert, but it can’t certify its own rollouts. The harness re-runs each candidate episode, and a separate reviewer checks it against the task contract. In our agent comparison (appendix), getting 5 admitted episodes per task took 5–18 capture attempts, depending on the task and the agent. One concrete case: a reviewer rejected the first five beat-drum episodes because the arm moved faster than the robot’s 3 rad/s joint-speed limit, so the agent went back and repaired the expert. The same case shows why the reviewer has to be independent. One agent’s own review missed exactly this defect, and in a check we ran afterwards, 4 of its 5 “admitted” beat-drum episodes were over the limit.
Rendered video. Episodes are rendered as overlapping chunks. A VLM compares every chunk with the untextured simulator video and re-renders it if it is flagged. Across our runs this came to 1.1–1.8 renders per chunk: about 1.1 on pull drawer, 1.3 on stack cups, and 1.75 on weigh apple, where the scale’s display is easy to get wrong. A rejected render is retried from the same simulated episode, so the action labels never change to make a video pass.
Final admission of the whole episode. An automatic verifier checks the finished multi-view episode. It looks for missing or extra objects, events at the wrong time, flicker, a moving background, wrong appearance, and object layout copied from the real reference frame. An independent reviewer then compares the decoded video with the real references and the simulated state. We calibrate the verifier itself like mutation testing: we corrupt an accepted episode by deleting, duplicating or enlarging objects, shifting events, swapping views or recolouring, and every corruption must be caught before the verifier is trusted.
In short, the judges reject quite often, and they’re meant to. That’s why the cost comparison in the paper is per admitted episode, with all rejected renders included.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper