Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization

Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization

Zhengyu Hu¹    Zheyuan Xiao¹    Yike Wang^1,2    Hamish Ivison¹    Haoyu Wang³    Yige Yuan¹    Rulin Shao¹    Yuetai Li¹    Niqi Lyu⁴    Yicheng Ding^4*    Shu Yang⁵    Zhichen Zeng^1,6    Banghua Zhu^1,6    Jiaxin Pei^5,7    Kaize Ding⁸    Pradeep Dasigi²    Dan Roth³    Hannaneh Hajishirzi¹    Noah A. Smith^1,2    Teng Xiao^1,2*†    Radha Poovendran^1*

¹University of Washington   ²Allen Institute for AI   ³University of Pennsylvania   ⁴Pyromind Dynamics   ⁵Stanford University   ⁶RadixArk   ⁷The University of Texas at Austin   ⁸Northwestern University
^†Project leader   ^*Corresponding authors

Can frontier agents keep improving real AI infrastructure for 24 hours? Infrathon gives an agent an H100 workstation, a real ML system and a 12–24 hour budget: 25 tasks from GPU operators to full training and serving systems, scored by an independent Judge on held-out workloads. We evaluate 6 models over 150 runs and 10,000+ GPU-hours; 135 of the 150 runs still improve in the second half of their budget.

🏠 Homepage · 📝 Blog · 💻 Code · 🛠️ Infrathon-build · 🤗 Dataset · 🐳 Docker Hub

Mean rank scores range from 0 to 100; 100 means first place on every task.

Mean values: human codec → Claude's codec. Quality metrics use the uncompressed BF16 reference.

Geometric means of task–model score ratios; 100% means equal performance at equal GPU-hours.

Everything is open: the tasks, the Work/Controller/Judge images, the held-out data, and 🛠️ Infrathon-build for turning an expert's task definition into a new Infrathon task.

📄 If you find Infrathon useful, please cite:

@inproceedings{Hu2026InfrathonBL,
  title={Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization},
  author={Zhengyu Hu and Zhe-Yuan Xiao and Yike Wang and Hamish Ivison and Hao-Yu Wang and Yi-Ge Yuan and Ru-Lin Shao and Yue-Tai Li and Niqi Lyu and Yi-Cheng Ding and Shu Yang and Zhi-Chen Zeng and Bang-Hua Zhu and Jia-Xin Pei and Kai-Ze Ding and Pradeep Dasigi and Dan Roth and Hanna Hajishirzi and Noah A. Smith and Teng Xiao and Radha Poovendran},
  year={2026},
  url={https://api.semanticscholar.org/CorpusID:295731967}
}