Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, Jonathan Tremblay

Researchers have introduced RoboLab, a simulation benchmarking framework designed to test the true generalization capabilities of robotic foundation models. The framework addresses a critical gap in robotics evaluation: existing benchmarks suffer from domain overlap between training and evaluation data, inflating success rates and masking real robustness limitations. RoboLab includes 120 tasks across three competency axes (visual, procedural, relational) and three difficulty levels, plus systematic analysis tools that measure how policies respond to controlled perturbations. Early evaluation reveals significant performance gaps in current state-of-the-art models when tested on genuinely novel scenarios.