• 

  Paper

Abstract

Machine learning for robotics promises to unlock generalization to novel tasks and environments. Guided by this promise, many recent works have focused on scaling up robot data collection and developing larger, more expressive policies to achieve this. But how do we measure progress towards this goal of policy generalization in practice? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce, settings. In this work, our goal is (1) to outline the forms of generalization we believe are important in robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose ★-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. We discuss how our taxonomy encompasses most prior notions of generalization in robotics. Next, we instantiate ★-Gen with a concrete real-world benchmark based on the widely-used Bridge V2 dataset. We evaluate a variety of state-of-the-art models on this benchmark to demonstrate the utility of our taxonomy in practice. Our taxonomy of generalization can yield many interesting insights into existing models: for example, we observe that current vision-language-action models struggle with various types of semantic generalization, despite the promise of pre-training on internet-scale language datasets. We believe ★-Gen and our guidelines can improve the dissemination and evaluation of progress towards generalization in robotics, which we hope will guide model design and future data collection efforts.

Our Taxonomy: ★-Gen

Generalization in robotics can be a nebulous, ill-defined concept. We formalize generalization intuitively as perturbations relative to some base task. For visual-lingual control policies, such as vision-language-action models (VLAs), these perturbations can fall under three main types:

A given perturbation might lie at the intersection of one or more of these types. For example, changing the location of a carrot in the task "put carrot on plate" changes both the image (Visual) and the required actions (Behavioral).

Therefore, we can categorize perturbations into categories of generalization depending on which combination of the above perturbation types they affect. For example, changing the location of the carrot would fall under the Visual + Behavioral category. We further group different perturbations in each category into human-interpretable axes of generalization. Below you can find the axes we have enumerated for each category of generalization.

These axes span a wide range of possible perturbations, and we show below that ★-Gen encompasses notions of generalization in prior work:

We walk through an example of instantiating our taxonomy ★-Gen into a real-world benchmark for generalization evaluation. We use the popular Bridge-V2 dataset as a starting point, and we train several state of the art models: OpenVLA, MiniVLA, and \( \pi_0 \). We pick the following base tasks which are aligned with the Bridge-V2 dataset, and collect 20-50 additional demos per task to ensure they are in distribution for our setup.

For each of these base tasks, we chose perturbations along a subset of our axes, since some axes did not have meaningful instatiations on this dataset. Please refer to our paper for details on the specific evaluation tasks we used.

We also consider varying different model design decisions to better understand their impact on generalization.

Citation

Acknowledgements

The website template was borrowed from Jon Barron.