Home

Benchmarks

Simulation benchmarks for embodied AI evaluation

Tasks
Manip Manipulation   Nav Navigation   WBC Whole-Body Control
Observation modalities
RGB Color camera   RGB-D RGB + Depth   S Semantic segmentation   PC Point cloud   Pose Joint/end-effector pose
Benchmark Task Scenes / Objects Observation Physics Built Upon Embodiments Description
robosuite Manip — / 10 RGB-DS MuJoCo —
Fixed Base
Panda PandaDexRH PandaDexLH Sawyer IIWA Jaco UR5e Baxter Kinova3 XArm7
Wheeled
PandaOmron Tiago
Legged
SpotWithArm SpotWithArmFloating GR1 GR1FixedLowerBody GR1ArmsOnly GR1FloatingBody
Modular framework, 11 tasks
RoboCasa Manip 120 / 2.5K RGB MuJoCo robosuite Franka Panda (mobile) 100 kitchen tasks, photorealistic
LIBERO Manip — / — RGB MuJoCo robosuite Franka Panda 130 tasks in 4 task suites
Meta-World Manip 1 / 80 Pose MuJoCo — Sawyer 50 Manip tasks for Meta-RL
LeVERB-Bench Nav WBC 4 / — RGB PhysX (Isaac Sim) Isaac Sim Unitree H1 Humanoid control
ManiSkill Manip — / 162 RGB-DPCS PhysX SAPIEN Franka Panda XArm 4 tasks, 36K demos
ManiSkill 2 Manip — / 2.1K RGB-DPC PhysX ManiSkill Franka Panda XArm Extended task diversity
ManiSkill 3 Nav Manip WBC — / — RGB-DPCS PhysX ManiSkill 2 Franka Panda Unitree H1 Fetch GPU-parallelized simulation
ManiSkill-HAB Manip 105 / 92 RGB-D PhysX ManiSkill 3, Habitat 2.0 Fetch HAB tasks from Habitat 2.0
RoboTwin Manip — / 731 RGB-D PhysX SAPIEN AgileX AgiBot G1 (dual-arm) Dual-arm tasks
Ravens Manip — / — RGB-D PyBullet — UR5 10 tabletop tasks
VIMA-BENCH Manip — / 29 RGBS PyBullet Ravens UR5 17 multimodal prompt tasks
LoHoRavens Manip 1 / 3 RGB-D PyBullet Ravens UR5 Long-horizon planning
CALVIN Manip 4 / 7 RGB-D PyBullet — Franka Panda Long-horizon lang-cond tasks
Habitat Nav 185 / — RGB-DS Bullet — LoCoBot Fast, Nav only
Habitat 2.0 Nav Manip 105 / 92 RGB-D Bullet Habitat Fetch Spot Mobile manipulation (HAB)
Habitat 3.0 Nav Manip 211 / 18K RGB-D Bullet Habitat 2.0 Fetch Spot Human avatars support
RLBench Manip 1 / 28 RGB-DS PyBullet V-REP Franka Panda Tiered task difficulty
THE COLOSSEUM Manip 1 / 107 RGB-D PyBullet RLBench Franka Panda 20 tasks, 14 env variations
AI2-THOR Nav Manip — / 118 RGB-DS Unity — LoCoBot Object states, task planning
CHORES Nav 191K / 40K RGB Unity AI2-THOR LoCoBot Shortest-path planning
SIMPLER Manip 4 / 17 RGB PhysX SAPIEN, Isaac Sim Google Everyday Robot WidowX 250 Real-to-sim evaluation
RoboArena Manip — / — RGB Real — Real-world (unspecified) Distributed real-world evaluation