Garbage in, garbage out. The quality of your dataset will impact the performance of your model.
Here are some best practices for recording a good dataset for imitation learning in robotics,
based on what we learned from training our models
Keep the robot’s operating area free of unnecessary changes. Avoid placing the robot in spaces with moving objects (e.g., people walking by, machinery operating) or shifting backgrounds, as these introduce noise that can confuse the model.
Ensure even, stable lighting across all recordings. Shadows can obscure objects, so use tools like ring lights or diffused lamps to maintain uniformity. Avoid natural light from windows, which varies with time and weather, and keep lighting consistent for each object throughout the dataset.
Match the camera arrangement used in the model’s pretraining phase. For example, if using a model like pi0 by Physical Intelligence, include wrist cameras on each robot arm and a first-person view (FPV) context camera for a broad perspective.
Secure cameras to prevent movement during recording. Test the setup by asking: “Could I control the robot effectively using only these camera views?” If the answer is no, adjust until the robot and target are fully visible.
Plan the robot’s trajectory so the target object enters the camera’s field of view—especially the wrist cameras—as early as possible. For example, avoid occluding the object with the gripper during the approach; instead, angle the arm to keep the target in sight.
Use a repeatable grasping strategy to help the model learn reliable patterns, while allowing slight variations for robustness (see Dataset Diversity below).
Think of diversity as defining the “space” where the model can operate. If objects are always on the left, the model won’t learn to pick them up from the right. Test the limits of your task to ensure coverage.
Outliers are examples that deviate significantly from the norm. They can mislead the model, causing it to learn incorrect patterns.
Try to keep the examples close to the norm.
The quantity of data matters as much as its quality.Aim for 40-50 episodes per task as a starting point. An “episode” is a complete execution of the task, from start to finish (e.g., picking up an object and placing it elsewhere).
Robotics datasets take a long time to collect and are hard to edit.
Ensure that the time you invest in collecting demonstrations is worthwhile.We recommend first recording a few episodes and then checking that the data is good.Use the Visualize Dataset space from LeRobot to verify that the data is correct and can be loaded properly.Do a full cycle of data collection, training, and test inference to check that the full pipeline works.
Then scale the process.Once a model is trained, check it learns to at least replay an episode.
Put it in the same state as the dataset and check that it can execute the task.