Data is one of the core components of the embodied AI pipeline. It is tightly connected to both the design of the models and the robots those models ultimately control. While the amount of data a model ingests is important, an arguably even more important factor is the quality of that data.
A common saying in machine learning is “trash in, trash out,” and that definitely holds true in embodied AI. Poor quality data can lead to weak policies, inconsistent behavior, and systems that struggle to generalize outside of narrow environments no matter how large the dataset is [1].
In this post, we will go through some of the main things to look for when evaluating robotic training data and discuss how different dataset characteristics can affect downstream performance. The goal is not to create a universal definition of “good” data, but rather to provide a practical framework for thinking about how datasets should be structured depending on the type of robotic system being trained.
Why Should You Care
Evaluating robotic datasets is not always straightforward. Unlike traditional machine learning, there is no single metric that can tell you whether a dataset is "good" or "bad." The qualities that make a dataset useful often depend on how it will be used, the robot being trained, and the environment in which that robot will operate.
As a result, selecting data is often less about finding the largest dataset available and more about understanding whether that dataset contains the information needed to learn the desired behaviors. A dataset that works well for pre-training may be poorly suited for fine-tuning, while a highly specialized dataset may provide little value for learning general representations.
Before discussing what makes a dataset high quality, it is therefore important to first understand the role that dataset will play within the broader embodied AI pipeline.
Before You Select a Dataset
Before you even begin collecting data, you need to consider a few aspects of your system’s design. The first and arguably most important question is: What part of the pipeline will this dataset be used for?
The answer will largely determine the type of data you should be looking for. For example, if your goal is VLA pretraining, egocentric or UMI-style datasets may be a good fit since they provide large amounts of diverse interaction data from which models can learn broad priors about the world [2], [3]. If your goal is fine-tuning, however, teleoperation demonstrations may be more valuable because they more closely resemble the behavior you ultimately want the robot to execute [4].
This is also the stage where you should think carefully about the tasks represented in the dataset. During pretraining, relatively simple tasks can still be extremely useful because they help establish strong behavioral and visual priors [5]. For fine-tuning, on the other hand, it is often beneficial to select demonstrations that closely resemble the target behaviors you want the robot to perform [6].
While this may sound obvious, it is one of the most important steps in data evaluation. A dataset cannot be considered good or bad in isolation. Its quality is largely determined by how well it aligns with the role it is expected to play within the training pipeline and the capabilities you ultimately want the robot to acquire.
General Quality
While the rest of this post focuses on what quality means for different parts of the embodied AI pipeline, there are a few general characteristics that should be evaluated regardless of where the data will ultimately be used.
Accuracy
One of the first things to evaluate is the accuracy of the data. The information collected by your sensors should represent physical reality as closely as possible. If measurements consistently differ from the real world, the model will learn from an inaccurate representation of the environment, which can negatively impact performance during deployment [7].
If you are collecting your own data, an important first step is understanding how accurate your setup is. This means testing your sensors, identifying sources of systematic error, and verifying that the recorded measurements match real-world conditions.
Evaluating accuracy becomes more difficult when working with third-party datasets. Some providers explicitly report sensor specifications and measurement errors, while others provide little information about the collection process. Whenever possible, it is worth reviewing the collection methodology or consulting the supplier directly to understand how closely the recorded data reflects reality.
It is also important to remember that the required level of accuracy depends heavily on the intended application. A robot performing surgical procedures may need positional accuracy within a few millimeters, while a warehouse robot moving packages can often tolerate errors of several centimeters without significantly affecting performance.
For this reason, accuracy should always be evaluated relative to the task at hand. The goal is not necessarily to obtain perfect measurements, but to ensure that the data is accurate enough for the behavior the robot is expected to learn.
Precision
Another important aspect of data quality is precision. Precision refers to the consistency of measurements when observing the same state or event multiple times. In other words, if the underlying conditions remain unchanged, the recorded values should remain relatively consistent as well.
Low precision often manifests as noisy sensor readings, inconsistent position estimates, or measurements that fluctuate significantly despite little or no change in the environment. This additional variability can make learning more difficult by introducing uncertainty that is unrelated to the task itself [8].
While some amount of noise is unavoidable in real-world robotics, highly precise data helps models learn meaningful patterns rather than artifacts introduced by the sensing process.
Similarly to accuracy, there is no universal threshold for precision. Different applications place different demands on the data. A robotic arm performing precision assembly may require significantly more consistent measurements than a mobile robot navigating a warehouse, even if both systems are considered to have acceptable data quality for their respective tasks.
Temporal Alignment
Data also needs to be temporally aligned. By this, we mean that observations from different sensors should correspond to the same moment in time. If sensor streams are not properly synchronized, the model may end up learning relationships between observations that never actually occurred together [1].
This can be particularly challenging because different sensors often operate at different sampling rates. For example, an IMU may record measurements at 100 Hz while a camera captures images at 30 frames per second. Without proper synchronization, it becomes difficult to determine which sensor readings correspond to a particular image or action.
To address this, robotic systems typically rely on timestamp synchronization, interpolation, resampling, or downsampling techniques to align sensor streams onto a common timeline. The exact approach depends on the application and the sensors involved, but the objective remains the same: ensuring that all observations accurately represent the same state of the world.
Poor temporal alignment can introduce subtle errors into a dataset that are difficult to detect during collection but can significantly impact downstream learning and control performance.
Completeness
Completeness refers to whether the dataset contains all the information that was intended to be collected. In practice, this means ensuring that sensor streams do not contain large gaps, missing segments, or prolonged periods where data was not recorded correctly.
This is particularly important in robotics and other physical systems, where communication issues, hardware failures, network instability, or changing environmental conditions can result in information being lost during collection. Missing observations may prevent a model from fully understanding what occurred during a task and can make certain trajectories difficult or impossible to use for training [4].
In most real-world applications, some degree of data loss is unavoidable and often acceptable. The key is understanding whether the missing information is infrequent and random or whether it represents a systematic issue within the data collection pipeline. Large gaps in sensor streams, repeated sensor dropouts, or consistently missing modalities may indicate underlying problems that should be addressed before the dataset is used for training.
When evaluating completeness, it is important to consider not only how much information is missing, but also which information is missing. Losing a few frames from a camera stream may have little impact in some applications, while missing a critical segment during a grasping action or sensor failure event could significantly reduce the value of the demonstration.
Bias
Another important aspect of data quality is bias. While bias is often discussed in the context of machine learning models, many biases originate in the data itself [3].
In robotics, bias occurs when certain environments, objects, tasks, operators, or interaction patterns are overrepresented relative to the situations the robot will encounter during deployment. As a result, the model may develop assumptions that hold within the dataset but fail when exposed to new conditions.
Bias can appear in many forms. A dataset may contain mostly household environments while the target deployment is industrial. Certain object categories may appear far more frequently than others. Demonstrations may be collected primarily by a small group of operators who all perform tasks in similar ways. Even the success and failure distributions of a dataset can introduce bias if one outcome is significantly overrepresented [9].
These biases are not always harmful. In fact, some degree of bias is often intentional, particularly during fine-tuning where the goal is to adapt a model to a specific deployment environment. Problems arise when the distribution represented in the dataset differs significantly from the distribution the robot will encounter in practice.
When evaluating a dataset, it is therefore worth asking what experiences are overrepresented and, just as importantly, what experiences may be missing entirely. Understanding the limits of a dataset’s coverage can often be as valuable as understanding its strengths.
The objective is not to eliminate bias completely, which is rarely possible, but to ensure that any biases present are aligned with the intended use case rather than accidental artifacts of the data collection process.
Getting these fundamentals right will not magically solve every data quality problem, but it does ensure that the information being fed into the model reflects reality as closely as possible. Before worrying about annotations, task distributions, or dataset scale, it is worth making sure the data itself is accurate, precise, temporally aligned, and complete.
Pre-Training Quality
Pre-training and fine-tuning serve very different purposes, which means they require different types of data. During pre-training, the objective is not to teach a robot how to perform a particular task. Instead, the goal is to expose the model to enough of the world that it can develop useful priors about objects, environments, and physical interactions [10], [11].
This changes how we think about data quality. During pre-training, diversity is often more valuable than specialization. Rather than asking whether a dataset contains the exact behavior we care about, we should ask whether it exposes the model to a broad enough range of experiences to support future learning.
The following are some of the most important qualities to look for when evaluating a dataset intended for pre-training.
Diversity
Diversity is one of the most important characteristics of high-quality pre-training data. The goal is not simply to provide more demonstrations, but to expose the model to a broader range of objects, environments, tasks, and interaction patterns from which it can learn useful representations [12], [13].
Diversity should be considered across multiple dimensions. A dataset may contain thousands of demonstrations but still lack meaningful variety if those demonstrations take place in similar environments or involve the same types of objects. High-quality pre-training data should ideally include variation in environments, viewpoints, lighting conditions, object categories, interaction types, and even robot embodiments [3].
By broadening the range of experiences available during training, diverse datasets help models develop physical and semantic priors that transfer more effectively to unseen situations.
Manipulation Richness
Pre-training data should also contain a rich variety of manipulation behaviors. While diversity focuses on exposing the model to different environments, objects, and tasks, manipulation richness focuses on the complexity and variety of the interactions themselves [1].
A high-quality pre-training dataset should go beyond simple pick-and-place demonstrations and include interactions with articulated, deformable, and dynamic objects. Tasks involving doors, drawers, cloth, cables, containers, tools, and other challenging objects expose models to a wider range of physical phenomena and interaction patterns [13].
Similarly, behaviors such as grasping, pushing, pulling, rotating, stacking, inserting, pouring, and bimanual manipulation help build a more comprehensive understanding of how actions affect the world. The more varied these interactions are, the more opportunities the model has to learn representations that transfer to downstream tasks.
Coverage of the Physical World
While manipulation richness focuses on the variety of actions being performed, physical world coverage focuses on the variety of physical phenomena those actions expose the model to.
Many robotic tasks depend on understanding concepts such as contact, friction, gravity, object permanence, occlusion, and collision dynamics. A model trained exclusively on simple and highly controlled demonstrations may struggle when faced with situations where these factors become important [7].
For this reason, high-quality pre-training datasets should include examples of objects slipping, falling, colliding, becoming partially occluded, or behaving in unexpected ways. Interactions involving unstable objects, cluttered environments, moving targets, and changing environmental conditions can provide valuable information about the dynamics of the real world.
Failure cases can be particularly useful. While successful demonstrations are important, observing failed grasps, dropped objects, and recovery behaviors helps expose the model to situations that robots frequently encounter during deployment [4].
A dataset with strong physical world coverage helps ensure that the model develops an understanding of how the world behaves, not just how tasks are typically performed.
While every robotic application will have its own requirements, these characteristics provide a useful framework for evaluating pre-training data quality. A strong pre-training dataset should expose models to diverse environments, rich manipulation behaviors, a broad range of physical interactions, and sufficiently varied language supervision [3], [5].
None of these factors exist in isolation. A massive dataset with poor diversity may provide less value than a smaller but more varied one [12]. Similarly, rich demonstrations can lose much of their usefulness if the accompanying annotations fail to capture the semantics of the task. Ultimately, the goal of pre-training is not to teach a specific behavior, but to build representations that can support many behaviors in the future.
When evaluating a dataset, it is therefore worth looking beyond raw scale and asking a more important question: does this data expose the model to enough of the world for it to learn something useful about how that world works?
Fine-Tuning Datasets
Data used for fine-tuning is typically much more specific than data used for pre-training. While pre-training aims to expose a model to a broad range of experiences, fine-tuning focuses on adapting that model to a particular environment, task, robot, or deployment scenario [6].
Because of this, fine-tuning datasets benefit most from alignment with the conditions under which the model is expected to operate. The closer the training data is to the target environment, users, and tasks, the less the model has to rely on generalization and the more it can focus on learning the behaviors that actually matter.
When evaluating a fine-tuning dataset, the key question is not how much of the world it covers, but how well it represents the specific situations the robot will encounter after deployment. A smaller dataset collected in the target environment may often be more valuable than a much larger dataset collected under different conditions [4].
This shift in objectives also changes how we evaluate data quality. Instead of prioritizing broad coverage and diversity, fine-tuning datasets should emphasize relevance, consistency, and alignment with the intended use case.
Environment Alignment
Unlike pre-training, where diversity is often one of the primary objectives, fine-tuning benefits more from environment alignment [13]. At this stage, the goal is no longer to expose the model to as many situations as possible, but rather to adapt its behavior to the conditions in which it will actually operate.
Observing tasks performed in environments that closely resemble the deployment setting helps the model learn the layouts, object arrangements, and spatial constraints it is likely to encounter in practice. This can be particularly important in robotics, where even small differences in workspace organization, camera placement, lighting conditions, or object positioning can affect performance [7].
Environment alignment also allows the model to learn behaviors that are specific to a particular setup. A robot operating in a warehouse may face very different constraints from one working in a home or laboratory, and even small differences in layout, lighting, or object placement can affect performance.
For this reason, when evaluating fine-tuning data, it is often more valuable to ask how closely the dataset matches the target deployment environment than how many different environments it contains. While some diversity can still help improve robustness, relevance to the intended operating conditions is usually the more important factor.
Task Alignment
Task alignment is one of the most important aspects of fine-tuning data quality. While pre-training focuses on teaching broad priors about the world, fine-tuning is intended to refine a model’s behavior toward a specific set of objectives. As a result, the demonstrations used during this stage should closely match the tasks the robot is ultimately expected to perform [6].
This may sound obvious, but it is a surprisingly important consideration when evaluating datasets. A model trained to organize shelves is unlikely to benefit as much from demonstrations of table cleaning as it would from additional examples of shelf organization, even if both tasks involve similar manipulation skills.
At the same time, task alignment should not be confused with task uniformity. While demonstrations should focus on the target task, they should still contain diversity in how that task is accomplished. Objects may appear in different locations, environments may vary slightly, and operators may use different strategies to achieve the same objective.
The goal of fine-tuning is not to have the model memorize individual demonstrations, but to learn the underlying behavior that makes the task successful [9]. A high-quality fine-tuning dataset therefore balances relevance with variability, exposing the model to different ways of solving the same problem while remaining focused on the behavior that is ultimately desired.
Demonstration Quality
The quality of the demonstrations themselves is another critical factor during fine-tuning. Unlike pre-training, where broad exposure is often the primary objective, fine-tuning relies heavily on the model learning from examples of the behavior it is expected to reproduce [4].
For this reason, demonstrations should be consistent, purposeful, and representative of successful task execution. Large pauses, unnecessary movements, operator hesitation, or erratic behavior can introduce noise into the dataset and make it more difficult for the model to identify the patterns that actually contribute to task completion.
Demonstrations should also be collected with safety in mind. The objective is to provide examples of the behaviors that should be replicated during deployment, not accidental actions or unsafe operating practices.
Perhaps most importantly, operators should remember that they are not simply completing a task, but teaching a model how to complete it. High-quality demonstrations therefore focus on producing trajectories that are clear, repeatable, and aligned with the desired outcome while still exposing the model to enough variation to generalize beyond the specific examples it has seen [8].
Annotation Precision
Annotations for fine-tuning data should be rich, temporally aligned, and consistent throughout the dataset. Their purpose is not simply to describe what happened during a trajectory, but to create the language space through which the robot will understand and execute tasks during deployment [14].
This means annotations should accurately capture tasks, subtasks, object interactions, and state transitions as they occur. Semantic annotations, instruction sequences, success conditions, and dense trajectory descriptions can all provide valuable supervision signals when they are properly aligned with the robot’s actions [10].
Precision is especially important during fine-tuning because the model is no longer learning broad representations of the world. Instead, it is refining behaviors that will later be deployed in real environments. Ambiguous instructions, inconsistent terminology, or poorly aligned annotations can introduce confusion into the learning process and ultimately degrade downstream performance.
When evaluating a fine-tuning dataset, it is worth asking whether the annotations describe the behavior with enough detail and consistency for the robot to reliably connect language, perception, and action. In many cases, a smaller dataset with highly precise annotations may be more valuable than a much larger dataset with noisy or inconsistent supervision. [4]
Failure and Edge Cases
Data should also contain failure cases and edge cases. These are the situations where manipulation does not go as planned, sensors produce unexpected observations, objects behave differently than anticipated, or the environment introduces challenges that are uncommon but still realistic [1].
Many datasets are heavily biased toward successful demonstrations because they are easier to collect and evaluate. While successful trajectories are important, they only show the model what correct behavior looks like. Failure cases, on the other hand, provide information about the limits of the robot’s capabilities and expose it to situations it is likely to encounter during deployment [4].
Examples may include failed grasps, dropped objects, objects slipping during manipulation, partial occlusions, cluttered environments, or tasks that cannot be completed due to environmental constraints. These scenarios help the model learn more robust representations and reduce the risk of brittle behavior when operating outside ideal conditions.
Edge cases are equally important. Real-world environments are rarely as clean and predictable as curated datasets. Objects may appear in unusual orientations, lighting conditions may change, or users may interact with the robot in unexpected ways. Exposure to these situations during training can significantly improve a model’s ability to generalize and recover when things do not go according to plan [15].
A useful rule of thumb is that if a failure mode is likely to occur during deployment, the model should ideally have encountered some version of it during training. The goal is not to eliminate failure entirely, but to ensure that failures are familiar rather than completely novel.
Taken together, these characteristics provide a useful framework for evaluating fine-tuning data quality. Unlike pre-training datasets, which are primarily concerned with breadth and exposure, fine-tuning datasets are ultimately judged by how well they prepare a model for a specific deployment scenario.
A high-quality fine-tuning dataset should closely reflect the environment, tasks, and conditions the robot will encounter in practice. It should contain demonstrations that are representative of the desired behavior, annotations that accurately describe what is occurring throughout the trajectory, and enough variation to ensure the model learns robust skills rather than memorizing individual examples.
Most importantly, fine-tuning data should not only teach a robot how to succeed when everything goes according to plan, but also how to respond when it does not. Failure cases, edge cases, and realistic sources of variability are often where deployed systems are truly tested.
When evaluating a fine-tuning dataset, the question is therefore not simply whether the data is good, but whether it is good for the task, environment, and deployment scenario you care about. The closer the dataset aligns with those requirements, the more effective the fine-tuning process is likely to be.
Closing Thoughts
Data quality is not a fixed concept. What makes a dataset valuable depends heavily on how that data will be used and the objectives of the system being trained. A dataset that is excellent for pre-training may be poorly suited for fine-tuning, just as a highly specialized fine-tuning dataset may provide little value when the goal is to learn broad representations of the world [1].
The most important takeaway is that dataset quality cannot be reduced to a single metric. Scale, diversity, annotation quality, environment alignment, demonstration quality, and failure coverage all matter, but their relative importance changes depending on the application [3]. Evaluating data therefore requires understanding not only the dataset itself, but also the problem it is intended to solve.
As embodied AI continues to advance, the conversation around data is increasingly shifting away from “how much data do we have?” and toward “what kind of data do we need?” [10]. Answering that question correctly is often the difference between a model that performs well on a benchmark and one that performs reliably in the real world.
References
[1] O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” Journal of Machine Learning Research, vol. 22, no. 30, pp. 1–82, 2021, Available: https://arxiv.org/abs/1907.03146
[2] C. Chi et al., “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” in Proceedings of robotics: Science and systems (RSS), 2024. Available: https://arxiv.org/abs/2402.10329
[3] Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic learning datasets and RT-X models,” in Proceedings of the IEEE international conference on robotics and automation (ICRA), 2024. Available: https://arxiv.org/abs/2310.08864
[4] A. Mandlekar et al., “What matters in learning from offline human demonstrations for robot manipulation,” in Conference on robot learning (CoRL), 2021. Available: https://arxiv.org/abs/2108.03298
[5] A. Brohan et al., “RT-1: Robotics transformer for real-world control at scale,” in Proceedings of robotics: Science and systems (RSS), 2023. Available: https://arxiv.org/abs/2212.06817
[6] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proceedings of robotics: Science and systems (RSS), 2023. Available: https://arxiv.org/abs/2304.13705
[7] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in IEEE international conference on robotics and automation (ICRA), 2018. Available: https://arxiv.org/abs/1710.06537
[8] P. Florence et al., “Implicit behavioral cloning,” in Conference on robot learning (CoRL), 2021. Available: https://arxiv.org/abs/2109.00137
[9] J. Ho and S. Ermon, “Generative adversarial imitation learning,” Advances in Neural Information Processing Systems (NeurIPS), vol. 29, 2016, Available: https://arxiv.org/abs/1606.03476
[10] B. Zitkovich et al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” in Proceedings of the 7th conference on robot learning, in Proceedings of machine learning research, vol. 229. 2023, pp. 2165–2183. Available: https://arxiv.org/abs/2307.15818
[11] S. Reed et al., “A generalist agent,” in Transactions on machine learning research, 2022. Available: https://arxiv.org/abs/2205.06175
[12] S. Dasari et al., “RoboNet: Large-scale multi-robot learning,” in Conference on robot learning (CoRL), 2019. Available: https://arxiv.org/abs/1910.11215
[13] F. Ebert et al., “Bridge Data: Boosting generalization of robotic skills with cross-domain datasets,” in Conference on robot learning (CoRL), 2022. Available: https://arxiv.org/abs/2109.13396
[14] C. Lynch et al., “Interactive language: Talking to robots in real time,” IEEE Robotics and Automation Letters, vol. 8, no. 6, pp. 3980–3987, 2023, doi: 10.1109/LRA.2023.3295255.
[15] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” in AAAI conference on artificial intelligence, 2008. Available: https://cdn.aaai.org/AAAI/2008/AAAI08-227.pdf
