Nowadays with the development of embodied AI, data collection sources have been focused on two main pillars: scale (quantity, hours and episodes) and quality (reliable tasks and high-tech sensors). However, there is a complex variable that may affect both processes and that developers sometimes overlook when drawing the data architecture diagram: the human being that is completing the task during the tests. Currently, some data collection methods, such as imitation learning and behavioral cloning, rely heavily on manual teleoperation. Whether through VR headsets, exoskeletons, or data collection gloves, a human operator is physically connected to the robotic equipment while performing a task. Human-in-the-loop teleoperation has therefore become an important approach for collecting demonstrations and interventions for robotic learning systems.  [1]

This human involvement introduces a source of variability that does not exist in purely scripted or simulated data collection. Unlike a simulation script that executes predefined and highly consistent trajectories, the human body is a biological system capable of producing unexpected movements and behaviors, while also being susceptible to fatigue over time. 

During data collection, it is easy to assume that the operator will maintain expert-level performance throughout the recording process. However, even highly experienced operators can exhibit micro-tremors, hesitations, and sudden variations in their trajectories, all of which can affect the resulting data. This is what we refer to as "teleoperator noise."

 This time, in our data collection section, we will explore the nature of this noise: how biomechanics and human ergonomics affect dataset quality at a physiological and operational level, and what impact this variability has on the training of robotic models. 

In order to analyze how human biomechanics and cognition inject unwanted "noise" into the state-action space at the moment of collection, we first need to understand and categorize the different ways this happens. Human noise is rarely gaussian (white noise); instead, it exhibits physical and psychological patterns that we can classify into four categories.  [2]  

Neuromuscular Tremor & Biomechanical Fatigue

At the most basic physiological level, the human arm is never perfectly still. When the body experiences stress or fatigue during cyclic activities or after long periods of time, involuntary tremors become a persistent feature of manual control. [3][4] Knowing this, we can understand how tracking error can increase over time as the operator's arm jitters, directly affecting the reliability of the recorded trajectory. 

Fortunately, this phenomenon is already well understood in teleoperation research, and existing hardware and filtering approaches have been developed to compensate for it. One example is the wearable teleoperation system proposed by Huijin Zhu et al. in IMU Motion Capture Method with Adaptive Tremor Attenuation in Teleoperation Robot System. The authors use wearable IMU and sEMG sensors combined with a variable-gain Extended Kalman Filter (EKF) to attenuate physiological tremor while preserving the operator's intended movements. Their experiments showed that the filtering method reduced trajectory error during teleoperation, demonstrating that physiological tremor can be directly addressed at the human–robot interface.  [4]

However, biomechanical fatigue can compound this problem over the course of a session. As muscles become fatigued, involuntary corrections may become more pronounced, meaning that the "noise profile" of a single operator is not necessarily constant even within one recording episode. Instead, it can change as the session progresses, introducing an additional source of variability into the collected trajectory. 

Cognitive Hesitation & Sub-Optimal Decisions

Teleoperation requires operators to simultaneously perceive the remote environment, plan an action, and execute fine motor control, resulting in high mental loads that may affect performance. This workload has been studied extensively in human-robot interaction literature; frameworks such as the NASA Task Load Index have been widely used to measure operator workload and its effect on performance across teleoperation scenarios. [5][6] The design of the feedback channel between the robot and the operator has therefore become a critical factor in reducing mental effort, particularly in tasks that require high levels of precision.

 Using these frameworks, it can be stated that operators under heavy cognitive load will not simply become slower; they may hesitate more and unintentionally fall into sub-optimal or incorrect trajectories.  [7] At scale, prolonged cognitive load may also affect the diversity of collected demonstrations, as operators can unconsciously converge toward repetitive or stereotypical, low-effort motion patterns as a way to manage the mental demands of the task. 

Proprioceptive Drift & Ergonomic Mismatch

The third source of noise emerges from the mismatch between the operator's body geometry and the hardware used for data collection, such as VR headsets, motion-capture gloves, or haptic arms. When these systems do not conform well to the operator's anatomy, they can create ergonomic mismatches that force uncomfortable postures or unnatural movements to compensate for the hardware's limitations. These compensatory movements introduce systematic deviations into the recorded trajectories, which can be mistaken for intentional motion rather than artifacts of poor physical fit. [8][9]

Also, proprioceptive drift comes from the sensorimotor disconnection between what the operator is feeling and what they visually observe. [10] Over prolonged periods this reduces the user's spatial perception, causing a gradual shift in the baseline of the task and reducing the fluency of the recorded movement. 

Inter-Operator Heterogeneity

(Teleoperators don't have the same signature)

Finally, even when the hardware and the task remain constant, different operators may generate different movement trajectories while still successfully completing the same task. In other words, the movement does not need to be identical, because differences in body geometry, physical characteristics, and individual movement strategies can lead to different trajectories for accomplishing the same objective. This variability can be understood not necessarily as noise, but as the existence of a space of valid trajectories through which the task can be accomplished. 

Although this category is not typically framed as a standalone taxonomy item in embodied AI, it is well documented in the adjacent field of robotic surgery, where similar master teleoperation dynamics have been extensively studied across operators. Research comparing novice and experienced surgeons using the same surgical robot has found that teleoperation can amplify movement differences between skill levels more strongly than freehand movement, with experienced operators exhibiting smoother velocity trajectories and smaller errors than novices under identical hardware conditions.  [11] [12] Rather than treating these differences as purely undesirable noise, they can be understood as different ways of selecting trajectories within the space of valid solutions. The trajectory selected by an operator can be influenced by their accumulated internal model of the teleoperator's dynamics, their level of experience, dominant hand, physical build, and individual movement style. [11] [12]

This perspective is analogous to the paradigm of large language models, where the model does not necessarily produce a single predetermined valid output, but instead selects among multiple possible tokens according to their learned probabilities. Similarly, an embodied AI system can learn to select an appropriate action or trajectory from a set of valid alternatives while accounting for the variability introduced by the operator and the interaction dynamics. 

In practice, two operators can successfully perform the same task while producing noticeably different trajectories. One may move quickly with higher acceleration, while another may use slower and more cautious movements. Both trajectories may be valid, but these differences become part of the variation in the dataset. If information about the operator is not explicitly recorded, it can be difficult to distinguish this variation from other sources of variability. This is particularly relevant in reinforcement learning, where the learned policy is often treated as a black box, making it difficult to determine how specific characteristics of the data influence the resulting policy. Therefore, inter-operator heterogeneity should be considered a potential structured source of variation rather than simply random noise. 

Nurvai's closing thoughts

Mapping out this taxonomy is the first step toward taking operator noise seriously as a data quality variable, not just an inconvenient artifact. Neuromuscular tremor, cognitive hesitation, proprioceptive drift, and inter-operator heterogeneity don't all behave the same way, and they don't all belong in the same bucket as "random noise" some of them are structured, recoverable, and even informative about the space of valid trajectories. Before we can decide what to do about this variability, we first need to be precise about where it comes from. That's exactly the groundwork this taxonomy lays for the rest of the series. 

Next in this series — Part 2: Why Operator Noise Breaks Downstream Policies?

If you are interested in learning more about how human factors shape teleoperation data, exploring this taxonomy in greater depth, or require support collecting data for robotics applications, feel free to connect with us: Free Consultation with Nurvai

Connect with us on socials
LinkedIn | X

Picture from Dexrobot-DexCap equipment 

References

[1]Learning effects in variable autonomy human-robot systems: How much training is enough? arXiv.arXiv - Learning effects in variable autonomy human-robot systems

[2] Noise in Cognition: Bug or Feature?. PMC.Noise in Cognition: Bug or Feature?

[3] Szymon Kuliś, Physiological limb tremor modulation by physical exertion: a systematic review and meta-analysis in healthy adults. British Medical Bulletin.Physiological limb tremor modulation by physical exertion: a systematic review and meta-analysis in healthy adults

[4] IMU motion capture method with adaptive tremor attenuation in teleoperation robot system. PMC.PMC - IMU motion capture with adaptive tremor attenuation

[5] Steinfeld, A., et al. (2006). Common metrics for human-robot interaction. Robotics Institute, Carnegie Mellon University.CMU - Common metrics for human-robot interaction

[6]Sandra G. Hart, Lowell E Staveland. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, ScienceDirect. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research

[7] Design of multi-modal feedback channel of human–robot cognitive interface for teleoperation in manufacturing. Journal of Intelligent Manufacturing (2024).Springer Nature - Journal of Intelligent Manufacturing article

[8] NuExo: A wearable exoskeleton covering all upper limb ROM for outdoor data collection and teleoperation of humanoid robots. arXiv (2025).arXiv - NuExo

[9] An upper-limb teleoperation exoskeleton with stepless arm-length parameterization and adaptive force-triggered impedance blending. Scientific Reports.Nature - Upper-limb teleoperation exoskeleton

[10] Human-robot interaction / teleoperation research. Frontiers in Robotics and AI (2020).Frontiers - Toward Enhanced Teleoperation Through Embodiment

[11] Ben-Israel, D., et al. Effects of robotic manipulators on movements of novices and surgeons. Surgical Endoscopy.Springer Nature - Effects of robotic manipulators on movements of novices and surgeons

[12] Draper, J. V. (2021). Human factors in telemanipulation: Perspectives from the Oak Ridge National Laboratory experience. ResearchGate.ResearchGate - Human factors in telemanipulation