Robotics Device Control and Haptic Feedback

Challenges in Robotic Device Control and the React EMG Project

• The Problem of Intent in Device Control: In robotics, particularly prosthetics and exoskeletons, the goal is to ensure movements—such as opening or closing a robotic hand while holding a cup—occur only when the user specifically intends them to. Control is not merely about a single prediction but about accurately predicting transitions between states.
• Predicting Transitions: Accuracy must be attuned to the specific moment a user decides to change their intent. It is insufficient to predict a state once; the system must capture the precise switch from one action to another.
• The React EMG Project: This project, titled "React EMG" (Reacting to EMG for intent changes), focuses on this temporal accuracy of capturing shifting ideas through Electromyography ($EMG$) signals.

Machine Learning Frameworks for Intent Prediction

• Supervised Learning Basics: Current supervised learning for robotics involves collecting $EMG$ signals paired with a "supervised intent" (the ground truth). For example, a user is instructed to "try to open" or "try to close," creating labeled data for the model.
• Masked Modeling Prediction:
  • The Framework: A masked modeling approach is utilized to help the robot pay attention to transitions and maintain signal consistency despite noise.
  • Intent as Tokens: Intents are treated as tokens within embeddings.
  • Transformer Architecture: A transformer is used to learn these patterns. By masking some of the data (similar to "filling in the blank"), the model learns to predict what should happen when data is missing, particularly during the transition from "relaxed" to "open" or "open" to "close."
• Demonstration Methodology:
  • Healthy Subject Setup: Subjects wear an $EMG$ armband.
  • Mannequin Interaction: The robot is mounted on a spring-loaded mannequin. This allows researchers to distinguish between the human participant's physical movement and the robot's reactive movement.
  • Robustness Goals: The model must be robust enough to ignore "background" signals, such as wiggling the arm, and only respond when the user activates the specific muscles required for grasping.

Intent Ambiguity and Stroke Participant Research

• The Challenge of Impairment: For participants who have suffered a stroke, the ability to perform or even know how to perform a movement may be lost, making traditional data collection difficult.
• The "Ear-Wiggling" Metaphor: If a person who does not know how to wiggle their ears is asked to do so, they may scrunch their face in various, inconsistent ways. This mirrors the struggle of stroke participants trying to generate motor signals.
• Inconsistency in Ground Truth: Machine learning assumes humans generate consistent ground truths for data labeling. However, in impaired populations, the signal often crosses "conservation lines," leading to intent ambiguity where a model cannot make clear classifications.
• Reciprocal Learning and Augmented Feedback:
  • Augmented Feedback: This adds an extra layer of information to the human's perception loop. While the human can see if the robot opened or closed, the system provides feedback on the robot's internal confidence.
  • LED Confidence Display: A visual display uses two bars: a green bar (probability/confidence of "open") and a red bar (probability/confidence of "close").
  • The Goal of Training: At the start, confidence might be high for both if the robot is confused. Training aims to achieve high confidence for the intended intent and very low confidence for the incorrect intent.
• Reciprocal Learning Process:
  • Human Phase: The user explores muscle activations while watching the LED bars, attempting to maximize the correct color (green for open, red for close).
  • Robot Phase: The robot updates its classifier based on the user's best attempts.
  • Outcomes: After multiple rounds (e.g., two rounds of learning), the "separability" of the intent improved. In high-dimensional space, the data points for "open," "close," and "relaxed" formed distinct, separable "blobs," significantly aiding classification.

Haptic Research and Proprioception

• Proprioception Defined: Proprioception is the body's internal sense of its position and movement in space.
• The PSO (PIEZO) Ion Channel: This biological mechanism is responsible for proprioception. All cells contain these strain sensors that detect when cells are stretching or being compressed.
  • PIEZO2 Function: These channels inform the stomach when it is full and the lungs when they are expanding. Specifically, $PIEZO2$ channels are critical for sensing light touch and vibrations.
• Clinical Implications of PIEZO2 Loss: Individuals born with a loss-of-function mutation in their $PIEZO2$ channels suffer from severe proprioceptive deficits.
  • The Blindfold Test: A study with the National Institutes of Health ($NIH$) showed that a person without this channel can only walk by visually monitoring the floor. When blindfolded, they cannot touch their nose or objects reliably because their movements operate in an "open loop." They do not know where their limbs are until they collide with something.

Sensory Substitution and Haptic Actuators

• Sensory Substitution: The concept of using one functional sense to compensate for a lost sense.
  • Ian Waterman Case Study: A famous example from a 1997 documentary; Waterman lost both touch and proprioceptive senses and retrained himself to move entirely by watching his own body.
  • Everyday Example: Video subtitles use vision to substitute for hearing.
• Textile Pneumatic Pouch Actuators:
  • Design: Inflatable fabric pouches that can be made in various sizes, from large modules to tiny fingertip caps.
  • Leader-Follower Systems: A sensor on one part of the body can trigger the inflation of a pouch on another part to provide tactile cues.
  • ACL Injury Application: Researchers are exploring haptic feedback to prevent Anterior Cruciate Ligament ($ACL$) injuries. High-speed inflation of pouches could alert athletes of risky leg orientations during sprinting or side-step cutting maneuvers, retraining their biomechanics.

Learning from Human Behavior and Tactile Context

• The Vision Dominance Problem: Most robots currently rely on vision alone. However, humans use distributed sensing (vision plus touch).
• The Bag Identification Example: During a Vogue "What's in my bag" interview, Ariana Grande was observed looking into her bag every time she pulled an item out. While she looked, humans are biologically capable of identifying familiar objects in a bag purely through touch.
• Haptic Deprivation and Adaptation: Human performance degrades when touch is removed (e.g., numbing/anesthetizing a finger makes striking a match or tying shoes in the cold difficult), but humans eventually adapt. Robots often lack this tactile "context," making them less capable than humans in unstructured tasks.
• Shared Pressure Teleoperation:
  • Experimental Setup: A user wears a HoloLens for hand tracking and a haptic glove with inflatable pouches. They teleoperate a robot to pick and place a deformable "hacky sack."
  • Shared Experience: Whatever the robot's fingertip sensors feel is conveyed to the user through the glove's pneumatic pouches.
  • Data Collection: The system records RGB camera views, robot kinematics, gripper state, and shared pressure data.

Diffusion Policy and Decision Making

• Diffusion Policy Defined: A denoising strategy used in robotics to determine the next movement state.
• The Process:
  1. The model takes a sequence of video or sensor inputs.
  2. It introduces random noise into the potential next-state sequence.
  3. It performs gradient descent to "denoise" the data.
  4. The model identifies the state with the least entropy—representing the most likely and logical next action.
• Ablation Studies: Research shows that the robot's behavior significantly changes when pressure data is removed. For example, without pressure information, the robot may suffer from early liftoff or fail to recognize when an object has successfully made contact with the ground.

Strategic Takeaways for Wearable Robotics

• Prediction Reliability: A wearable robot must be "right twice": it must correctly predict the intent to grasp to avoid dropping an object, and it must correctly predict the intent to release.
• Latency Challenges: If a model takes too long to predict, users become impatient and change their behavior, which corrupts the training data with "noise."
• Stakeholder Engagement: Researchers must talk directly with the target population (e.g., stroke survivors), especially if they do not belong to that group.
• Data Efficiency Over Big Data: Because humans use different strategies every time they move, researchers should focus on data-efficient strategies that leverage small, high-quality samples rather than massive datasets.

Questions & Discussion

• Q: How do you map glove motion to robot joint commands when structures differ? (Ame)
  • A: The project controls the "end effector" (the gripper point). We map the position and orientation of the palm and thumb as a proxy. This is more intuitive for a novice operator than controlling specific joint angles. However, this can create kinematic conflicts if the two systems have vastly different linkages.
• Q: Does the solution for dexterity lie in better AI or better hardware (soft robotics/biomimicry)? (Nurtis)
  • A: Both are essential. Robotics goes through cycles: once humans were good at complex hardware but poor at control; now AI (like $LLMs$ and $VLAs$) is good at identifying data but poor at understanding physics. The current strategy is to let humans handle high-level decision-making while the robot manages local, fast reactive tasks (like opening/closing).
• Q: What advice do you have for this career path? (Mary Anne)
  • A: You need sufficient math to read research papers and deep spatial reasoning (gained through sports, building things, or life experience). PhD training is heavily focused on communication; you must be able to convey ideas clearly through presentation and art. Above all, find what you are excited about doing on a day-to-day basis.
• Q: How do you handle robotic mistakes in high-stakes settings? (Anton)
  • A: Multiple levels of safety are used:
    1. Testing in free space before using objects.
    2. Hardware constraints like motor torque limits to prevent injury.
    3. Clinical supervision by a licensed Occupational Therapist ($OT$).
• Q: What prevents assistive tech like robotic hands from reaching the mass market? (Leo)
  • A: Several factors:
    1. Utility: People are very good at adapting their environment. One device shouldn't try to "do it all."
    2. Complexity: Many researchers focus on dexterity, but users might prioritize simpler solutions that don't require strapping on complex equipment.
    3. Cost: High-powered, motorized, data-heavy systems are currently too expensive for worldwide access.