STREAMS: An Assistive Multimodal AI Framework for Empowering Biosignal Based Robotic Controls

End-effector based assistive robots face persistent challenges in generating smooth and robust trajectories when controlled by human's noisy and unreliable biosignals such as muscle activities and brainwaves. The produced endpoint trajectories are often jerky and imprecise to perform complex ta...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-10
Main Authors: Rabiee, Ali, Ghafoori, Sima, Bai, Xiangyu, Ostadabbas, Sarah, Abiri, Reza
Format: Article
Language:English
Subjects:
Adaptive control
Adaptive systems
Deep learning
End effectors
Grasping (robotics)
Head movement
Interactive control
Robot arms
Robot control
Robot dynamics
Robot learning
Robotics
Robust control
Service robots
Streams
Task complexity
Trajectories
User satisfaction
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:End-effector based assistive robots face persistent challenges in generating smooth and robust trajectories when controlled by human's noisy and unreliable biosignals such as muscle activities and brainwaves. The produced endpoint trajectories are often jerky and imprecise to perform complex tasks such as stable robotic grasping. We propose STREAMS (Self-Training Robotic End-to-end Adaptive Multimodal Shared autonomy) as a novel framework leveraged deep reinforcement learning to tackle this challenge in biosignal based robotic control systems. STREAMS blends environmental information and synthetic user input into a Deep Q Learning Network (DQN) pipeline for an interactive end-to-end and self-training mechanism to produce smooth trajectories for the control of end-effector based robots. The proposed framework achieved a high-performance record of 98% in simulation with dynamic target estimation and acquisition without any pre-existing datasets. As a zero-shot sim-to-real user study with five participants controlling a physical robotic arm with noisy head movements, STREAMS (as an assistive mode) demonstrated significant improvements in trajectory stabilization, user satisfaction, and task performance reported as a success rate of 83% compared to manual mode which was 44% without any task support. STREAMS seeks to improve biosignal based assistive robotic controls by offering an interactive, end-to-end solution that stabilizes end-effector trajectories, enhancing task performance and accuracy.
ISSN:2331-8422