Multi-Modal System for Walking Safety for the Visually Impaired: Multi-Object Detection and Natural Language Generation

This study introduces a system for visually impaired individuals in a walking environment. It combines object recognition using YOLOv5 and cautionary sentence generation with KoAlpaca. The system employs image data augmentation for diverse training data and GPT for natural language training. Further...

Full description

Saved in:
Bibliographic Details
Published in:Applied sciences 2024-09, Vol.14 (17), p.7643
Main Authors: Lee, Jekyung, Cha, Kyung-Ae, Lee, Miran
Format: Article
Language:English
Subjects:
Augmented reality
Classification
Deep learning
KoAlpaca
Language
Natural language
natural language generation
Neural networks
object detection
Orthopedic apparatus
Real time
Roads & highways
Smart devices
Smartphones
Vibration
Visual impairment
visually impaired
walking assistance sentence
YOLOv5
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This study introduces a system for visually impaired individuals in a walking environment. It combines object recognition using YOLOv5 and cautionary sentence generation with KoAlpaca. The system employs image data augmentation for diverse training data and GPT for natural language training. Furthermore, the implementation of the system on a single board was followed by a comprehensive comparative analysis with existing studies. Moreover, a pilot test involving visually impaired and healthy individuals was conducted to validate the system’s practical applicability and adaptability in real-world walking environments. Our pilot test results indicated an average usability score of 4.05. Participants expressed some dissatisfaction with the notification conveying time and online implementation, but they highly praised the system’s object detection range and accuracy. The experiments demonstrated that using QLoRA enables more efficient training of larger models, which is associated with improved model performance. Our study makes a significant contribution to the literature because the proposed system enables real-time monitoring of various environmental conditions and objects in pedestrian environments using AI.
ISSN:2076-3417
2076-3417
DOI:10.3390/app14177643