Extracting Visual Micro-Doppler Signatures From Human Lips Motion Using UoG Radar Sensing Data for Hearing Aid Applications

This study proposes a secure and effective lips-reading system that can accurately detect lips movements, even when face masks are worn. The system utilizes radio frequency (RF) sensing and ultra-wideband (UWB) radar technology, which overcomes the challenges posed by traditional vision-based system...

Full description

Saved in:
Bibliographic Details
Published in:IEEE sensors journal 2023-10, Vol.23 (19), p.22111-22118
Main Authors: Saeed, Umer, Shah, Syed Aziz, Ghadi, Yazeed Yasin, Khan, Muhammad Zakir, Ahmad, Jawad, Shah, Syed Ikram, Hameed, Hira, Abbasi, Qammer H.
Format: Article
Language:English
Subjects:
Algorithms
Artificial neural networks
Auditory system
Datasets
Deep learning
Face recognition
Hearing aids
Human motion
Inceptionv3
Lips
lips-reading
Machine learning
Mouth
Radar signatures
Radio frequency
radio frequency (RF) sensing
residual neural network (ResNet50)
Spectrograms
Speech recognition
ultra-wideband (UWB) radar
Ultrawideband radar
Vision systems
visual geometry group (VGG16)
Voice communication
Voice control
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This study proposes a secure and effective lips-reading system that can accurately detect lips movements, even when face masks are worn. The system utilizes radio frequency (RF) sensing and ultra-wideband (UWB) radar technology, which overcomes the challenges posed by traditional vision-based systems. By leveraging deep learning models, the system interprets lips and mouth movements and achieves an overall accuracy of 90% for both mask-on and mask-off scenarios. The study utilized a trusted dataset from the University of Glasgow (UoG), consisting of spectrograms of lips motions stating five vowels and a voiceless class from distinct participants. The cutting-edge deep learning algorithm, residual neural network (ResNet50), was used for the evaluation of the dataset and achieved an 87% accurate detection rate with a mask-on scenario, which is a 14% improvement compared to prior published work. The findings of this study contribute to the development of a robust lips-reading framework that can enhance communication accessibility in applications such as hearing aids, voice-controlled systems, biometrics, and more.
ISSN:1530-437X
1558-1748
DOI:10.1109/JSEN.2023.3308972