Lipreading Using Liquid State Machine with STDP-Tuning

Lipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker’s lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of comput...

Full description

Saved in:
Bibliographic Details
Published in:Applied sciences 2022-10, Vol.12 (20), p.10484
Main Authors: Yu, Xuhu, Wan, Zhong, Shi, Zehao, Wang, Lei
Format: Article
Language:English
Subjects:
Accuracy
Cameras
Communication
Datasets
Deep learning
Firing pattern
Genetic algorithms
lipreading
liquid state machine
Methods
Neural networks
Neurons
Sensors
State machines
STDP
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Lipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker’s lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of computing resources, which is not conducive to the migration of the system to edge devices. Inspired by the work of Spiking Neural Networks (SNNs) in recognizing human actions and gestures, we propose a lipreading system based on SNNs. Specifically, we construct the front-end feature extractor of the system using Liquid State Machine (LSM). On the other hand, a heuristic algorithm is used to select appropriate parameters for the classifier in the backend. On small-scale lipreading datasets, our recognition accuracy achieves good results. We claim that our network performs better in terms of accuracy and ratio of learned parameters compared to other networks, and has superior advantages in terms of network complexity and training cost. On the AVLetters dataset, our model achieves a 5% improvement in accuracy over traditional methods and a 90% reduction in parameters over the state-of-the-art.
ISSN:2076-3417
2076-3417
DOI:10.3390/app122010484