A computer-aided speech analytics approach for pronunciation feedback using deep feature clustering

• Published in: Multimedia systems 2023-06, Vol.29 (3), p.1699-1715
• Main Authors: Nazir, Faria, Majeed, Muhammad Nadeem, Ghazanfar, Mustansar Ali, Maqsood, Muazzam
• Format: Article
• Language: English
• Subjects: Algorithms; Clustering; Computer Communication Networks; Computer Graphics; Computer Science; Cryptology; Data Storage Representation; Error correction; Feedback; Learning; Linguistics; Multimedia Information Systems; Operating Systems; Phonemes; Role of Deep Learning Models & Analytics in Industrial Multimedia Environment; Special Issue Paper; Speech; Support vector machines
• ISSN: 0942-4962; 1432-1882
• DOI: 10.1007/s00530-021-00822-5

Nowadays, the demand for language learning is increasing because people need to communicate with other people belonging to different regions for their business deals, study, etc. During language learning, a lot of pronunciation mistakes occur due to unfamiliarity with a new language and differences in accent. In this paper, we perform speech mistakes analysis using deep feature-based clustering. We proposed two novel methods for speech analysis, one to deal with phonemic errors (confusing phonemes) and the other to deal with the prosodic errors (partially changed pronunciation variation of phones). For accurate and efficient language learning, it is important to learn both phonemic as well as prosodic error corrections. In our first method, we perform speech analysis by combining deep CNN features and clustering algorithm to detect the phonemic errors. We classify the phonemes using K-nearest neighbor, Naïve Bayes, and support vector machine (SVM). We perform experiments on the six most frequently mispronounced confusing pairs of Arabic to handle phonemic errors and achieve an accuracy of 94%. In our second method, we proposed the unsupervised phone variation model (PVM) to detect prosodic errors. In PVM, each phone is extended to represent the different types of pronunciation variation of that phone with different proficiency levels. We use an Arabic dataset of 28 individual phones for speech analysis and provide feedback based on the variation of each phone and achieves an accuracy of 97%.