How to identify patient perception of AI voice robots in the follow-up scenario? A multimodal identity perception method based on deep learning

• Published in: Journal of biomedical informatics 2024-12, Vol.160, p.104757, Article 104757
• Main Authors: Liu, Mingjie, Chen, Kuiyou, Ye, Qing, Wu, Hong
• Format: Article
• Language: English
• Subjects: AI voice robots; Algorithms; Artificial Intelligence; Deep Learning; Female; Follow-up; Follow-Up Studies; Human-AI interactions; Humans; Male; Multimodal analysis; Neural Networks, Computer; Post-diagnosis management; Robotics; Voice
• ISSN: 1532-0464; 1532-0480; 1532-0480
• DOI: 10.1016/j.jbi.2024.104757

[Display omitted] Post-discharge follow-up stands as a critical component of post-diagnosis management, and the constraints of healthcare resources impede comprehensive manual follow-up. However, patients are less cooperative with AI follow-up calls or may even hang up once AI voice robots are perceived. To improve the effectiveness of follow-up, alternative measures should be taken when patients perceive AI voice robots. Therefore, identifying how patients perceive AI voice robots is crucial. This study aims to construct a multimodal identity perception model based on deep learning to identify how patients perceive AI voice robots. Our dataset includes 2030 response audio recordings and corresponding texts from patients. We conduct comparative experiments and perform an ablation study. The proposed model employs a transfer learning approach, utilizing BERT and TextCNN for text feature extraction, AST and LSTM for audio feature extraction, and self-attention for feature fusion. Our model demonstrates superior performance against existing baselines, with a precision of 86.67%, an AUC of 84%, and an accuracy of 94.38%. Additionally, a generalization experiment was conducted using 144 patients’ response audio recordings and corresponding text data from other departments in the hospital, confirming the model’s robustness and effectiveness. Our multimodal identity perception model can identify how patients perceive AI voice robots effectively. Identifying how patients perceive AI not only helps to optimize the follow-up process and improve patient cooperation, but also provides support for the evaluation and optimization of AI voice robots.