Development of audio-visual speech recognition using deep-learning technique

How, Chun Kit and Mohd Khairuddin, Ismail and Mohd Razman, Mohd Azraai and Anwar, P. P. Abdul Majeed and Mohd Isa, Wan Hasbullah (2022) Development of audio-visual speech recognition using deep-learning technique. Mekatronika - Journal of Intelligent Manufacturing & Mechatronics, 4 (1). pp. 88-95. ISSN 2637-0883. (Published)

Development of audio visual speech recognition.pdf
Available under License Creative Commons Attribution Non-commercial.

Download (1MB) | Preview


Deep learning is a technique with artificial intelligent (AI) that simulate humans’ learning behavior. Audio-visual speech recognition is important for the listener understand the emotions behind the spoken words truly. In this thesis, two different deep learning models, Convolutional Neural Network (CNN) and Deep Neural Network (DNN), were developed to recognize the speech’s emotion from the dataset. Pytorch framework with torchaudio library was used. Both models were given the same training, validation, testing, and augmented datasets. The training will be stopped when the training loop reaches ten epochs, or the validation loss function does not improve for five epochs. At the end, the highest accuracy and lowest loss function of CNN model in the training dataset are 76.50% and 0.006029 respectively, meanwhile the DNN model achieved 75.42% and 0.086643 respectively. Both models were evaluated using confusion matrix. In conclusion, CNN model has higher performance than DNN model, but needs to improvise as the accuracy of testing dataset is low and the loss function is high.

Item Type: Article
Uncontrolled Keywords: Audio-Visual; Speech Recognition; Deep-Learning; Emotion; Spectrogram
Subjects: T Technology > TJ Mechanical engineering and machinery
T Technology > TK Electrical engineering. Electronics Nuclear engineering
T Technology > TS Manufactures
Faculty/Division: Institute of Postgraduate Studies
Faculty of Manufacturing and Mechatronic Engineering Technology
Depositing User: Mrs Norsaini Abdul Samat
Date Deposited: 09 Mar 2023 03:50
Last Modified: 09 Mar 2023 03:50
Download Statistic: View Download Statistics

Actions (login required)

View Item View Item