How, Chun Kit and Mohd Khairuddin, Ismail and Mohd Razman, Mohd Azraai and Anwar, P. P. Abdul Majeed and Mohd Isa, Wan Hasbullah (2022) Development of audio-visual speech recognition using deep-learning technique. Mekatronika - Journal of Intelligent Manufacturing & Mechatronics, 4 (1). pp. 88-95. ISSN 2637-0883. (Published)
|
Pdf
Development of audio visual speech recognition.pdf Available under License Creative Commons Attribution Non-commercial. Download (1MB) | Preview |
Abstract
Deep learning is a technique with artificial intelligent (AI) that simulate humans’ learning behavior. Audio-visual speech recognition is important for the listener understand the emotions behind the spoken words truly. In this thesis, two different deep learning models, Convolutional Neural Network (CNN) and Deep Neural Network (DNN), were developed to recognize the speech’s emotion from the dataset. Pytorch framework with torchaudio library was used. Both models were given the same training, validation, testing, and augmented datasets. The training will be stopped when the training loop reaches ten epochs, or the validation loss function does not improve for five epochs. At the end, the highest accuracy and lowest loss function of CNN model in the training dataset are 76.50% and 0.006029 respectively, meanwhile the DNN model achieved 75.42% and 0.086643 respectively. Both models were evaluated using confusion matrix. In conclusion, CNN model has higher performance than DNN model, but needs to improvise as the accuracy of testing dataset is low and the loss function is high.
| Item Type: | Article |
|---|---|
| Uncontrolled Keywords: | Audio-Visual; Speech Recognition; Deep-Learning; Emotion; Spectrogram |
| Subjects: | T Technology > TJ Mechanical engineering and machinery T Technology > TK Electrical engineering. Electronics Nuclear engineering T Technology > TS Manufactures |
| Faculty/Division: | Institute of Postgraduate Studies Faculty of Manufacturing and Mechatronic Engineering Technology |
| Depositing User: | Mrs Norsaini Abdul Samat |
| Date Deposited: | 09 Mar 2023 03:50 |
| Last Modified: | 09 Mar 2023 03:50 |
| URI: | http://umpir.ump.edu.my/id/eprint/37244 |
| Download Statistic: | View Download Statistics |
Actions (login required)
 |
View Item |