Feature-Fusion based Audio-Visual Speech Recognition using Lip Geometry Features in Noisy Environment

M. Z., Ibrahim and Mulvaney, D. J. and M. F., Abas (2015) Feature-Fusion based Audio-Visual Speech Recognition using Lip Geometry Features in Noisy Environment. ARPN Journal of Engineering and Applied Sciences, 10 (23). pp. 17521-17527. ISSN 1819-6608. (Published)

	PDF jeas_1215_3203.pdf Restricted to Repository staff only Download (646kB) | Request a copy
Preview	PDF fkee-2015-zamri-Feature-Fusion based Audio-Visual.pdf Download (694kB) | Preview

Abstract

Humans are often able to compensate for noise degradation and uncertainty in speech information by augmenting the received audio with visual information. Such bimodal perception generates a rich combination of information that can be used in the recognition of speech. However, due to wide variability in the lip movement involved in articulation, not all speech can be substantially improved by audio-visual integration. This paper describes a feature-fusion audio-visual speech recognition (AVSR) system that extracts lip geometry from the mouth region using a combination of skin color filter, border following and convex hull, and classification using a Hidden Markov Model. The comparison of the new approach with conventional audio-only system is made when operating under simulated ambient noise conditions that affect the spoken phrases. The experimental results demonstrate that, in the presence of audio noise, the audio-visual approach significantly improves speech recognition accuracy compared with audio-only approach.

Item Type:	Article
Uncontrolled Keywords:	Lip geometry, feature fusion, audio-visual speech recognition, OpenCV
Subjects:	T Technology > TK Electrical engineering. Electronics Nuclear engineering
Faculty/Division:	Faculty of Electrical & Electronic Engineering
Depositing User:	Mr. Zamri Ibrahim
Date Deposited:	20 Apr 2016 04:21
Last Modified:	20 Mar 2018 06:51
URI:	http://umpir.ump.edu.my/id/eprint/12890
Download Statistic:	View Download Statistics

Actions (login required)

View Item