ENTERFACE2010ProjectProposal.docVIP

ENTERFACE 2010 Project Proposal Title: Audio-visual speech recognition Principal Investigators: Dr. Hakan Erdogan and Saygin Topkaya Date: 14/12/2009 Abstract: The project aims to develop real-time audio-visual speech recognition software that will show the recent developments in audio-visual speech recognition research. 1. INTRODUCTION AND PROJECT OBJECTIVES Speech recognition is an open research area that requires continuing research effort for further advancement. Although one can obtain high recognition rates in audio-only speech recognition in controlled environments, recognition accuracy degrades in noisy environments. For such cases, supporting audio information with visual information is a commonly recommended approach in the literature. This approach, called audio-visual speech recognition, aims to provide the supplemental visual information from a video of the lip region of the subject of interest. We propose an Enterface 2010 workshop project to work on building real-time software that showcases the current and emerging audio-visual speech recognition technology. Our group at Sabanci University has an ongoing nationally supported project on improving the performance of audio-visual speech recognition. In this project, we propose a tandem classifier fusion approach for combining audio and visual information. In addition, we work on improving visual feature extraction by extracting stable features correcting for head motion artifacts and normalized features correcting for speaker and environment variability. We can list the visual feature extraction aspects that we aim to include in the proposed software as follows: Extracting visual features using edge and texture information. Using dynamic visual features that account for lip motion in the region of interest. Normalization of visual features using warping. In addition, we propose to implement the following methods in the core recognition part of the audio-visual speech recognition system: We propose