Vision based speech recognition system

Abstract

Visual Speech Recognition is a field that has, in recent years, been subject to further study since it can help in improving noise reduction in Automatic Speech Recognition systems. Lip reading in humans is used to understand speech from visual cues mainly related to the movement of lips. This project explores the idea of recognizing visemes, the basic unit of lip reading, using a computerized system. This can lead to improving results from audio-only ASRs which are affected by noisy audio or provide speech recognition where no audio is available. The project is divided in modular parts which concern lip segmentation based on colour space information, lip feature extraction, forced audio alignment and viseme classification using a number of classifiers; Artificial Neural Networks, Support Vector Machines, Hidden Markov Models and Hidden Random Conditional Fields. The outputs of the different parts were used to generate the final results, which show negligible recognition rates. This was mainly attributed to errors accumulating from one step to another, producing noise which affected the data used for training. Finally, the way forward is discussed in view of the results obtained.