Speech and gesture alignment in conversational Maltese : the issue of temporal alignment between speech and non-verbal behaviour

Abstract

This dissertation deals with the issue of temporal alignment between speech and non-verbal behaviour. Research on the importance of this alignment and the way it is perceived in multimodal behaviour is a central topic for studies of multimodal communication, and it is also important for the development of believable conversational software agents. The empirical data studied in this thesis are produced through an experiment which aimed to verify the importance of synchronization of speech and gesture during natural conversations by studying how this synchronization is perceived by listeners and viewers. MAMCO, a collection of video-recorded Maltese conversations, was provided by the Institute of Linguistics and was chosen for this study. From selected videos, which showed a natural Maltese conversation between two people, clips were taken in which one of the speakers performed a beat gesture. These clips were temporally manipulated to produce a number of target clips with different types of delays between the audio and the video channels. Participants were then asked to judge whether the clips produced were synchronous or not. The results show that the synchronization of the audio and visual channels of a conversation is important for the well-formedness of multimodal utterance. We find that participants are able to discern altered timing of gesture and speech if the gesture comes later or earlier than normal. These findings partly confirm the results from previous studies and, more importantly, do so using naturally-occurring Maltese conversational data.