| CODE | LIN3502 | ||||||||||||
| TITLE | Multimodal Interaction and Communication | ||||||||||||
| UM LEVEL | 03 - Years 2, 3, 4 in Modular Undergraduate Course | ||||||||||||
| MQF LEVEL | 6 | ||||||||||||
| ECTS CREDITS | 4 | ||||||||||||
| DEPARTMENT | Institute of Linguistics and Language Technology | ||||||||||||
| DESCRIPTION | Most natural language interaction with computers is based on text. This "single channel" perspective is the legacy of interface techniques that were developed even before personal computers were invented in the 1970s. Face-to-face communication between humans, however, is a completely different experience which involves multiple modalities. When two humans communicate with each other in a face-to-face situation, they do not only exchange speech. Other so-called channels or modalities, such as gaze (e.g., eye contact), facial expression, intonation, voice quality, and gestures (e.g., pointing) also play an important role in both the semantic and the socio-emotional aspects of communication. If interaction with computers is to become more natural, then it is crucial to understand the practical implications of multimodality for the development of advanced user interfaces. The topics to be studied will include annotation and representation of multimodal data, the use of speech, prosody, gesture, mage, and the integration of these modalities. Study-unit Aims The aim of this unit is to demonstrate and analyse the different modalities that figure in human-human interaction and to examine the ways in which the richness of multimodal communication can be applied to human-computer interfaces Learning Outcomes 1. Knowledge & Understanding: By the end of the study-unit the student will be able to: - Understand the difference between linguistic and non-linguistic (especially gestural) communication; - construct workable computational models of production integrating output in different modalities; - make use of existing technologies (such as software agents and avatars) to create multimodal interfaces. 2. Skills: By the end of the study-unit the student will be able to: - annotate multimodal data in standard formats such as XML; - use existing software libraries and packages to create novel applications. Main Text/s and any supplementary readings - H. Bunt and R-J Beun (2001). Cooperative Multimodal Communication. Berlin & Heidelberg: Springer [Unavailable in UoM library] - B. Granström and D. House (2002). Multimodality in Language and Speech Systems. Dordrecht: Kluwer Academic Publishers [Unavailable in UoM library] - D. Jurafsky & J. H. Martin (2009). Speech and Language Processing. (2nd edition). Indiana: Prentice Hall [Unavailable in UoM Library] |
||||||||||||
| STUDY-UNIT TYPE | Lecture and Seminar | ||||||||||||
| METHOD OF ASSESSMENT |
|
||||||||||||
| LECTURER/S | Patrizia Paggio |
||||||||||||
|
The University makes every effort to ensure that the published Courses Plans, Programmes of Study and Study-Unit information are complete and up-to-date at the time of publication. The University reserves the right to make changes in case errors are detected after publication.
The availability of optional units may be subject to timetabling constraints. Units not attracting a sufficient number of registrations may be withdrawn without notice. It should be noted that all the information in the description above applies to study-units available during the academic year 2023/4. It may be subject to change in subsequent years. |
|||||||||||||