Please use this identifier to cite or link to this item:
Title: What and where : an empirical investigation of pointing gestures and descriptions in multimodal referring actions
Authors: Gatt, Albert
Paggio, Patrizia
Keywords: Natural language processing (Computer science)
Corpora (Linguistics)
Linguistic analysis (Linguistics)
Reference (Linguistics)
Issue Date: 2013
Publisher: Association for Computational Linguistics
Citation: Gatt, A., & Paggio, P. (2013). What and where: an empirical investigation of pointing gestures and descriptions in multimodal referring actions.14th European Workshop on Natural Language Generation, Sofia. 82-91.
Abstract: Pointing gestures are pervasive in human referring actions, and are often combined with spoken descriptions. Combining gesture and speech naturally to refer to objects is an essential task in multimodal NLG systems. However, the way gesture and speech should be combined in a referring act remains an open question. In particular, it is not clear whether, in planning a pointing gesture in conjunction with a description, an NLG system should seek to minimise the redundancy between them, e.g. by letting the pointing gesture indicate locative information, with other, nonlocative properties of a referent included in the description. This question has a bearing on whether the gestural and spoken parts of referring acts are planned separately or arise from a common underlying computational mechanism. This paper investigates this question empirically, using machine-learning techniques on a new corpus of dialogues involving multimodal references to objects. Our results indicate that human pointing strategies interact with descriptive strategies. In particular, pointing gestures are strongly associated with the use of locative features in referring expressions.
ISBN: 9781937284565
Appears in Collections:Scholarly Works - InsLin

Files in This Item:
File Description SizeFormat 
enlg2013-pointing.pdf739.7 kBAdobe PDFView/Open

Items in OAR@UM are protected by copyright, with all rights reserved, unless otherwise indicated.