Multimodal reference in dialogue : towards a balanced corpus

Abstract

Generation of Referring Expressions (GRE), e.g., Dale and Reiter (1995), is one of the core tasks of Natural Language Generation (NLG) systems. Usually it is formulated as an identification problem: given a domain representing entities and their properties, construct a referring expression for a target referent or set of target referents which singles it out from its distractors. Recently, researchers in this area have turned their attention to multimodal referring acts, in particular, the interaction between the two modalities of pointing and describing – e.g., Kranstedt et al. (2006), Piwek (2007), and Van der Sluis and Krahmer (2007). Additionally, psycholinguistic work is increasingly investigating the conditions governing the use of pointing gestures as part of referring acts in dialogue, opposed to monologue. Here, we present the design of an experiment on multimodal reference in two-party dialogue. The purpose of the experiment is to create a corpus that can inform the development of multimodal GRE algorithms.