Corpus-based evaluation of referring expressions generation

Abstract

Corpus-based evaluation of NLP systems has become a dominant methodology. Typically, some metric is invoked to evaluate the results produced by a system against a ‘gold standard’ represented in the corpus. Despite growing recognition of the importance of empirical evaluation in NLG, resources and methodologies for evaluation of Generation of Referring Expressions (GRE) are in their infancy (but c.f. Viethen and Dale (2006)), although this area has been studied intensively since the publication of the Incremental Algorithm (IA) by Dale and Reiter (1995). This contribution describes some of the difficulties which inhere in any corpus-based evaluation exercise involving GRE, as well as a methodology to create a corpus aimed at overcoming these difficulties.