Predicting spatial relationships for image descriptions

Abstract

This dissertation studies and demonstrates how machine generated spatial relationships between image objects can improve the accuracy in retrieval-based image caption generation systems. To this end, a Random Forest Tree based spatial preposition predictive model is developed. This model outperforms current best preposition prediction accuracy rates. The main contribution of this dissertation lies on a proposed image description framework that casts the generation of image descriptions as a combination of generation-retrieval process. In contrast to all current retrieval-based methods, the suggested novel approach is designed to extract image descriptions from the endless multimedia content found on the Web. The system is evaluated by both human and computational methods. Results show that the proposed image description system is highly competitive to current state-of-art retrieval-based image description algorithms. Good results were achieved when describing images containing two image objects with object labels connected by spatial prepositions, while the web-retrieval based approach was notably effective when describing single-object images.