The effect of orientation and pose on spatial relation detection

Abstract

Detecting relationships between objects is an important way to thoroughly understand images. In this work we explore the effect of human pose, as a proxy for orientation, on detecting a specifi c subset of visual relationships, namely Spatial Relations. We use a human pose detector to detect 2D poses in images, impute the poses to correct missing joints and then encode the full poses in a number of representations. A number of models are trained using geometric and language features, incorporating pose features into a subset of the models. Overall, models trained without pose features produced better results. However, models trained with pose features showed a performance improvement for the relationship opposite. This signifi es the importance of pose features for the subset of Spatial Relations where the use of the intrinsic frame of reference is required.