Gesture recognition for hologram interaction : an application for museums

Abstract

The aim of this research project is to develop a deeper understanding of hand gesture recognition for the manipulation of hologram objects. The project hinges on the human-computer interaction (HCI) for an augmented user experience. This can have a wide range of uses across various fields and domains, such as in cultural heritage and museum visits. Hand gesture techniques and holographic object manipulation is an emerging research field in AI, making use of novel computer vision techniques and technologies, such as machine learning algorithms, deep learning and neural networks, feature extraction from images and intelligent interfaces. By evaluating the different hand gesture recognition techniques and making use of the optimal method within this study, a system which is highly efficient and accurate can be produced to achieve the goal of a more immersive and interactive user experience. Therefore, this study aims to take a new approach in which the HCI is a very natural one, almost simulating a completely new way of how society should build both museums and any educational sites. For this project, hand gestures are captured by the camera using the hand-in-frame. Features of the hand-in-frame are extracted and used to make a calculated prediction using the trained neural network (NN) model for the predefined hand gestures. The NN model is trained using a custom designed dataset built from scratch using raw data samples and a one-hot-encoding method of classification. The hand gesture is used to manipulate the 3D-model of the historical site/artefact shown as a hologram with transformations such as lateral shifts, rotations, zooming in and out. This is rendered in real time, with an immediate and visible result shown to the user. Results from experiments carried out indicate that the model can accurately and e↵ectively translate designated human hand gestures into functional commands for the system to make use of after undergoing several training stages. Training data for this NN model covers a wide variety of hand sizes, proportions and other physical features to ensure the model does not respond only to specific users’ hands, thus making the entire system usable for anyone who attempts to interact with it.