Drone object detection based on real-time sign language communication

Abstract

In recent years Unmanned Aerial Vehicles (UAVs) such as drones have progressively advanced in various sectors essentially hardware technologies, autonomous maneuver, as well as computational power have led for drones to be commercially available which were once only used in military and governmental possession. Combining this with the continuous rise of Artificial Intelligence (AI) allows for limits in ways that these vehicles can be used to be pushed, through different types of image processing and recognition. In this dissertation, drones, image processing and object recognition are merged together with the aim that a personal assistant helper is created. However, this is not just any personal assistant like Alexa, Siri, and Google Assistant, but rather an assistant that will communicate through sign language. Sign language is a visually transmitted language which is made up of sign patterns constructed together to form a specific meaning. Due to the complexities of digitally capturing and translating sign language, this research domain has lacked to compare to the advanced speech recognition available nowadays. Therefore, this project will merge the use of drones, to follow the user around and be in frame of hand gestures, as well as object recognition for sign language characters that make part of the American Sign Language (ASL). The aim is that, while the drone follows the person around, the user is able to spell out a word, letter by letter, forming a word referring to an object. The drone will then pivot while scanning the area for the user’s desired object. If the object is found, the drone maneuvers itself towards the object. The drone being used in this project is a DJI Tello drone which even though it has a limiting battery life of around 13 minutes, it also allows for Python code to be used as means of control. This is beneficial as Python is a necessity in most areas of computer vision allowing for a real-time stream of the drone’s view to be processed and outputted to the user. This project makes use of various object recognition such as face by using Viola-Jones, sing language recognition based on a custom data set and object detection using the COCO data set.