Real‐time sign language translator using deep learning

Abstract

This research focuses on the development of a real‐time sign language translator capable of identifying both alphabetic characters and gesture‐based signs. The system was built using deep learning models trained on purpose‐specific datasets. Over the course of the project, three core neural network architectures were designed, resulting in a total of seven different models, each tested with different data inputs. Among these, the Feed Forward Neural Network (FNN) using hand and face landmarks delivered the highest accuracy 97.87% for alphabet recognition and 97.27% for gesture classification. The application functions by activating the device’s camera, which detects hand and facial landmarks for live gesture recognition, supporting up to 30 predefined gesture signs. A separate version of the program is used for alphabet recognition, identifying letters from A to Z including the space character, allowing users to construct words manually. During testing, the top performing alphabet recognition model which was the landmark model had a 100% accuracy during the five gesture test cases, while the leading gesture recognition model the FNN model with hand landmarks only correctly identified three out of five. In summary this study contributes to the development of an efficient real‐time sign language recognition by using landmark‐based alphabet translator and gesture translation using different neural network architectures. The proposed system has significant potential to enhance accessibility and promote inclusive communication between both communities through assistive technology applications.