Multimedia interfaces for blind people

Abstract

Visually impaired individuals are a growing segment of our population. However social mechanisms are not designed with this fraction of people in mind, making the development of electronic assistive tools essential in order to perform basic day-to-day activities. Generally, assistive technologies come in the form of expensive and specialized devices which the visually challenged have to independently look after and carry around. Due to the increasing computational capabilities and on board sensing components found in many mobile phones used today, such devices have become an ideal candidate for designing solutions to aid the visually impaired. The objective of this FYP is to develop a multimedia user interface which is aimed to aid the visually challenged. We propose and design a grocery products recognition system utilizing computer vision and machine learning techniques. Finally, our system works towards allowing visually impaired or blind individuals to identify products in grocery stores and supermarkets without any additional assistance, thus encouraging them to perform daily activities with as least help as possible, further promoting their social wellness. Our approach is composed of two main modules one capable of classifying grocery products using an unsupervised feature extraction methods posed by deep learning techniques while the other module is capable of recognizing products in an image using the traditionally handcrafted feature extraction algorithms. We considered multiple robust approaches to identify the one most suited for our task. Through evaluation we determined that the best approach for classification is to finetune a convolutional neural network pre-trained on a larger dataset. We were successful in not only surpassing our target accuracy of 21.19% but also obtaining an accuracy of 63%. Evaluation on the recognition module highlighted the difficulties met for recognizing similar objects in a scene, by achieving a recognition rate of 41.38%.