Indian Sign Language Detection and Translation using Machine Learning in English

Abstract

Indian Sign Language (ISL) is a critical means of communication for India's deaf and hard-of-hearing population of millions. However, the lack of accessible tools prevents their communication with non-ISL speakers and leaves them vulnerable to social isolation. To address this lack, this study proposes a gesture recognition system for ISL-to-speech translation using machine learning algorithms—Support Vector Machine (SVM), k-Nearest Neighbors (k-NN), and Recurrent Neural Networks (RNN). A performance-evaluating dataset with all ISL alphabets, comprising 20 frames per sign, was generated. The evaluation in static and dynamic gesture recognition indicates the performance of these models, with RNN being the most appropriate for the dynamic sequence. Real-time experimentation using webcam input confirmed the flexibility of the developed approach, indicating more than 90% accuracy by using preprocessing methods. Moreover, the combination with Google Text-to-Speech (gTTS) also enables enhanced real-time translation, thus allowing the system to be applicable for real-world uses. The outputs offer a model for building mobile applications and public service tools to enable inclusive communication. Future work will add more data to the dataset, improve recognition of complex gestures, and incorporate context-aware comprehension in order to enhance the functionality of the system.