Enhancing Sign Language Learning with Augmented Reality

Sign Language is a way of communication used by hearing-impaired people. As many normal people don’t know the Sign Language it is difficult to understand by them to understand thus, increasing the communication gap between normal and hearing-impaired people. To overcome this problem many systems,using Machine Learning(ML), Artificial Intelligence(AI), Deep Learning(DL), computer vision methods, were developed and being developed. These systems are used to translate Indian Sign Language(ISL) to understandable text, facilitating the communication between normal and hearing-impaired people. This study briefly explains various type of Indian Sign Language(ISL) recognition systems.

Communication plays a major role in day to day life. But it is very difficult for normal people to communicate with deaf, dumb & blind people & vice versa. Because of a deaf, dumb use sign language for communication and normal people are unable to understand sign language. So there is a need of converting sign language. This paper minimises the barrier of communication between them. In this paper, Hand gestures are captured, processed with the help of MATLAB and then converted into speech & text. For speech & text, two languages are chosen English & Hindi. In this paper features value of images is evaluated based on moment technique. Two classification techniques (PNN & KNN) are used & performance is compared between both classifiers. This paper will help to do the easiest communication between normal & hearing impaired people.

Sign language is a system for communication using visual gestures and signs for deaf and dumb people. This language is especially used by people who can’t speak or hear. Sign language is oldest and natural form of language for communication, since most of the people are not aware of sign language, hence it is tedious to understand, for solving this problem we have built a real time method using convolution neural networks (CNN) for hand-based gestures to detect sign language. In our CNN model, first the hand is passed through a filter and then the filter is applied. Finally, the hand is passed through a classifier which predicts the class of the hand gestures. Our model provides 98% accuracy for the alphabet A-Z letters. On the other hand, the most common hurdle deaf and dumb come across is communication with the normal people and with the fact i.e., each and every normal person doesn't know the sign language. Another main feature of the project is to create a communication system for the deaf people. The functionality of these part of the system is to translate audio message to the corresponding sign language. This part of the system takes the audio message as input, and converts the audio recorded message into the respective sign image and videos and displays the relevant American sign language or GIFs which we have already defined. With the aid of this part of the system, communication between normal and deaf people gets feasible. Overall, the main idea of the project is to make a system which can help deaf and dumb to interact with normal people.

In this world, we have more than 466 million deaf dumb peoples. Normal peoples are communicating with voice, likewise deaf and dumb people are communicating by the use of sign language. Generally, without knowing the language, one cannot talk to other people in the world, similarly without knowing the sign language normal people cannot talk to deaf and dumb people. Within themselves, deaf and dumb people will communicate using sign language, which normal people cannot understand, but still hearing is one way to understand the people, so expression and gesture are also a way to understand thoughts of deaf and dumb people. To find a solution for this problem, either normal person needs to learn the sign language or we need to create a platform which will convert the sign language to voice and voice to sign language. Though we have many different types platform, still it not efficient so we need to use the advanced method which recognizes the input much efficiently. Since we are using gesture and voice as an input, this will act as bidirectional way communication. The focus of this work is to create a bidirectional-based system to identify sign language gestures from the sign language images and converting that to voice output. Since we are developing the application to computer and android, it should have two-way communication so that both can act as input and output node. For efficient voice input and output, we are using android mobile.KeywordsGestureSign languageBidirectionalDeaf and dumbTwo-way communication systemComputer visionSIFT algorithm

Hand gesture is one of the method used in sign language for non-verbal communication. It is most commonly used by deaf & dumb people who have hearing or speech problems to communicate among themselves or with normal people. Various sign language systems has been developed by many makers around the world but they are neither flexible nor cost-effective for the end users. Hence in this paper introduced software which presents a system prototype that is able to automatically recognize sign language to help deaf and dumb people to communicate more effectively with each other or normal people. Pattern recognition and Gesture recognition are the developing fields of research. Being a significant part in nonverbal communication hand gestures are playing key role in our daily life. Hand Gesture recognition system provides us an innovative, natural, user friendly way of communication with the computer which is more familiar to the human beings. By considering in mind the similarities of human hand shape with four fingers and one thumb, the software aims to present a real time system for recognition of hand gesture on basis of detection of some shape based features like orientation, Centre of mass centroid, fingers status, thumb in positions of raised or folded fingers of hand

<p> Communication between normal and handicapped person such as deaf people, dumb people, and blind people has always been a challenging task. It has been observed that they find it really difficult at times to interact with normal people with their gestures, as only a very few of those are recognized by most people. Since people with hearing impairment or deaf people cannot talk like normal people so they have to depend on some sort of visual communication in most of the time. Sign Language is the primary means of communication in the deaf and dumb community. As like any other language it has also got grammar and vocabulary but uses visual modality for exchanging information. The importance of sign language is emphasized by the growing public approval and funds for international project. Interesting technologies are being developed for speech recognition but no real commercial product for sign recognition is actually there in the current market. So, to take this field of research to another higher level this project was studied and carried out. The basic objective of this research is to develop MATLAB based real time system for hand gesture recognition which recognize hand gestures, features of hands such as centroid, peak calculation, angle calculation and convert gesture images into voice and vice versa. To implement this system we used simple night vision web-cam with 20 megapixel intensity. The idea consisted of designing and building up an intelligent system using image processing, data mining and artificial intelligence concepts to take visual inputs of sign languages hand gestures and generate easily recognizable form of outputs in the form of text and voice with 82% accuracy.</p> <p><a href="https://journalnx.com/journal-article/20150118">https://journalnx.com/journal-article/20150118</a></p>

An increasing number of ratios in illness and injuries has lead to increase in the number of disabled people at the early age of birth. In this work we presented a web based e-learning system that is completely concerned with the sign language. In Pakistan, there is a scarcity of sign language resources available to assist deaf children in strengthening the language skills necessary for academic advancement. Thus, e-Learning for hearing impaired people is an application developed not only for deaf children but also for normal people to communicate with the deaf people. It is a web based application focusing on sign language of Pakistan which allows non-vocal information exchange. The purpose of this research work is twofold. First is to provide an e-learning system for educating sign language to deaf/hard-of-hearing people of Pakistan. Secondly, to develop a text translation system to translate English into its equivalent Pakistani sign language gestures in order to overcome communication barriers. The system also contains feedback for improving the quality of application by taking suggestions from normal and disable people. The proposed application is distributed in modular client server entities; the framework is specially designed for parents to edify their children the indispensable communication skills. We believe that proper communication lays the foundation for accelerated learning and reduced frustration.

Communication is very essential for each and every individual person, because it allows us to express our feelings, sentiments, and ideas etc. Humans can communicate with one another through a variety of means, including natural language, body language, and even writing. People who do not have any disabilities, such as hearing or speaking impairments, can use all of the foregoing modes of communication to communicate with other normal people. Natural language, which includes words, sentences, and so on, is the most often utilized type of communication by normal people. On the other hand, People with disabilities use Sign Language to communicate with others. Deaf and dumb people comprehend sign language easily and quickly, but it is difficult for them to communicate with normal people who do not understand sign language. Deaf and dumb persons are unable to effectively convey their feelings and thoughts to other normal people due to hearing and speaking issues, which is a serious problem all over the world. To address this issue, numerous systems were developed that interpret sign language (Hand movements, facial expression etc.), recognize signs (Hand Gestures), and display the relevant text on the system, allowing normal people to understand the language of the deaf and dumb. The fundamental purpose of this survey paper is to read all of the methods that were used in various systems and to identify the most efficient method that can be adopted in future research work.

In day-to-day life, deaf and dumb people are facing a lot of battles in speaking with typical individuals. Sign language plays a very important role for the deaf and people interact with the outside world. Due to the nonattendance of wonderful gesture-based communication interpreters still it leads the communication break. Powerful collaboration possible only if the deaf and dumb persons nearby sign language translator. To overcome this problematic circumstance in our approach utilizing smartphones for establishing active communication with normal people. The main effort of our proposed framework is to design smart mobile for helping the deaf and the dumb community to convey their thoughts anywhere they are located so it reduces the communication hole among normal, hearing, and speech-impaired people. Smartphones act as an interface to translate sign language to speech and vice versa. The various gestures-based signs captured automatically and translating sign language to speech. To process an image of gestures using several image processing techniques and deep learning models. Investigation result shows that this smartphone-based dynamic hand gesture recognition model provides an exact and effective system for the interaction between non-signer and signer.

Deaf people are one of those with disabilities who cannot speak and hear. Deaf people using sign language for communication who use sign language with hand gestures, body and convey facial expressions using sign language. One important component in sign language is the alphabetical finger or the manual alphabet needed to complete communication. The alphabet of the finger is done by spelling words on spoken language, by spelling letter by letter using a finger. This method is used to spell the name or mention the word. However, not everyone understands sign language, so tools needed to bridge communication between people who are deaf with normal people. One solution that will be offered is to use computer technology as a tool to recognize sign language. The technology is an automatic language translator system that process input images using by using the Convolutional Neural Network (CNN). On this research consist of 3 class such us A, assalamualaikum and hallo which get the accuracy respectively 100% for each class.

Sign language is the main communication language for deaf and dumb people. There have many difficulties for deaf and dumb people when they communicate with normal people. Only a few people can use Sign Language proficiently and most of the people don't know how to communicate the deaf people. From the point of view of computer vision, we can develop the sign language recognition for deaf and dumb people. Most of the sign language recognition systems are based on the static sign. In motion-based sign language recognition system, we proposed feature extraction method and recognition using machine learning. Every nation has its own sign language and our proposed system is based on Myanmar National Sign Language.

lagging in education as compared to their hearing peers. This can affect their communication skills and affect their academic achievement. However, the existing applications developed for sign language education have a few drawbacks, such as only consisting of a Learning Module, and the media used was only learning videos. Hence, the Mobile Learning for Manually Coded Malay Sign Language using Augmented Reality (KTBM AR) is developed to provide users with Learning and Activity Modules where users can interact with and test their knowledge. Not only that, but KTBM AR also included Augmented Reality (AR) as an interactive learning medium to engage users more in learning. The methodology used in developing this application is the ADDIE model, which comprises five main phases: analysis, design, development, implementation, and evaluation. The result from testing shows that the application was successfully developed with a range of 94% acceptable according to the System Usability Scale (SUS). Thus, it can be concluded that KTBM AR is suitable to be implemented in sign language learning.

The inability to communicate verbally is widely acknowledged as a significant disability. The primary objective of this research is to create a practical system aimed at individuals with hearing impairments, particularly those who depend on sign language as their primary means of communication. This system aims to enable deaf individuals to express themselves, communicate effectively and facilitate understanding of their language which would otherwise be challenging since most people are unfamiliar with sign language. By employing human gesture interpretation and motion capture, this technology can facilitate the translation of sign language into spoken language and vice versa. Despite the existence of various methods to convert sign language into voice, none of them currently provide an entirely intuitive user interface. Our objective is to create a system that not only translates sign language but also integrates a natural user interface, thus enhancing accessibility for individuals who are blind or have visual impairments. This system will achieve this by recognizing facial expressions and effectively conveying emotions behind words assisting visually impaired individuals in expressing themselves more effectively. Hence, in this project, we tentatively aim to build a system that can ease the lives of blind, deaf and dumb people to some extent. Normal communication like normal people do might never be possible for differently abled people but through our project, we try to provide them with a tool that can help them experience normalcy while communicating with normal people.

Communication is the best means to express ourselves to others. There are various ways to communicate with others like writing, vocally and signing with hands. Generally normal people prefer to communicate vocally. Deaf and dumb people use the sign language for interaction. Sign language is a language expressed with hands and facial expression without speaking. Communication between normal and impaired people is difficult as there is no common language. To establish communication, a human sign language interpreter is required. To build a communication bridge between normal and impaired people, a sign language interpretation system can be used. Using this system one can translate sign words to text. In this paper real time conversion of word level sign language to English is presented. For the training of models, a user defined dataset is created and labeled. Three different models of Tensorflow are used for Sign Language interpretation SSD 320, SSD 640 and Centernet Resnet50. SSD 640 is more preferable as compared to others as it can detect sign words in less time and accurately.

Real time Indian sign language recognition system