A Vision-Based System for Automated Sign Language Teaching

This research aims to uncover higher education students’ experiences regarding sign language learning with online learning platforms and learning experiences during the coronavirus COVID-19 pandemic in one of the teacher-preparation programs in the Kingdom of Saudi Arabia (KSA) as learning the basics of sign language is one of the essential elements in the teacher-preparation program for future teachers to be able to interact with their deaf students. Qualitative research method, structured individual interviews were employed to collect data to understand students’ experiences, knowledge, expectations, and thoughts of the impact of the COVID-19 crisis and the sudden and unplanned movement to online teaching tools on their sign language learning and practices. A total of 24 female students enrolled in a special education bachelor program were interviewed in this study. Fifteen participants thought that the COVID-19 crisis influenced their sign language learning and practices, while six of the interviewees reported no impact, and three were not sure. Data revealed some disadvantages, difficulties, and concerns of learning sign language through e-learning tools and some advantages of sign language learning using e-learning tools during the COVID-19 pandemic and are presented alongside examples of the interviewees’ statements. This study presented some practice recommendations like integrating digital game-based learning, web and phone applications to practice sign language to endorse effective and innovative sign language learning and practices techniques.

The purpose of this study was to investigate the potential of a tangible game, Inclusive Glossary of Mathematical Terms (GIM), to support the teaching of math and Portuguese sign language (Língua Gestual Portuguesa; LGP) to preschool and first-grade children, with an emphasis on inclusive education, knowledge acquisition, and engagement. GIM is composed of physical cards that are complemented by digital videos that run on an interface designed for this game. The game consists of two sets of cards and respective animations. As an inclusive math glossary, it also includes videos where concepts, characters, and objects are identified through images, words, and sign language. During the development process of GIM, it was possible to involve a multidisciplinary team, of which a Deaf sign language teacher and a sign language interpreter were part to ensure the clarity of the sign language and refine the written sentences accompanying the glossary. In order to validate its contents, a focus group of preschool and primary teachers specialized in the teaching of Deaf and Hard of Hearing (DHH) children was formed. A total of 120 children, aged between three and eight years old (M = 5.61; SD = 0.78), and seven LGP and special education teachers from three Portuguese schools participated in the study. The children used GIM in their classrooms, with the assistance of the participating teachers, who were subsequently interviewed to assess their engagement, interest, and learning outcomes. The interviews were analyzed using content analysis procedures, including descriptive statistics and Pearson correlations. The results indicated that GIM has the potential to effectively support the teaching of mathematics and LGP to preschool and first-grade children, especially in inclusive education contexts. The tangible nature of the game facilitated children's understanding of mathematical concepts and LGP, leading to improved involvement in the teaching-learning process. Moreover, the game promoted awareness of diversity and the principles of inclusive education, making it a valuable tool for promoting social inclusion and understanding. However, the study has reduced sample and context-specific limitations, suggesting a need for further research in this area.

Sign language is the communication language used by people with disabilities, especially deaf and hearing-impaired people. The communication between normal and disabled people using sign language will help them carry out their daily activities. Unfortunately, normal people are not aware of the importance of sign language because they are not directly dealing with disabled persons. Besides, some normal people found that sign language is difficult to learn. This study focuses on developing a mobile application for sign language with augmented reality features. This application is targeted at normal people as a sign language learning tool, and augmented reality will make the learning process effective and exciting. The methodology for this study is the ADDIE model that consists of five phases, namely Analysis, Design, Development, Implementation, and Evaluation. This application will assist the users in learning sign language interactively and interestingly. The evaluation of the application was done by the expert in the related background and the normal people. The usability test result revealed that the sign language application is usable for normal people to know and learn the basic sign language. In conclusion, the sign language application is an interesting application for normal people to use and learn sign language.

Sign language communication has become a permanent remedy for those who are deaf or hard of hearing. It is a vital asset for people who cannot express their opinions and feelings to the rest of the world. It smooths out and simplifies the combination cycle between them and others. In any event, simply inventing gesture-based communication isn’t enough. This gift comes with a slew of surprises. The hand signs are frequently mixed up and misinterpreted by someone who has never known or learned sign language, and some people may know it but use it in a different language. In any case, the long-standing correspondence gap can now be narrowed thanks to the discovery of several ways for automating the recognition of sign signals. In this review, the client should be able to take photographs or video input of the hand signals using a camera, and the framework should anticipate the name of the hand sign from the captured picture.

In the rapidly evolving digital learning landscape, sign language mobile apps are vital in advancing signlanguage teaching. However, ensuring the quality of these apps remains a critical challenge. To addressthis gap, this study employs the fuzzy Delphi technique to establish a robust set of criteria for evaluatingthe quality of sign language mobile apps. By leveraging the collective wisdom and expertise of a panelof experts, the fuzzy Delphi technique facilitates a structured process for achieving consensus on theessential factors contributing to evaluating sign language mobile apps. Through rigorous rounds ofiterative feedback and analysis, the study identifies a comprehensive list of reliable criteriaencompassing various dimensions, including functionality, usability, accessibility, and pedagogicaleffectiveness. The criteria established through this method serve as a valuable resource for developers,educators, and clients in selecting and developing top-notch sign language mobile apps. Developers canuse the criteria as a guide during the design and development stages, ensuring that their apps meet thehighest quality and user experience standards. Educators can rely on the criteria as a checklist forevaluating and selecting appropriate apps that align with their teaching objectives and cater to thediverse learning needs of their students. Clients, such as educational institutions or individuals seekingsign language learning resources, can make informed decisions by referring to the established criteria,promoting the adoption of clear and impactful sign language mobile apps. This study emphasizes thesignificance of applying the fuzzy Delphi method in the context of sign language mobile appassessment. Involving experts from relevant fields ensures that the established criteria capture themultifaceted nature of compelling sign language learning experiences. Developing a comprehensiveand reliable set of criteria contributes to improving existing apps and encourages innovation in creatingnew apps that better serve the needs of sign language learners. Overall, this research extends theknowledge base of sign language teaching in the digital age by providing a robust framework forassessing the quality of sign language mobile apps. The findings of this study empower stakeholders inthe education and technology sectors to make informed decisions, fostering the advancement of signlanguage teaching and promoting inclusivity in digital learning environments.

Sign Language Recognition by using Spatio-Temporal Features

Deaf people are one of those with disabilities who cannot speak and hear. Deaf people using sign language for communication who use sign language with hand gestures, body and convey facial expressions using sign language. One important component in sign language is the alphabetical finger or the manual alphabet needed to complete communication. The alphabet of the finger is done by spelling words on spoken language, by spelling letter by letter using a finger. This method is used to spell the name or mention the word. However, not everyone understands sign language, so tools needed to bridge communication between people who are deaf with normal people. One solution that will be offered is to use computer technology as a tool to recognize sign language. The technology is an automatic language translator system that process input images using by using the Convolutional Neural Network (CNN). On this research consist of 3 class such us A, assalamualaikum and hallo which get the accuracy respectively 100% for each class.

Bu çalışmanın amacı, iletişim ortamı olarak geliştirilen prototip İDÇS (İşaret Dili Çeviri Sistemi)’nin iletişim sürecini ve akademik başarıyı destekleme potansiyelini belirlemektir. Çalışma nitel araştırma yöntemlerinden biri olan durum çalışması ile yürütülmüştür. Araştırmanın çalışma grubunu işitme engelliler öğretmenliğinden mezun özel eğitim öğretmenleri, uzman öğreticiler ve işaret dili tercümanları olmak üzere 11 uzman oluşturmaktadır. Geliştirilen sistemle ilgili uzman görüşleri alınmış ve toplanan veriler analiz edilmiştir. Elde edilen veriler iletişim süreçleri ve akademik başarı olmak üzere iki tema altında toplanmıştır. İletişim süreçlerine yönelik bulgular incelendiğinde katılımcılar tarafından İDÇS’nin en çok iletişim performansını ve sosyalleşmeyi arttırma potansiyelinin olduğu düşünülmektedir. Bununla birlikte İDÇS’nin başlangıç seviyesinde öğrenme, kelime dağarcığını artırma potansiyelinin olduğu da katılımcılar tarafından ifade edilmiştir. İDÇS’nin ders çalışma süreçlerine etki potansiyelinin bulunması da elde edilen diğer bulgular arasındadır. İDÇS’nin iletişim performansını, sosyalleşmeyi, kelime dağarcığını, akademik başarıyı, başlangıç seviyesinde işaret dili öğrenmeyi ve işaret dili öğrenmeye ilgiyi arttırma potansiyelinin olduğuna ilişkin görüşler bildirilmiştir. İşitme yetisinde oluşturabileceği tembelliğin soyutlanmaya yol açma ihtimalinin bulunduğu da elde edilen sonuçlar arasındadır.

This study chronicles our efforts in exploring challenges faced in traditional sign language teaching methods and proposes AI- driven solutions that effectively address these issues. We place this study in the framework of Technology Acceptance Model (TAM). This experimental study was carried out with 40 undergraduate university students in Bambili – North West Region of Cameroon.An experimental research design was employed involvingreal-time training using Neural Network (DenseNet 201 Model) online tool. Observation sessions and 40 questionnaires were used as instruments for data collection. Findings indicate that: 1.Traditional teaching methodscharacterized: Passive learning, heavy reliance on the teacher as the sole source of knowledge, inadequate opportunities for practical applications, rote memorization and repetition of gestures hindered smooth teaching and learning.2.The use of AI- driven tools: Video demonstrations for sign recognition includeddatasets of 100 images,26 letters of manual alphabets, 30 numbers, 50 fingerspelledwords, 40 common phrases and 25 basic signs, capturing handshapes and gestures, angles, positions, and proximity enhanced students learning skills.By adopting these modern teaching methods with AI tools, teachers create a learning environment that promotes active engagement, practical application and effective communication skills in sign language. The study recommends that Governments, educational institutions, and organizations should promote sign language and challenge negative stereotypes by encouraging innovative technologies that foster the teaching and learning of sign language.

deaf individuals rely heavily on one another's use of sign language as a means of communication. In this era, the significance of accessibility has become more and more important. However, for most people with normal hearing, learning sign language is a very difficult thing. Many research teams around the world are using Deep Learning to develop translators for sign language recognition. However, there is a lack of larger and more semantically rich sign language datasets. In this article, we propose a new Chinese sign language dataset- 109 unique sentences from 50 sign language signers and 27,250 clips. On the new Chinese sign language dataset, we further propose a sequence-to-sequence deep learning approach in order to demonstrate how deep learning may continually lower the communication barriers that exist between persons who are deaf and those who have normal hearing.

In spite of the fastest growing of technologies of information and communication in the education and the development of very sophisticated environments to improve learning and education, there are a few tools dedicated to deaf education due to the difficulty to create content in sign language. In this context, we present in this paper an ICT environment we developed to aid deaf people improving their social integration and communication capabilities. Our environment is a specialized LCS that generates multimedia courses to teach and learn sign language. These courses can be used either by deaf pupils to learn (or e-learn) sign language or also by hearing people to be able to communicate with deaf people. This educational environment uses mainly a Web-based interpreter of sign language developed in our research laboratory and called Websign. It is a tool that permits to interpret automatically written texts in visual-gestured-spatial language using avatar technology.

The act of sharing or exchanging data, concepts, or emotions is known as communication. Both parties must be able to speak and comprehend the same language in order for communication to be established. It can be quite challenging for people with speech and hearing problems to communicate with the broader public. The way they communicate is through sign language. To communicate with the outside world, they are dependent on interpreters who use sign language. Being a minority, most people are unaware of the sign language they employ. Sign language communication is always necessary. Without an interpreter, communication might be challenging. One can create a machine learning-based model to get around this obstacle. It is possible to train a model to translate various sign language motions into English by recognizing them. Many individuals will find this useful while interacting and chatting with the deaf and ignorant.

소리가 잘 들리지 않는 청각장애인은 소리를 표시하는 글자를 익히고 복잡한 개념을 전달하는 글을 이해하는 데 어려움이 많다. 그래서 자연스럽게 표정, 몸짓, 손짓으로 의사를 전달하는 수화가 청각장애인들의 주요 의사소통수단으로 자리잡아왔지만 사회에서는 글과 말이 주요 정보전달 수단으로 이용되고 있어 청각장애인이 정보에 접근하고 지식을 넓혀 직업을 갖는데 큰 어려움이 있다. 특히 근래에 인터넷이 보편화됨에 따라 정상인의 정보 습득량은 크게 늘었지만, 글로 표현된 인터넷을 이해하기 어려운 청각장인들은 인터넷 정보에 접근하는데 어려움이 많다. 본 연구에서는 청각장애인을 위한 수화 웹페이지를 제작할 수 있는 수화 웹페이지 제작시스템을 개발하였다. 수화 웹페이지 제작 시스템은 청각장애인용 수화 웹페이지 제작자가 통상적인 인터넷 사용에 필요한 하드웨어 및 소프트웨어 환경만 갖추면 운영할 수 있도록 웹기반으로 개발하였다. 수화용 웹 페이지 제작자는 게시판 형태의 인터페이스를 이용하여 변역하고자하는 문장을 서버에 전송할 수 있다. 서버는 수신된 문장을 수화로 번역하여 3D 아바타 기반의 수화 그래픽 애니메이션을 랜더링한다. 수화 그래픽 애니메이션은 MP4 형식의 동영상으로 변환되며, 스트리밍 서버의 저장소에 보관된다. 저장된 수화 동영상의 이름과 주소는 수화 웹페이지 제작자가 처음 문장을 입력하였던 게시판에 표시되도록 하여 향후 청각장애인용 웹 페이지를 제작할 때 활용할 수 있도록 하였다. 또한 본 연구에서는 웹페이지 제작자가 공공기관의 웹페이지를 제작할 수 있도록, 이들 기관에서 현재 사용된 글들을 수화로 번역할 때 필요한 수화 단어를 수화 웹페이지 제작 시스템의 수화 사전에 추가하였다. 수화 단어를 추가함으로써 웹기반 수화 웹페이지 제작 시스템이 공공부문에 보다 더 많이 활용될 수 있을 것이다. Hearing-impaired people have difficulty in hearing, so it is also hard for them to learn letters that represent sound and text that conveys complex and abstract concepts. Therefore it has been natural choice for the hearing-impaired people to use sign language for communication, which employes facial expression, and hands and body motion. However, the major communication methods in daily life are text and speech, which are big obstacles for the hearing-impaired people to access information, to learn and make intellectual activities, and to get jobs. As delivering information via internet become common the hearing-impaired people are experiencing more difficulty in accessing information since internet represents information mostly in text forms. This intensifies unbalance of information accessibility. This paper reports web-based text-to-sign language translating system that helps web designer to use sign language in web page design. Since the system is web-based, if web designers are equipped with common computing environment for internet browsing, they can use the system. The web-based text-to-sign language system takes the format of bulletin board as user interface. When web designers write paragraphs and post them through the bulletin board to the translating server, the server translates the incoming text to sign language, animates with 3D avatar and records the animation in a MP4 file. The file addresses are fetched by the bulletin board and it enables web designers embed the translated sign language file into their web pages by using HTML5 or Javascript. Also we analyzed text used by web pages of public services, then figured out new words to the translating system, and added to improve translation. This addition is expected to encourage wide and easy acceptance of web pages for hearing-impaired people to public services.

Communication is fundamental to human interaction, yet individuals who are deaf or speech-impaired often face barriers due to limited accessibility to sign language resources. This study presents the development of a mobile sign language dictionary designed to enhance communication and learning. Traditional sign language dictionaries, whether in printed or video formats, present challenges in accessibility, usability, and portability. In contrast, a mobile application offers an interactive, efficient, and user-friendly solution. The proposed application integrates gesture recognition, video demonstrations, and text-based explanations to facilitate sign language learning. By leveraging artificial intelligence (AI) and cloud-based storage, the app ensures accurate sign recognition and provides a comprehensive database of gestures, including letters, numbers, root words, prefixes, suffixes, and commonly used phrases. A structured methodology involving requirement analysis, data collection, design, development, testing, and revision was applied to develop a functional and accessible tool. User testing demonstrated high engagement and learning effectiveness, with 85% of participants reporting increased comfort with mobile learning tools, 82.5% finding the app beneficial for sign language acquisition, and 90% improving their familiarity with sign words. These results confirm that a mobile sign language dictionary is an efficient educational tool that fosters inclusivity and communication accessibility. Future enhancements, including database expansion, additional sign language dialects, and interactive exercises, can further improve the application’s impact.

Wearable triboelectric pressure sensors hold tremendous promise in sign language recognition. Nevertheless, the poor stability and low output are unfavorable to acquire accurate and stable electrical signals for wearable sensors. Herein, we have prepared a waterproof and stable MoBT x /PDMS‐based triboelectric pressure sensor with a controllable dielectric constant. This pressure sensor illustrates high output performance (144 V, 2800 nA, and 46 nC), high stability (5000 cycles stretching, 5000 cycles bending, and 5000 cycles after washing), and long‐term robustness (60 days). The sensor arrays with 7 channels consisting of step‐type and stripe‐type pressure sensors, can generate distinctive responses corresponding to specific hand gestures, contributing to a higher resolution of sign language letters. Integrated with a microcontroller, this system achieves precise gesture recognition and effective conversion of sign language movements into an audio signal for real‐time communication. This work offers a robust strategy toward reliable, high‐resolution, and real‐time sign language communication, advancing assistive technologies for individuals with disabilities.