A survey on Sign Language machine translation

BackgroundApproximately 235,000 deaf and hard of hearing (DHH) people live in Germany. Due to communication barriers, medical care for this group is difficult in many respects. Especially in the case of acute illnesses, the possibilities of communication, e.g., through sign language interpreters, are limited. This study investigates the satisfaction of DHH patients with medical care in Germany in unplanned medical consultations. The aim of this study is to provide insights into DHH patient's perception of medical care, to identify barriers and avoidance behaviours that stem from fears, miscommunication, and prior experiences.MethodsWe obtained data from adult DHH participants between February and April 2022 throughout Germany via an online survey in German Sign Language. The responses of N = 383 participants (65% female, M = 44 years, SD = 12.70 years) were included in statistical analyses. Outcomes were convictions of receiving help, satisfaction with healthcare provision, and avoiding healthcare visits; further variables were concerns during healthcare visits, incidences of miscommunication, and a communication score. We calculated t-tests, ANOVAs, correlations, and linear and logistic regression analyses.ResultsOur main findings show that (1) DHH patients were unsatisfied with provided healthcare (M = 3.88; SD = 2.34; range 0–10); (2) DHH patients reported many concerns primarily about communication and treatment aspects when visiting a doctor; and (3) 57% of participants deliberately avoided doctor visits even though they experienced symptoms. Factors such as concerns during doctor’s visits (B = -0.18; 95%CI: -0.34--0.02; p = .027) or miscommunication with medical staff (B = -0.19; 95%CI: -0.33–0.06; p = .006) were associated with satisfaction with medical care, while we found almost no associations with gender and location, and only few with age and education.ConclusionsOverall, our findings suggest that DHH patients are unsatisfied with provided healthcare, they deliberately avoid doctor visits, and they face various communication barriers. This study revealed several communication-related determinants of satisfaction with healthcare in DHH patients, such as incidences of miscommunication and the communication score. Communication-related barriers have high potential to be addressed in collaboration with the DHH community. To improve the medical care and the satisfaction with healthcare in DHH patients, training healthcare professionals, digital technologies, and other communication-enhancing interventions should be explored in future intervention studies.

The Shape of Air: American Sign Language as Narrative Prosthesis in 21st Century North American Media

Hearing Impaired individuals routinely encounter limitations in their involvement in social interactions, access to intriguing information, and participation in everyday activities, among various other aspects. However, the hardest part of their interactions with regular people is communication, because sign language is the primary language of those who are hearing impaired. However, the general public is unaware of sign language. Each country has its own sign language. However, there are some striking similarities between them. In Sri Lanka, hearing impaired people use Sri Lankan Sign Language (SLSL) as their communication language. There is several research done on Sign Language recognition and translation. But no fully functioning system is utilized for Sri Lankan Sign Language translation. To find the gap in this area, we conducted a systematic literature review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method that analyses 12 studies on Sign Language Translation (SLT). As per the literature review, Image Processing (IP) and Convolutional Neural Networks (CNN) are the most used techniques for Sign Language translation. But these methods have limitations: not enough data, differences in how people use sign language, difficulty in translating in real-time, not capturing cultural aspects, needing specific equipment, and understanding the context of conversations. Recognizing and solving these problems is important, especially for languages like SLSL. Future research should focus on getting more data, making translation work for different cultures, and improving real-time translation. This will help hearing impaired people communicate better with others.

The purpose of this study was to investigate the potential of a tangible game, Inclusive Glossary of Mathematical Terms (GIM), to support the teaching of math and Portuguese sign language (Língua Gestual Portuguesa; LGP) to preschool and first-grade children, with an emphasis on inclusive education, knowledge acquisition, and engagement. GIM is composed of physical cards that are complemented by digital videos that run on an interface designed for this game. The game consists of two sets of cards and respective animations. As an inclusive math glossary, it also includes videos where concepts, characters, and objects are identified through images, words, and sign language. During the development process of GIM, it was possible to involve a multidisciplinary team, of which a Deaf sign language teacher and a sign language interpreter were part to ensure the clarity of the sign language and refine the written sentences accompanying the glossary. In order to validate its contents, a focus group of preschool and primary teachers specialized in the teaching of Deaf and Hard of Hearing (DHH) children was formed. A total of 120 children, aged between three and eight years old (M = 5.61; SD = 0.78), and seven LGP and special education teachers from three Portuguese schools participated in the study. The children used GIM in their classrooms, with the assistance of the participating teachers, who were subsequently interviewed to assess their engagement, interest, and learning outcomes. The interviews were analyzed using content analysis procedures, including descriptive statistics and Pearson correlations. The results indicated that GIM has the potential to effectively support the teaching of mathematics and LGP to preschool and first-grade children, especially in inclusive education contexts. The tangible nature of the game facilitated children's understanding of mathematical concepts and LGP, leading to improved involvement in the teaching-learning process. Moreover, the game promoted awareness of diversity and the principles of inclusive education, making it a valuable tool for promoting social inclusion and understanding. However, the study has reduced sample and context-specific limitations, suggesting a need for further research in this area.

This chapter describes a project to introduce co-enrollment practices for deaf and hard-of-hearing (DHH) pupils in a Japanese regular primary school. This project consisted of three parts: (1) instruction of sign language to the DHH pupils, (2) instruction of sign language to the teachers and hearing pupils, and (3) sign language interpretation in regular classrooms. The focus was on the challenges of sign language interpretation. The results showed that the multiple flows of information and overhearing others’ speech in the classrooms provided challenging conditions for sign language interpretation. New ways of communication and interaction (actually introducing a new culture) and the presence of deaf adults were needed to establish full participation by DHH pupils in the co-enrollment classrooms. The possible future of practices concerning the inclusion of DHH children is discussed.

The typical medical education curriculum does not address language development for deaf and hard-of-hearing (DHH) children. However, this issue is medical because of the frequency with which DHH children as a population face health complications due to linguistic deprivation. The critical period for language development is early; if a child does not acquire an intact language before age five, the child is unlikely to ever have native-like use of any language. Such linguistic deprivation carries risks of cognitive delay and psycho-social health difficulties. Spoken language is inaccessible for many DHH children despite assistive-technology developments. But sign languages, because they are visual, are accessible to most DHH children. To ensure language development, DHH children should have exposure to a sign language in their early years, starting at birth. If they also receive successful training in processing and producing a spoken language, they will have the many benefits of bimodal bilingualism. Undergraduate medical education curricula should include information about early language acquisition so that physicians can advise families of deaf newborns and newly deafened young children how to protect their cognitive health. Graduate medical education in primary care, pediatrics, and otolaryngology should include extensive information about amplification/cochlear implants, language modality, and the latest research/practices to promote the development and education of DHH children. Training in how to establish connections with local authorities and services that can support parents and child should be included as well. Further, students need to learn how to work with sign language interpreters in caring for DHH patients. We offer suggestions as to how medical curricula can be appropriately enriched and point to existing programs and initiatives that can serve as resources.

Abstract: The number of deaf and hard-of-hearing (DHH) children who enroll in mainstream schools has increased in recent years. In the present paper, we discuss with the co-enrollment program for inclusive DHH pupils in the United States. Through observation of classroom activities, I attempt to ethnographically clarify what happens in co-enrollment classrooms. In particular, I focus on DHH pupil’s participation in classroom’s activities, describing the interaction between hearing and DHH pupils, the collaboration between regular teachers and teachers for DHH, and the relationships between signed and spoken languages. Two types of information flow were found in classroom discourses: (i) formal, single-track flow, such as the teacher talking to the whole class, and the teacher asking questions and the pupils responding to it; (ii) informal, multiple-track flow, such as the pupils murmuring spontaneously and simultaneously, and talking with each other locally. While the sign language interpreters mainly supported the first type of flow of discourse, the direct interactions between DHH and hearing pupils were found to necessary for the second type of information flow. Finally, I discuss how the learning would be constructed for DHH pupils in inclusive classrooms from a socio-cultural point of view. Key words: education for deaf and hard-of-hearing, co-enrollment, inclusive education, signed language, classroom observation

Abstract: The number of deaf and hard-of-hearing (DHH) children who enroll in mainstream schools has increased in recent years. In the present paper, we discuss with the co-enrollment program for inclusive DHH pupils in the United States. Through observation of classroom activities, I attempt to ethnographically clarify what happens in co-enrollment classrooms. In particular, I focus on DHH pupil’s participation in classroom’s activities, describing the interaction between hearing and DHH pupils, the collaboration between regular teachers and teachers for DHH, and the relationships between signed and spoken languages. Two types of information flow were found in classroom discourses: (i) formal, single-track flow, such as the teacher talking to the whole class, and the teacher asking questions and the pupils responding to it; (ii) informal, multiple-track flow, such as the pupils murmuring spontaneously and simultaneously, and talking with each other locally. While the sign language interpreters mainly supported the first type of flow of discourse, the direct interactions between DHH and hearing pupils were found to necessary for the second type of information flow. Finally, I discuss how the learning would be constructed for DHH pupils in inclusive classrooms from a socio-cultural point of view. Key words: education for deaf and hard-of-hearing, co-enrollment, inclusive education, signed language, classroom observation

Deaf and hard of hearing (DHH) individuals who use signed language and those who use spoken language face different challenges and stressors. Accordingly, the profile of their mental problems may also differ. However, studies of mental disorders in this population have seldom differentiated between linguistic groups. Our study compares demographics, mental disorders, and levels of distress and functioning in 40 patients using Norwegian Sign Language (NSL) and 36 patients using spoken language. Assessment instruments were translated into NSL. More signers were deaf than hard of hearing, did not share a common language with their childhood caregivers, and had attended schools for DHH children. More Norwegian-speaking than signing patients reported medical comorbidity, whereas the distribution of mental disorders, symptoms of anxiety and depression, and daily functioning did not differ significantly. Somatic complaints and greater perceived social isolation indicate higher stress levels in DHH patients using spoken language than in those using sign language. Therefore, preventive interventions are necessary, as well as larger epidemiological and clinical studies concerning the mental health of all language groups within the DHH population.

Introduction In this paper, I examine the practices of representing Deaf ‘voices’’ to hearing audiences in two recent US television crime dramas. More literally I look at how American Sign Language is framed and made visible on the screen through various production decisions. Drawing examples from an episode of CSI: New York that aired in December 2006 and an episode of Law and Order: Criminal Intent that aired in April 2007, I examine how the practices of filming Deaf people and the use of American Sign Language intersect with the production of a Deaf ‘voice’ on the screen. The problem of representing a Deaf ‘voice’ on the screen is akin to the problem of representing other minority languages. Film and television producers in the United States have to make choices about whether the majority audience of English speakers will have access to the minority language or not. In the face of this dilemma media producers have taken several approaches: subtitling foreign speech, translating foreign speech through other characters, or leaving the language inaccessible except to those who use it. The additional difficulty with representing national sign languages is that both the language and the recording medium are visual. Sometimes, filmmakers make the choice of leaving some portions of the signed dialogue inaccessible to a non-signing hearing audience. On the one hand this choice could indicate a devaluing of the signed communication, as its specific content is considered irrelevant to the plot. On the other hand it could indicate that Deaf people have a right to be visible on television using their own language without accommodating hearing people. A number of choices made in the filming and editing can subtly undermine positive representations of Deaf ‘voices’ particularly to a Deaf audience. These choices often construct an image of sign languages as objectified, exoticised, disjointed, incomplete, or a code for spoken language. Simple choices such as using simultaneous speaking and signing by Deaf characters, cropping the scene, translating or not translating the dialogue have powerful implications for the ways that Deaf ‘voices’ are becoming more visible in the 21st century. Typical filming and editing conventions effectively silence the Deaf ‘voice.’ Over 20 years ago, in the comprehensive book, Hollywood Speaks: Deafness and the Film Entertainment Industry (1988), Schuchman’s complaint that the filming and editing techniques of the day often did not attend to preserving the visibility and comprehensibility of sign language eon the screen, still applies today. As editing techniques have evolved over the years, fr om reliance on wide and medium shots to frequent intercutting of close-ups, the tendency to cut sign language off the screen, and out of the comprehensible view of the audience, may have even increased. Recent Portrayals of Deaf People on Television During one television season in the United States between August 2006 and April 2007, 30 episodes of six different serial television programs portrayed signing Deaf characters. Three of these programs had on-going Deaf characters that appeared in a number of episodes throughout the season, while three other programs portrayed Deaf people in a one-off episode with a Deaf theme. Initial air date for the season Program and Season # of Episodes 1 14 Aug. 2006 Weeds, Season 2 5 2 20 Sep. 2006 Jericho, Season 1 13 3 28 Jan. 2007 The L Word, Season 4 9 Table 1. Dramas with Ongoing Deaf Characters during the 2006-2007 USA Television Season Initial air date Program, Season, Episode Episode Title 1 13 Dec. 2006 CSI: New York, Season 3, Episode 12 “Silent Night” 2 3 Apr. 2007 Law and Order: Criminal Intent, Season 6, Episode 18 "Silencer" 3 12 Apr. 2007 Scrubs, Season 6, Episode 16 “My Words of Wisdom” Table 2. One-off Episodes with Signing Deaf Characters during the 2006-2007 USA Television Seasons Ironically, although the shows with ongoing characters sometimes allow the Deafness of the character to be incidental to the character, it is only the one-off crime dramas that show Deaf people relating with one another as members of a vibrant community and culture based in sign language. Often, in the ongoing series, the characters remain isolated from the Deaf community and their interactions with other Deaf people are sparse or non-existent. For example, out of the 27 episodes with an ongoing Deaf character only two episodes of The L-Word have more than one Deaf character portrayed. In both Weeds and The L-Word the Deaf character is the love interest of one of the hearing characters, while in Jericho, the Deaf character is the sister of one of the main hearing characters. In these episodes though some of realities about Deaf people’s lives are touched on as they relate to the hearing characters, the reality of signing Deaf people’s social lives in the Deaf community is left absent and they are depicted primarily interacting with hearing people. The two episodes, from CSI: New York, and Law and Order: Criminal Intent, focus on the controversial theme of cochlear implants in the Deaf community. Though it is true that generally the signing Deaf community in the U.S.A. sees cochlear implants as a threat to their community, there is no record of this controversy ever motivating violent criminal acts or murder as portrayed in these episodes. In the episode of CSI: New York entitled “Silent Night” a conflict between a young Deaf man and Deaf woman who were formerly romantically involved is portrayed. The murdered young woman who comes from a Deaf family does not want her Deaf baby to have a cochlear implant while the killer ex-boyfriend who has a cochlear implant believes that it is the best option for his child. The woman’s Deaf parents are involved in the investigation. The episode of Law and Order: Criminal Intent, entitled “Silencer,” is also ultimately about a conflict between a Deaf man and a Deaf woman over cochlear implants. In the end, it is revealed that the Deaf woman is exploring the possibility of a cochlear implant. Her boyfriend projecting the past hurt of his hearing sister leaving him behind to go off and live her own life, doesn’t want his girlfriend to leave him once she gains more hearing. So he shoots the cochlear implant surgeon in the hand to prevent him from being able to perform the surgery. Then he accidentally kills him by crushing his voice box to prevent him from screaming. Analyzing Two Crime Dramas In both television dramas, the filmmakers use both sound and video editing techniques to mark the experiential difference between hearing and Deaf characters. In comparing the two dramas two techniques are evident : muting/distorting sounds and extreme close-ups on lips talking or hands signing. Though these techniques may heighten awareness of deaf experience to a non-signing audience they also point to a disabling stereotyping of the experience of being Deaf as lacking — framing their experience as hearing loss rather than Deaf gain (Bauman & Murray; Shakespeare 199). By objectifying sign language through extreme close ups American Sign Language is portrayed as something strange and unusual that separates Deaf signers from hearing speakers. The auditory silences can either jolt the hearing non-signer into awareness of the sensory aspect of sound that is missing or it can jolt them into awareness of the visual world that they often don’t really see. In the opening few scenes of the episodes both CSI: New York and Law and Order: Criminal Intent use sound editing alternately muting or distorting sounds as they cut between a ‘deaf’ auditory perspective and a ‘hearing’ perspective on the action as it unfolds. Even though the sound editing does play a part in the portrayal of Deaf people’s experience as lacking sound, the more important aspects of film production to attend to are the visual aspects where Deaf people are seen authentically signing in their own language. Scene Analysis Methodology In taking a closer look at a scene from each episode we can see exactly how the filming and editing techniques work to create an image of sign language. I have chosen comparable scenes where a Deaf individual is interviewed or interrogated by the police using a sign language interpreter. In each scene it can be assumed that all the communication is happening in both English and ASL through an interpreter, so at all times some signing should be occurring. In transcribing the scenes, I noted each point when the editor spliced different camera shots adjacent to each other. Because of the different visual aesthetics in each program where one relied heavily on continuous panning shots, I also noted where the camera shifted focus from one character to another marking the duration of screen time for each character. This allowed for a better comparison between the two programs. In my transcripts, I included both glosses of the ASL signs visible on the screen as well as the flow of the spoken English on the audio track. This enabled me to count how many separate shifts in character screen time segments contained signing and how much of these contained completely visible signing in medium shots. CSI:NY Witness Interview Scene In the first signing scene, Gina (played by Marlee Matlin) is brought in for an interview with Detective Taylor and a uniformed officer interpreter. The scene opens with a medium shot on Detective Taylor as he asks her, “What do you think woke you up?” The shot cuts to an extreme close up of her face and hands and pans to only the hands

This paper presents the development of a real time American Sign Language (ASL) detection system using Raspberry Pi and a Sequential Convolutional Neural Network (CNN) model. The system aims to bridge the communication gap between the Deaf and Hard of Hearing (DHH) community and the hearing population by translating ASL gestures into text and audio outputs. The proposed system leverages the Raspberry Pi 5, a cost effective and scalable hardware platform, combined with deep learning techniques to achieve high accuracy in gesture recognition. The system utilizes the custom dataset for training and employs hand landmark detection using MediaPipe for real-time gesture analysis. The results demonstrate an 85% accuracy in recognizing ASL gestures, with real-time text and audio outputs. The system is designed for personal, educational, and public applications, offering a practical solution for enhancing communication accessibility for the DHH community. Key Words: American Sign Language(ASL), Raspberry Pi, Machine Learning, Gesture recognition, Sign Language Translation, Text-to-Speech conversion

Sign language is the only way for deaf and mute people to represent their needs and feelings. Most of non-deaf-mute people do not understand sign language, which leads to many difficulties for deaf-mutes' communication in their social life. Sign language interpretation systems and applications get a lot of attention in the recent years. In this paper, we review sign language recognition and interpretation studies based on machine learning, image processing, artificial intelligence, and animation tools. The two reverse processes for sign language interpretation are illustrated. This study discusses the recent research on sign language translation to text and speech with the help of hand gestures, facial expressions interpretation, and lip reading. Also, state of the art in speech to sign language translation is discussed. In addition, some of the popular and highly rated Android and Apple mobile applications that facilitate disabled people communication are presented. This paper clarifies and highlights the recent research and real used applications for deaf-mute people help. This paper tries to provide a link between research proposals and real applications. This link can help covering any gap or non-handled functionalities in the real used applications. Based on our study, we introduce a proposal involves set of functionalities/options that separately introduced and discussed by the recent research studies. These recent research directions should be integrated for achieving more real help. Also, a set of non-addressed research directions are suggested for future focus.

Nowadays, web technologies are a very efficient way to ensure communication between a large and heterogeneous audience. Furthermore, web information is mainly based on textual and multimedia content and consequently, some people with special needs, such as deaf and hard of hearing people, have difficulties to access to information or to communicate with hearing people. This problem is due to the lack of services that facilitate sign language learning for hearing people or text translation into sign language for persons with hearing impairment. In this context, we present in this paper a new approach based on web services, X3D and android operating system to build a mobile translation system from text into sign language using virtual signing agent. The main feature of this work is that it can be used to learn sign language and to provide sign language translation of written text for people with hearing impairment.

The study aims to promote a design for deaf and hard of hearing (DHH) people to communicate fluently and in a good social manner through an assistive technology. Sixteen semi-interviews for DHH people were conducted. Findings were combined with prior literature to develop a new prototype. “IKnow”, which is an augmented reality glass, could translate between voice and text to help target users communicate with people without hearing disabilities fluently. Different from existing devices, “IKnow” relies on visual technology instead of audio one to help DHH people. Besides, devices focus on how to deal with manner and fluency problem in communication. To improve the design, future study should focus on the place of interface to ensure the activity of DHH people will not be affected by wearable devices.KeywordsHearing impairedHCIProduct designDisabilityAssistive technology

For people who are deaf or hard of hearing, sign language interpreters are essential in removing communication barriers. Their main duty is to mediate communication between persons who use sign language and others who rely on verbal or written language. This crucial function is present in a variety of contexts, such as meetings for business, court cases, medical appointments, educational settings, and daily encounters. Sign language interpreters have a thorough knowledge of both sign language and the intended spoken or written language. They are adept at communicating not only the words but also the subtleties, feelings, and cultural context of the conversation. Thorough training, certification, and ongoing professional development are required for this skill. The value of sign language interpreters goes beyond simple translation; They serve as interpreters and fill gaps in accessibility and inclusiveness. Interpreters give deaf and hard-of-hearing people the same access to social interactions, economic possibilities, healthcare, and other opportunities as hearing people by facilitating successful communication. Sign language interpreters act as catalysts for a more open and just society in a world that cherishes variety and inclusivity. Their dedication to facilitating communication increases respect for one another, lessens prejudice, and builds a feeling of community for the deaf and hard-of-hearing population.