Sign Language Translator and Teaching System

Introduction In this paper, I examine the practices of representing Deaf ‘voices’’ to hearing audiences in two recent US television crime dramas. More literally I look at how American Sign Language is framed and made visible on the screen through various production decisions. Drawing examples from an episode of CSI: New York that aired in December 2006 and an episode of Law and Order: Criminal Intent that aired in April 2007, I examine how the practices of filming Deaf people and the use of American Sign Language intersect with the production of a Deaf ‘voice’ on the screen. The problem of representing a Deaf ‘voice’ on the screen is akin to the problem of representing other minority languages. Film and television producers in the United States have to make choices about whether the majority audience of English speakers will have access to the minority language or not. In the face of this dilemma media producers have taken several approaches: subtitling foreign speech, translating foreign speech through other characters, or leaving the language inaccessible except to those who use it. The additional difficulty with representing national sign languages is that both the language and the recording medium are visual. Sometimes, filmmakers make the choice of leaving some portions of the signed dialogue inaccessible to a non-signing hearing audience. On the one hand this choice could indicate a devaluing of the signed communication, as its specific content is considered irrelevant to the plot. On the other hand it could indicate that Deaf people have a right to be visible on television using their own language without accommodating hearing people. A number of choices made in the filming and editing can subtly undermine positive representations of Deaf ‘voices’ particularly to a Deaf audience. These choices often construct an image of sign languages as objectified, exoticised, disjointed, incomplete, or a code for spoken language. Simple choices such as using simultaneous speaking and signing by Deaf characters, cropping the scene, translating or not translating the dialogue have powerful implications for the ways that Deaf ‘voices’ are becoming more visible in the 21st century. Typical filming and editing conventions effectively silence the Deaf ‘voice.’ Over 20 years ago, in the comprehensive book, Hollywood Speaks: Deafness and the Film Entertainment Industry (1988), Schuchman’s complaint that the filming and editing techniques of the day often did not attend to preserving the visibility and comprehensibility of sign language eon the screen, still applies today. As editing techniques have evolved over the years, fr om reliance on wide and medium shots to frequent intercutting of close-ups, the tendency to cut sign language off the screen, and out of the comprehensible view of the audience, may have even increased. Recent Portrayals of Deaf People on Television During one television season in the United States between August 2006 and April 2007, 30 episodes of six different serial television programs portrayed signing Deaf characters. Three of these programs had on-going Deaf characters that appeared in a number of episodes throughout the season, while three other programs portrayed Deaf people in a one-off episode with a Deaf theme. Initial air date for the season Program and Season # of Episodes 1 14 Aug. 2006 Weeds, Season 2 5 2 20 Sep. 2006 Jericho, Season 1 13 3 28 Jan. 2007 The L Word, Season 4 9 Table 1. Dramas with Ongoing Deaf Characters during the 2006-2007 USA Television Season Initial air date Program, Season, Episode Episode Title 1 13 Dec. 2006 CSI: New York, Season 3, Episode 12 “Silent Night” 2 3 Apr. 2007 Law and Order: Criminal Intent, Season 6, Episode 18 "Silencer" 3 12 Apr. 2007 Scrubs, Season 6, Episode 16 “My Words of Wisdom” Table 2. One-off Episodes with Signing Deaf Characters during the 2006-2007 USA Television Seasons Ironically, although the shows with ongoing characters sometimes allow the Deafness of the character to be incidental to the character, it is only the one-off crime dramas that show Deaf people relating with one another as members of a vibrant community and culture based in sign language. Often, in the ongoing series, the characters remain isolated from the Deaf community and their interactions with other Deaf people are sparse or non-existent. For example, out of the 27 episodes with an ongoing Deaf character only two episodes of The L-Word have more than one Deaf character portrayed. In both Weeds and The L-Word the Deaf character is the love interest of one of the hearing characters, while in Jericho, the Deaf character is the sister of one of the main hearing characters. In these episodes though some of realities about Deaf people’s lives are touched on as they relate to the hearing characters, the reality of signing Deaf people’s social lives in the Deaf community is left absent and they are depicted primarily interacting with hearing people. The two episodes, from CSI: New York, and Law and Order: Criminal Intent, focus on the controversial theme of cochlear implants in the Deaf community. Though it is true that generally the signing Deaf community in the U.S.A. sees cochlear implants as a threat to their community, there is no record of this controversy ever motivating violent criminal acts or murder as portrayed in these episodes. In the episode of CSI: New York entitled “Silent Night” a conflict between a young Deaf man and Deaf woman who were formerly romantically involved is portrayed. The murdered young woman who comes from a Deaf family does not want her Deaf baby to have a cochlear implant while the killer ex-boyfriend who has a cochlear implant believes that it is the best option for his child. The woman’s Deaf parents are involved in the investigation. The episode of Law and Order: Criminal Intent, entitled “Silencer,” is also ultimately about a conflict between a Deaf man and a Deaf woman over cochlear implants. In the end, it is revealed that the Deaf woman is exploring the possibility of a cochlear implant. Her boyfriend projecting the past hurt of his hearing sister leaving him behind to go off and live her own life, doesn’t want his girlfriend to leave him once she gains more hearing. So he shoots the cochlear implant surgeon in the hand to prevent him from being able to perform the surgery. Then he accidentally kills him by crushing his voice box to prevent him from screaming. Analyzing Two Crime Dramas In both television dramas, the filmmakers use both sound and video editing techniques to mark the experiential difference between hearing and Deaf characters. In comparing the two dramas two techniques are evident : muting/distorting sounds and extreme close-ups on lips talking or hands signing. Though these techniques may heighten awareness of deaf experience to a non-signing audience they also point to a disabling stereotyping of the experience of being Deaf as lacking — framing their experience as hearing loss rather than Deaf gain (Bauman & Murray; Shakespeare 199). By objectifying sign language through extreme close ups American Sign Language is portrayed as something strange and unusual that separates Deaf signers from hearing speakers. The auditory silences can either jolt the hearing non-signer into awareness of the sensory aspect of sound that is missing or it can jolt them into awareness of the visual world that they often don’t really see. In the opening few scenes of the episodes both CSI: New York and Law and Order: Criminal Intent use sound editing alternately muting or distorting sounds as they cut between a ‘deaf’ auditory perspective and a ‘hearing’ perspective on the action as it unfolds. Even though the sound editing does play a part in the portrayal of Deaf people’s experience as lacking sound, the more important aspects of film production to attend to are the visual aspects where Deaf people are seen authentically signing in their own language. Scene Analysis Methodology In taking a closer look at a scene from each episode we can see exactly how the filming and editing techniques work to create an image of sign language. I have chosen comparable scenes where a Deaf individual is interviewed or interrogated by the police using a sign language interpreter. In each scene it can be assumed that all the communication is happening in both English and ASL through an interpreter, so at all times some signing should be occurring. In transcribing the scenes, I noted each point when the editor spliced different camera shots adjacent to each other. Because of the different visual aesthetics in each program where one relied heavily on continuous panning shots, I also noted where the camera shifted focus from one character to another marking the duration of screen time for each character. This allowed for a better comparison between the two programs. In my transcripts, I included both glosses of the ASL signs visible on the screen as well as the flow of the spoken English on the audio track. This enabled me to count how many separate shifts in character screen time segments contained signing and how much of these contained completely visible signing in medium shots. CSI:NY Witness Interview Scene In the first signing scene, Gina (played by Marlee Matlin) is brought in for an interview with Detective Taylor and a uniformed officer interpreter. The scene opens with a medium shot on Detective Taylor as he asks her, “What do you think woke you up?” The shot cuts to an extreme close up of her face and hands and pans to only the hands

Communication obstacles in health care settings adversely impact patient-practitioner interactions by impeding service efficiency, reducing mutual trust and satisfaction, or even endangering health outcomes. When interlocutors are separated by language, interpreters are required. The efficacy of interpreting, however, is constrained not just by interpreters' competence but also by health care providers' facility working with interpreters. Deaf individuals whose preferred form of communication is a signed language often encounter communicative barriers in health care settings. In those environments, signing Deaf people are entitled to equal communicative access via sign language interpreting services according to the Americans with Disabilities Act and Executive Order 13166, the Limited English Proficiency Initiative. Yet, litigation in states across the United States suggests that individual and institutional providers remain uncertain about their legal obligations to provide equal communicative access. This article discusses the legal and ethical imperatives for using professionally certified (vs. ad hoc) sign language interpreters in health care settings. First outlining the legal terrain governing provision of sign language interpreting services, the article then describes different types of "sign language" (e.g., American Sign Language vs. manually coded English) and different forms of "sign language interpreting" (e.g., interpretation vs. transliteration vs. translation; simultaneous vs. consecutive interpreting; individual vs. team interpreting). This is followed by reviews of the formal credentialing process and of specialized forms of sign language interpreting-that is, certified deaf interpreting, trilingual interpreting, and court interpreting. After discussing practical steps for contracting professional sign language interpreters and addressing ethical issues of confidentiality, this article concludes by offering suggestions for working more effectively with Deaf clients via professional sign language interpreters.

The Shape of Air: American Sign Language as Narrative Prosthesis in 21st Century North American Media

While a significant amount of work has been done on the commonly used, tightly -constrained weather-based, German sign language (GSL) dataset, little has been done for continuous sign language translation (SLT) in more realistic settings, including American sign language (ASL) translation. Also, while CNN - based features have been consistently shown to work well on the GSL dataset, it is not clear whether such features will work as well in more realistic settings when there are more heterogeneous signers in non-uniform backgrounds. To this end, in this work, we introduce a new, realistic phrase-level ASL dataset (ASLing), and explore the role of different types of visual features (CNN embeddings, human body keypoints, and optical flow vectors) in translating it to spoken American English. We propose a novel Transformer-based, visual feature learning method for ASL translation. We demonstrate the explainability efficacy of our proposed learning methods by visualizing activation weights under various input conditions and discover that the body keypoints are consistently the most reliable set of input features. Using our model, we successfully transfer-learn from the larger GSL dataset to ASLing, resulting in significant BLEU score improvements. In summary, this work goes a long way in bringing together the AI resources required for automated ASL translation in unconstrained environments.

Sign language plays a crucial role as a communication tool for both the deaf and hard-of- hearing communities, enabling them to engage and interact effectively within their own community as well as with others.However, communication barriers arise when individuals unfamiliar with sign language engage with those who rely on it, underscoring the need for inclusive solutions. Real-time sign language interpretation systems, leveraging machine learning and computer vision technologies, present a promising approach to bridging this gap. These systems convert sign language gestures into spoken or written language by utilizing gesture recognition algorithms, neural networks, and natural language processing. By analyzing hand movements, facial expressions, and body language, the systems provide accurate, context-aware translations of various sign languages, such as American Sign Language (ASL), with minimal delay. This enables seamless, natural interactions, making such technologies essential for fostering inclusive communication in diverse settings. Key Words: Sign language recognition, real- time interpretation, machine learning, com- puter vision, gesture recognition, neural networks, natural language processing, communication barriers, inclusivity, American Sign Language (ASL).

Sign language is very important for millions of people across the globe, especially in deaf communities. With the increasing dependency on technology and digitalization, there is a requirement of identifying the sign language automatically so that the proper communication can take place between the deaf and dumb people. This paper focuses on building a strong and feasible system for identifying the sign language in the real-time. It takes note of recent advancements in techniques of computer vision and machine learning, like CNNs, detecting signs by people who sign mainly using American Sign Language as a case study. It is a large dataset of signs and gestures in ASL collected and preprocessed to ensure quality and consistency. Major modules of the system were detection, tracking, and translation of gestures. Using the combination of CNN's along with optical flow algorithms determines which hand movements to include as part of the gesture's interpretation and models that can model temporal sequences from such information. With such capability, the translation module does so to translate the identified signs into text thus being accessible to signers as well as non-signers. To test the power and flexibility of the system, evaluation is conducted based on a diverse set of signers, including different shapes, speeds, and even signing styles. Success criteria of this paper would be of high accuracy, less latency, and ease of incorporation into assistive technologies, applications such as educational tools or real-time communication platforms, with great implications in breaking down barriers in communication and building an inclusive society. Proposed approach would empower the hearing impaired with full participation in this world that nowadays totally depends upon the digital communication by providing technological solutions to understand and translate sign language in real time.

Contents: Preface. Part I: Reminiscences. L. Fant, Two Memorable Meals With Ursula and Ed. R.M. Battison, American Sign Language Linguistics 1970-1980: Memoir of a Renaissance. B. Bragg, A Fish Story for Ursula and Other Remembrances. Part II: Historical and Comparative Analyses of Sign Languages. J. Woodward, Sign Languages and Sign Language Families in Thailand and Vietnam. D. McKee, G. Kennedy, Lexical Comparisons of Signs From American, Australian, British, and New Zealand Sign Languages. H. Lane, R.C. Pillard, M. French, Origins of the American Deaf-World: Assimilating and Differentiating Societies and Their Relation to Genetic Patterning. Part III: Language in the Visual-Spatial Modality. E.L. Newport, T. Supalla, Sign Language Research at the Millennium. P. Siple, Attentional Resources and Working Memory: A New Framework for the Study of the Impact of Deafness on Cognition. M. Wilson, K. Emmorey, When Does Modality Matter? Evidence From ASL on the Nature of Working Memory. P.B. Braem, T. Bram, A Pilot Study of the Expressive Gestures Used by Classical Orchestra Conductors. N. Frishberg, An Interpreter Creates the Space. Part IV: Linguistic Analysis of Sign Languages. S.D. Fischer, More Than Just Handwaving: The Mutual Contributions of Sign Language and Linguistics. R.B. Wilbur, Phonological and Prosodic Layering of Nonmanuals in American Sign Language. R.C. Loew, C.T. Akamatsu, M. Lanaville, A Two-Handed Manual Alphabet in the United States. E. Pizzuto, V. Volterra, Iconicity and Transparency in Sign Languages: A Cross-Linguistic Cross-Cultural View. M. Kuntze, Codeswitching in ASL and Written English Language Contact. S.K. Liddell, Indicating Verbs and Pronouns: Pointing Away From Agreement. Part V: Language Acquisition. J.G. Bettger, Viewing Deaf Children in a New Way: Implications of Bellugi and Klima's Research for Education. R.P. Meier, Shared Motoric Factors in the Acquisition of Sign and Speech. A. Abrahamsen, Explorations of Enhanced Gestural Input to Children in the Bimodal Period. D. Lillo-Martin, Early and Late in Language Acquisition: Aspects of the Syntax and Acquisition of Wh-Questions in American Sign Language. J. Reilly, Bringing Affective Expression Into the Service of Language: Acquiring Perspective Marking in Narratives. C.A. Padden, V.L. Hanson, Search for the Missing Link: The Development of Skilled Reading in Deaf Children. L.A. Petitto, On the Biological Foundations of Human Language. Part VI: The Neural Organization of Sign Language. A.R. Damasio, H. Damasio, Language and the Brain. D.P. Corina, Some Observations Regarding Paraphasia in American Sign Language. H. Poizner, D. Brentari, M.E. Tyrone, J. Kegl, The Structure of Language as Motor Behavior: Clues From Signers With Parkinson's Disease. V.A. Fromkin, On the Uniqueness of Language.

People around the world with speech and hearing impairment use a media of communication known as ‘Sign language’. In recent times, Sign language is omnipresent. However, there exists a challenge for people who do not know sign language, to communicate with people who can communicate exclusively using sign language. This gap can be bridged by using technologies of recent times to recognize gestures and design intuitive systems with deep learning. The aim of this paper is to recognise American Sign Language gestures dynamically and create an intuitive system which provides sign language translation to text and speech of various languages. The system uses Convolutional neural network, natural language processing, language translation and text-to-speech algorithms. It is capable of recognizing hand gestures dynamically and predicting the corresponding letters to form a desired sentence accurately.

This paper presents the development of a real-time sign language translator that integrates speech recognition technology to bridge communication gaps between sign language users and non-signers. The system translates spoken or typed words into corresponding American Sign Language (ASL) and Indian Sign Language (ISL) gestures using a predefined database of sign representations. The paper discusses the system’s architecture, implementation challenges, results, and potential applications in education, public spaces, and assistive technology. Additionally, the research explores the integration of AI-driven enhancements to improve translation accuracy, accommodate a broader vocabulary, and incorporate non-manual sign components such as facial expressions and body language. Future work will focus on refining gesture recognition, improving speech processing, and ensuring accessibility across diverse linguistic and cultural contexts.

Hearing Impaired individuals routinely encounter limitations in their involvement in social interactions, access to intriguing information, and participation in everyday activities, among various other aspects. However, the hardest part of their interactions with regular people is communication, because sign language is the primary language of those who are hearing impaired. However, the general public is unaware of sign language. Each country has its own sign language. However, there are some striking similarities between them. In Sri Lanka, hearing impaired people use Sri Lankan Sign Language (SLSL) as their communication language. There is several research done on Sign Language recognition and translation. But no fully functioning system is utilized for Sri Lankan Sign Language translation. To find the gap in this area, we conducted a systematic literature review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method that analyses 12 studies on Sign Language Translation (SLT). As per the literature review, Image Processing (IP) and Convolutional Neural Networks (CNN) are the most used techniques for Sign Language translation. But these methods have limitations: not enough data, differences in how people use sign language, difficulty in translating in real-time, not capturing cultural aspects, needing specific equipment, and understanding the context of conversations. Recognizing and solving these problems is important, especially for languages like SLSL. Future research should focus on getting more data, making translation work for different cultures, and improving real-time translation. This will help hearing impaired people communicate better with others.

Little is known about the nature of fingerspelling during sign language interpretation. In this small-scale, exploratory study, we examined the fingerspelling of interpreters working in five different sign languages: American Sign Language (ASL), Australian Sign Language (Auslan), British Sign Language (BSL), Irish Sign Language (ISL), and Italian Sign Language (LIS). Sixteen interpreters were video-recorded as they rendered President Barack Obama’s 2009 inaugural address in their country’s sign language. After completing their interpretations, the participants engaged in a retrospective interview about their work. The data were analyzed both quantitatively (for frequency and type of fingerspelling) and qualitatively (for factors influencing fingerspelling). Results indicate that the most fingerspelled items ( n = 137) were produced in the ASL interpretations and the fewest ( n = 18) were produced in the LIS interpretations; variation between the groups was found in lexicalized fingerspelling and the fingerspelling of place names. We suggest that the variation in fingerspelling both within and between groups may be explained by sociolinguistic factors, including interpreters’ language attitudes and perceptions of the deaf audience. This exploratory study provides a first step in investigating the fingerspelling of interpreters in a variety of sign languages.

The purpose of this study was to investigate the potential of a tangible game, Inclusive Glossary of Mathematical Terms (GIM), to support the teaching of math and Portuguese sign language (Língua Gestual Portuguesa; LGP) to preschool and first-grade children, with an emphasis on inclusive education, knowledge acquisition, and engagement. GIM is composed of physical cards that are complemented by digital videos that run on an interface designed for this game. The game consists of two sets of cards and respective animations. As an inclusive math glossary, it also includes videos where concepts, characters, and objects are identified through images, words, and sign language. During the development process of GIM, it was possible to involve a multidisciplinary team, of which a Deaf sign language teacher and a sign language interpreter were part to ensure the clarity of the sign language and refine the written sentences accompanying the glossary. In order to validate its contents, a focus group of preschool and primary teachers specialized in the teaching of Deaf and Hard of Hearing (DHH) children was formed. A total of 120 children, aged between three and eight years old (M = 5.61; SD = 0.78), and seven LGP and special education teachers from three Portuguese schools participated in the study. The children used GIM in their classrooms, with the assistance of the participating teachers, who were subsequently interviewed to assess their engagement, interest, and learning outcomes. The interviews were analyzed using content analysis procedures, including descriptive statistics and Pearson correlations. The results indicated that GIM has the potential to effectively support the teaching of mathematics and LGP to preschool and first-grade children, especially in inclusive education contexts. The tangible nature of the game facilitated children's understanding of mathematical concepts and LGP, leading to improved involvement in the teaching-learning process. Moreover, the game promoted awareness of diversity and the principles of inclusive education, making it a valuable tool for promoting social inclusion and understanding. However, the study has reduced sample and context-specific limitations, suggesting a need for further research in this area.

Sign language is the only way for deaf and mute people to represent their needs and feelings. Most of non-deaf-mute people do not understand sign language, which leads to many difficulties for deaf-mutes' communication in their social life. Sign language interpretation systems and applications get a lot of attention in the recent years. In this paper, we review sign language recognition and interpretation studies based on machine learning, image processing, artificial intelligence, and animation tools. The two reverse processes for sign language interpretation are illustrated. This study discusses the recent research on sign language translation to text and speech with the help of hand gestures, facial expressions interpretation, and lip reading. Also, state of the art in speech to sign language translation is discussed. In addition, some of the popular and highly rated Android and Apple mobile applications that facilitate disabled people communication are presented. This paper clarifies and highlights the recent research and real used applications for deaf-mute people help. This paper tries to provide a link between research proposals and real applications. This link can help covering any gap or non-handled functionalities in the real used applications. Based on our study, we introduce a proposal involves set of functionalities/options that separately introduced and discussed by the recent research studies. These recent research directions should be integrated for achieving more real help. Also, a set of non-addressed research directions are suggested for future focus.

ABSTRACTThis paper discusses issues of language revitalization and linguistic prescriptivism related to the teaching of American Sign Language (ASL) in Canada. I describe the current and historical context for ASL teaching, including the increase in ASL course offerings at some postsecondary institutions, and discuss findings related to challenges in curriculum development and issues of prescriptivism as they recur in my study of developing an ASL curriculum for parents of deaf children that is aligned with the Common European Framework of Reference for Languages.

Sign language is the primary mode of communication for the hearing and speech impaired and there is a need for systems to translate sign languages to spoken languages. Prior research has been focused on providing glove based solutions which are intrusive and expensive. We propose a sign language translation system based solely on visual cues and deep learning for accurate translation. Our system applies Computer Vision and Neural Machine Translation for American Sign Language (ASL) gloss recognition and translation respectively. In this paper, we show that an end to end neural network system is not only capable of recognition of individual ASL glosses but also translation of continuous sign language videos into complete English sentences, making it an effective and practical tool for sign language communication.