Multi-modality American Sign Language recognition

You have accessThe ASHA LeaderFeature1 Apr 2003Serving Clients Who Use Sign Language Susanne Scott, and James H. Lee Susanne Scott Google Scholar More articles by this author and James H. Lee Google Scholar More articles by this author https://doi.org/10.1044/leader.FTR1.08062003.6 SectionsAbout ToolsAdd to favorites ShareFacebookTwitterLinked In Many speech-language and hearing professionals now encounter clients who use sign languages as their primary mode of communication. Can these professionals serve these clients, or do they need to get interpreters or make referrals? ASHA is committed to meeting the needs of culturally and linguistically diverse populations. All too often “linguistically diverse” is assumed to refer to users of spoken languages other than English. However, individuals who use a signed language, whether American Sign Language (ASL) or some other sign system, are also linguistically diverse. It is difficult, if not impossible, to determine how many people in the United States use some form of sign language to communicate. It might be easier to try to determine the number of signing individuals that are seen in hearing and speech centers and in educational placements throughout the United States. In keeping with ASHA’s vision, it is important to ensure that there are an adequate number of speech-language and hearing professionals to serve this population. Although it may be necessary to use interpreters in some situations or locales or to make referrals to larger urban areas, programs, clinics, and schools throughout the country should be striving to better and more directly serve clients who use sign. The Population Clients who use a sign language/system are a very diverse population. Not only do these individuals differ in age, gender, race, ethnicity, national origin, religion, sexual orientation, and socio-economic status, but also in the sign language/system that they use. Often the sign language/system used is determined by the individual’s identity or, in the case of a child, the educational placement. An individual who identifies as a member of the deaf community/culture in the United States will most likely use American Sign Language. Individuals who depend on their residual hearing and identify as hard of hearing or hearing impaired, will most likely use an invented sign system (i.e., SEE II, Signed English) or more English-based sign. Children who are deaf or hard of hearing may use ASL, a sign system, or English-based sign as determined by the school’s educational/communication philosophy (see sidebar for definitions). Cultural and Linguistic Competence Cultural and linguistic competence can enhance clinical practice by: increasing the bond/relationship between the clinician and the client and the client’s significant others demonstrating respect and sensitivity enhancing service provision facilitating achievement of goals and carryover/generalization of skills And, importantly, it’s the right thing to do! The National Center for Cultural Competence ( www.georgetown.edu/ research/gucdc/ nccc/faqs.html) has identified six overarching reasons why cultural competence is important to health care providers. In brief, they are: 1. To respond to current and projected demographic changes in the United States 2. To eliminate long-standing disparities in the health status of people of diverse racial, ethnic, and cultural backgrounds 3. To improve the quality of services and health outcomes 4. To meet legislative, regulatory, and accreditation mandates 5. To gain a competitive edge in the market place 6. To decrease the likelihood of liability/malpractice claims Strategies Clinicians who are able to sign should apply the following strategies in order to meet the communication needs of their clients: Respect the communication mode of your clients. The clinician should try to match the client’s mode of communication. Be willing to admit when you do not understand. Try not to “bluff” or use a passive behavior. Speech-language-hearing professionals should model appropriate communication strategies such as confirmation and repair strategies when the clinician is not able to understand the client’s signs. Use facial expressions and body movements with signs. Facial expressions and body movements are fundamental features of ASL. In spoken English, speakers often signal a question by using a rising vocal intonation. In ASL, users do so by raising the eyebrows and widening the eyes. ASL users may ask a question by tilting their bodies forward while signaling with their eyes and eyebrows. Try to be conceptually correct in the use of signs. Choose signs that appropriately convey the information you are trying to communicate. Remember each sign in a signed language/system is composed of a specific, unique combination of a hand shape, movement, palm orientation, and location. Changing any one of these aspects of the sign changes the meaning of the sign. Always fingerspell technical terms (e.g., audiogram, frequency) first before assigning a “made up” sign (e.g., “A” hand shape circling in front of the ear + sign for graph = audiogram; “F” hand shape moving like a “wave” from left to right = frequency). Always use visual aids to supplement your signs. Use pictures of the ear and the “audiogram of familiar sounds” to help explain unfamiliar concepts (e.g., type of hearing loss; sounds the client may never have heard). Working With an Interpreter If necessary, it may be best to obtain the services of a certified interpreter for the deaf. Certified interpreters follow a strict Code of Ethics developed by the Registry of Interpreters of the Deaf (RID). Interpreters can hold different types of RID certification. For example, a CI (Certificate of Interpretation) certificate indicates the interpreter can sign from English to ASL and ASL to English. A CT (Certificate of Transliteration) certificate means the interpreter has been evaluated and is proficient in interpreting from spoken English to a more English-based sign and vice versa. Some RID interpreters hold both certificates. In addition, the National Association of the Deaf also certifies interpreters using a system with varying skill levels. Finally, some states also have certification programs and tests for interpreters. It is essential to find out what is required in your area and obtain the services of qualified interpreters. When working with an interpreter, speech-language or hearing professionals should remember that they are consumers of interpreting services. Be sure to establish what you and your signing client need from the interpreter. For example, some signing individuals have intelligible speech and the interpreter would not need to voice what the signer expresses. Other times the speech may not be intelligible enough and the interpreter may need to voice the intent of the signing client. Finally, it is important to remember that the interpreter is a conduit of information between the clinician and the client. The signing client is addressed directly. The client should not be addressed through the interpreter (i.e., “Tell her…”). Instead the professional should speak directly to the client and allow the interpreter to convey the information. To facilitate this, it is often helpful to position the interpreter next to the hearing or speech-language professional and across from the signing client. This allows for a good visual environment. The signing client can see the interpreter and the professional. During interactions as much eye contact as possible should be maintained with the client. The signing client will need to divide the visual attention between the professional and the interpreter but the professional should remain focused on the client. Definitions American Sign Language (ASL) ASL is a complete, complex language that incorporates the use of hand shapes, position, palm orientation and movement, facial expressions, body movements and postures, and gestures and other visual cues to form its words. It is completely separate from English. ASL has its own rules for grammar, morphology, and sentence order. It is the first language of many North Americans who are deaf and is the fourth most commonly used language in the United States (www.aslinfo.com). Deaf Community/Culture A multi-cultural group of individuals who identify themselves as Deaf, as an ethnic identity, and not a physical condition. The Deaf community is composed of people who use American Sign Language as their primary means of communication. The Deaf Community considers itself a linguistic minority group, a separate entity because of its unique culture, language, and social norms. It is separate from other disability consumer groups by the virtue of communication process, not physical disabilities (www.signmedia.com/info/adc.htm). Finger Spelling (Dactylology) and Invented Sign Systems Finger spelling uses hand shapes to represent the letters of the English alphabet. Invented Sign Systems (Signing Exact English [SEE] and Signed English [SE]) use manual codes (use of the hands) to represent the structure of English. English-Based Signing English-based signing entails signing a combination of both English and ASL features executed in English word order. The language proficiency of the signer will determine the use of the features from either language. It is different from the invented sign systems because it does not follow a clear-cut set of principles. In the past, this type of signing has been labeled pidgin signed English (PSE) and more recently referred to as sign English or conceptually accurate sign English (CASE). It was not intended to model English or be used in an educational context or to teach English. Its purpose is for social communication. Print Publications Resources Paul P. V. (2001). Language and Deafness (3rd ed.). San Diego, CA: Thomson Learning. Google Scholar Vold F. C., Kinsella-Meier M., & Hughes Hilley M. C. (1990). Signing with your client: A practical manual for audiologists and speech/language pathologists. Washington DC: Gallaudet University Press. Google Scholar Web www.nad.org/infocenter/infotogo/dcc/difference.html http://clerccenter.gallaudet.edu/InfoToGo/549.html www.oregon.gov/DHS/odhhs/ References Author Notes Susanne Scott, is a supervising aural rehabilitationist in the department of audiology and speech-language pathology at Gallaudet University. In addition to clinical supervision, she teaches the Clinical Applications of Sign Communication classes to students in the clinical doctorate of audiology program. Contact her by e-mail at [email protected]. James H. Lee, is a clinical supervisor in the department of audiology and speech-language pathology at Gallaudet University. In addition to clinical supervision, he teaches Clinical Applications of Sign Communication to speech-language pathology graduate students. Contact him by e-mail at [email protected]. Advertising Disclaimer | Advertise With Us Advertising Disclaimer | Advertise With Us Additional Resources FiguresSourcesRelatedDetails Volume 8Issue 6April 2003 Get Permissions Add to your Mendeley library History Published in print: Apr 1, 2003 Metrics Current downloads: 4,634 Topicsasha-topicsleader_do_tagasha-article-typesCopyright & Permissions© 2003 American Speech-Language-Hearing AssociationLoading ...

As part of the development of an educational tool that can help students achieve fluency in American Sign Language (ASL) through independent and interactive practice with immediate feedback, this paper introduces a near real-time system to recognize grammatical errors in continuous signing videos without necessarily identifying the entire sequence of signs. Our system automatically recognizes if a performance of ASL sentences contains grammatical errors made by ASL students. We first recognize the ASL grammatical elements including both manual gestures and nonmanual signals independently from multiple modalities (i.e. hand gestures, facial expressions, and head movements) by 3D-ResNet networks. Then the temporal boundaries of grammatical elements from different modalities are examined to detect ASL grammatical mistakes by using a sliding window-based approach. We have collected a dataset of continuous sign language, ASL-HW-RGBD, covering different aspects of ASL grammars for training and testing. Our system is able to recognize grammatical elements on ASL-HW-RGBD from manual gestures, facial expressions, and head movements and successfully detect 8 ASL grammatical mistakes.

The Shape of Air: American Sign Language as Narrative Prosthesis in 21st Century North American Media

This study examined the effect of a 1-day, 6-hour key word signing (KWS) workshop on in-service special education teachers’ and speech-language pathologists’ (SLPs) (a) skill identifying American Sign Language (ASL) signs; (b) skill producing ASL signs; (c) use of KWS in the classroom or therapy room; and (d) perceived changes from taking part in a KWS workshop. Participants included five special education teachers, three SLPs, and four students with complex communication needs (aged 3 to 14 years) participated in the study. All eight adult participants participated in a pretest-posttest design with repeated posttest measures over time, survey design, and phenomenological research to examine the effect of a KWS workshop on their skill identifying and producing manual signs as well as their perceived changes from taking part in the KWS workshop. Three of the eight adult participants and four students participated in an A-B single-case design, which was used to determine the effect of the KWS workshop on the in-service staff’s use of KWS in their classrooms or therapy rooms. The adult participants demonstrated an immediate increase in their ability to identify and produce the ASL signs taught during the KWS workshop, and they exhibited an increasing trend over the six postworkshop assessment sessions (across 11 or 12 weeks) in their ASL sign identification and production skills. The adult participants’ percentage of nonoverlapping data (PND) for the number of signed utterances, signs, and different signs they used in their classrooms or therapy rooms indicated a large effect. Two of the four students did not produce signed utterances during the classroom activities throughout the three preworkshop observations and five postworkshop observations. The PND for one student, who produced signed utterances, suggested a large effect and the PND for the other student, who used signed utterances, suggested a small effect. A majority of the participants reported that students, who relied on AAC and used natural speech, as well as classroom staff increased their production (i.e., imitation and spontaneous production) of ASL signs after the adult participants attended a workshop and began to use KWS in their classrooms and therapy rooms.

Spatial language (how one talks about space) in English and American Sign Language (ASL) was compared by asking 10 native signers and 10 English speakers to solve a set of spatial puzzles. Subjects had to describe where to place a group of blocks so that the blocks completely filled a puzzle grid. There were no group differences between how signers and speakers solved the puzzles. However, language affected the types of spatial commands that were used to solve them. English speakers relied heavily on the grid co-ordinate labels marked on the puzzle, but ASL signers were able to use signing space to refer to puzzle positions. ASL signers also made significantly more overt references to orientation due to the ability to represent orientation directly with hand position or tracing. The semantics of English in and on was found to be different from that of ASL IN and ON. Finally, analysis of modality-specific aspects of ASL spatial language revealed phonological and morphological constraints on the use of the nondominant hand, constraints on how signing space is mapped to physical space, and a bias toward choice of ground object deriving from phonological dominance and discourse constraints. The study highlights the ramifications of a linguistic system in which space itself is used to convey spatial information compared to a linguistic system that conveys this same information via an auditory signal.

Introduction Deaf people join groups of people all over the world who must manage two languages, one of which is a dominant-world language and the other a minority, often unfavored language. In the United States and Canada, Deaf people who use American Sign Language (ASL) as the preferred everyday language interact, often intimately, with individuals who use English – hearing teachers, relatives, and co-workers. Deaf people have many opportunities to use only ASL, but rarely can they avoid contact with English. They are more likely to have parents who use English than parents who use ASL. They are more likely to have teachers who are native speakers of English. Many have co-workers who speak only English. The language lives of Deaf people involve constantly moving between languages, ASL and English, and between cultural worlds, the worlds of ASL signers and English speakers. Because ASL does not have a written system, Deaf people use written English both as a means of contact with English and as a means of storing information about themselves and their language. Deaf people recite ASL poems and make videotapes of poetry performances, but their analyses of poetry appear in written English. Deaf children in a third grade classroom read a paragraph together in English, and then explain its meaning to each other in ASL. Deaf teenagers read computer manuals in English and explain to friends in ASL how to write a short program on the computer.

Eye-tracking corpora have advanced our understanding of reading processes by providing large-scale datasets of naturalistic reading behavior. However, existing corpora have almost exclusively sampled from typically hearing readers of spoken languages. Here, we present the Signers’ Eye-movements in English Reading (SEER) Corpus, a dataset of eye-movement behaviors from 41 skilled deaf adult readers who are early signers of American Sign Language (ASL), as well as a comparative group of 101 typically hearing monolingual English readers. Participants read 200 English sentences presented one at a time. In addition to eye-tracking data, the corpus includes detailed participant information: a standardized measure of reading proficiency, spelling recognition, and nonverbal intelligence for all participants. Information for the deaf participants include ASL comprehension scores, age of ASL acquisition, and phonological awareness scores (for a subset of participants). We report comparative analyses of reading behaviors at both the word level and sentence level. We also examine group differences in the effects of word length, frequency, and surprisal on local measures. The results indicate stronger effects of length and surprisal, but equivalent frequency effects (on content words) for deaf compared to hearing readers. The SEER Corpus offers researchers the opportunity to test hypotheses about reading development and efficiency in bimodal bilinguals who are first language users of ASL and skilled readers of English, supporting broader investigations of visual language processing. The corpus is preregistered and publicly available (https://doi.org/10.17605/OSF.IO/7P4F2) to facilitate replication, cross-study comparisons, and exploration of preliminary hypotheses in this understudied population.

The project "Transforming Expression into Language: American Sign Language Recognition using Image Processing" intends to employ image processing techniques to identify gestures in American Sign Language (ASL). The goal of this project is to develop an efficient system for translating ASL signs using visual cues generated from hand motions and gestures. The program intends to use image processing technology to bridge the communication gap between ASL users and nonusers. The use of image processing for ASL identification may improve communication in a range of situations while also increasing accessibility for the deaf community. The paper provides a realistic approach to real-time ASL motion interpretation using image processing techniques, thereby advancing the field of sign language recognition technology. KEYWORDS ASL Recognition, Image Processing Techniques, Convolutional Neural Networks (CNNs), Feature Extraction in Image Processing, Transforming Expressions into Text

Abstract: This study examined the impact of language dominance on variation in American Sign Language (ASL) production among 100 proficient deaf and hard of hearing signers who acquired ASL before age eight. While ASL variation has traditionally been attributed to factors like age of acquisition, proficiency, and sociolinguistic influences, this study introduced language dominance, a known factor modulating the presence of linguistic elements from one language within another among bilingual speakers. Findings revealed that ASL-English language dominance moderately predicted the use of English mouthings (operationalized here as mouthings ) and ASL classifiers (operationalized as classifiers ): ASL-dominant signers produced fewer mouthings and more classifiers, while English-dominant signers displayed the opposite pattern. Notably, this influence was consistent in both native and early nonnative signers, suggesting that the integration of English elements is not solely due to proficiency limitations but also reflects bilingual language dynamics. These results indicate that sign-spoken bilinguals may often operate in a bilingual mode, accessing both ASL and English syntactic structures during ASL discourse. Implications extend to ASL documentation and proficiency tests, as traditional monolingual frameworks may not capture the fluid syntactic variation in signing ecologies. This finding also suggests that aspects of English grammar may be amodal for ASL signers, with potential applications for bilingual processing models.

Hand gesture recognition is a rich area of research and covers a wide scope of applications from human machine interaction (e.g in games) to Deaf people computer interface and from 3D animationto control of mechanical systems. In this work a new algorithm for hand gesture recognition is proposed and evaluated. This algorithm uses moment invariants for feature extraction and neural networks for classification. To evaluate the algorithm a subset from the ASL (American Sign Language) is used, this subset consists of the 26 American one-handed sign language alphabet as a training set for the system. The system then tested using images that are different from the training set in size,orientation and position and the results for these tests are presented and discussed.

This paper deals primarily with the phonological structure of American Sign Language (ASL) and zmânî âmâžaî kurdî (ZAK) -Kurdish Sign Language. It is concerned with sign language (SL) and the types of sign language. One type is primary sign languages which are used by the Deaf people. Sign language is a visual-gestural language which relies on the use of the hands, facial expressions and body movements.Generally, there are myths about SLs. People believe that SLs are universal and have no grammatical structure. However, sign languages, as spoken languages, have lexicon, phonology, morphology and syntax. As far as the phonological structure of SLs includes handshape, location, movement and orientation of the hand. Therefore, some questions have been raised focusing on ASL and ZAK phonological structure. One of the questions is that: are the parameters of the phonological structure of ASL and ZAK the same? The aim of the paper is to apply ASL phonological aspects on ZAK. In this study, a mixed method is adopted. What is related to the hypothesis is that ZAK has three phonological parameters. One of the findings is that both ASL and ZAK have the same phonological parameters. As far as recommendation is concerned, more work can be carried out on the differences between ASL and ZAK at all linguistic levels.

Cross-linguistic comparisons of compounds are difficult because of the varied criteria and terms used by different linguists (Scalise and Bisetto 2009). To address this problem, Scalise and Bisetto proposed a universal three-level classification of compound types. Although several researchers have shown that American Sign Language (ASL) has compound signs, a classification of compound types in ASL has not been completed. All of the potential compounds in an ASL dictionary (Costello 1994) were identified, then verified as compounds with the help of a fluent deaf signer by applying standard tests for composition. These compounds were then classified using the Scalise and Bisetto classification. We found that Scalise and Bisetto’s three-level hierarchical classification successfully captured cross-category relationships among subtypes of compounds but fails to predict the existence of one type of compound attested in ASL. In our revised classification, a consistent set of criteria is used at each level, resulting in a classification that is both conceptually simpler and empirically more adequate. The second tier category for hierarchical compounds are bifurcated into the categories expressed predicate and unexpressed predicate, according to whether each predicate in a compound’s semantic structure is expressed by one of the overt constituents. The revision has the further advantage of allowing us to avoid any reference to word class/grammatical category in applying our taxonomy, a goal that we show to be desirable on both theoretical and empirical grounds.

American Sign Language (ASL) is a visual language with unique syntactic and grammatical structures. Translating ASL into coherent English text is critical for accessibility and inclusion, yet remains challenging due to limited labeled datasets, irregular syntax, and domain-specific vocabulary. This paper presents a complete pipeline for ASL-to-English transformation using the T5 Transformer model, guided by real-world ASL transcription data. A custom web crawling process is implemented to extract structured ASLEnglish sentence pairs from accessible online Sentence-Piece based platforms involved tokenization, stop word filtering, noise reduction, and data augmentation using NLP techniques. A specialized vocabulary was designed to handle frequent ASL ngrams and was integrated into the tokenizer to better preserve linguistic nuances. T5 fine-tuning was performed using the Hugging-Face Transformers library in a GPU-accelerated environment. Model outputs were evaluation scores with BLEU and ROUGE metrics against traditional Seq2Seq baselines. We also used attention heatmaps to visualize decoder focus during translation. The proposed method significantly improves fluency, syntactic clarity and contextual meaning in ASL- to-English translation.

Motor-iconicity of sign language does not alter the neural systems underlying tool and action naming

American sign language recognition and training method with recurrent neural network