Improved dynamic gesture segmentation for Thai sign language translation

Hand gesture identification is one of problems being widely studied. There are two research trends corresponding to two data types, which are static and dynamic gestures. The static gesture is recognized based on the hand shape, while motion is the main feature in identifying dynamic gestures. In this paper, we propose an approach for modeling the dynamic hand gestures based on a combination of two mentioned information. At first, the hand silhouette is extracted using a skin-color filter. A sequence of geometric manipulations is then performed to remove the possible arm. The characteristics which describe the hand shape and motion orientation are estimated. Finally, the k-means clustering technique is combined with hidden Markov model to model each dynamic gesture. The experiments are performed on human-computer interaction dataset and obtain high efficiency.

Language bridges the gap of communication, and Sign Language (SL) is a native language among vocal and mute community. Every region has its own sign language. In Pakistan, Urdu Sign Language (USL) is a visual gesture language used by the deaf community for communication. The Urdu alphabet in Pakistan Sign Language consists not only of static gestures but also includes dynamic gestures. There are a total of 40 alphabets in Urdu sign language, with 36 being static and 4 being dynamic. While researchers have focused on the 36 static gestures, the 4 dynamic gestures have been overlooked. Additionally, there remains a lack of advancements in the development of Pakistan Sign Language (PSL) with respect to Urdu alphabets. A dataset named UAlpa40 has been compiled, comprising 22,280 images, among which 2,897 are originally created and 19,383 are created through noise or augmentation, representing the 36 static gestures and 393 videos representing the 4 dynamic gestures, completing the set of 40 Urdu alphabets. The standard gestures for USL are published by the Family Educational Services Foundation (FESF) for the deaf and mute community of Pakistan. This dataset was prepared in real-world environments under expert supervision, with volunteers ranging from males to females aged 20 to 45. This newly developed dataset can be utilized to train vision-based deep learning models, which in turn can aid in the development of sign language translators and finger-spelling systems for USL.

Losing the ability to speak exerts psychological and social impacts on the affected people due to the lack of proper communication. Thus, Sign Language (SL) is considered a boon to people with hearing and speech impairment. SL has developed as a handy mean of communication that form the core of local deaf cultures. It is a visual–spatial language based on positional and visual components, such as the shape of fingers and hands, their location and orientation as well as arm and body movements. The problem is that SL is not understood by everyone, forming a communication gap between the mute and the able people. Multiple and systematic scholarly interventions that vary according to context have been implemented to overcome disability-related difficulties. Sign language recognition (SLR) systems based on sensory gloves are significant innovations that aim to procure data on the shape or movement of the human hand to bridge this communication gap, as the proposed system. The proposed model is a glove equipped with five flex sensors, interfacing with a control unit fixed on the arm, translating American Sign Language (ASL) and Arabic Sign Language (ArSL) to both text and speech, displayed on a simple Graphical User Interface (GUI). The proposed system aims to provide an affordable and user friendly SL translator system, working on the basis of Machine Learning (ML). However, it adapts to each person’s hand instead of using a generic data set. The system achieved 95% recognition rate with static gestures and up to 88% with dynamic gestures.

Sign language recognition plays a crucial role in supporting communication between deaf communities and hearing individuals. In particular, Vietnamese Sign Language (VSL) recognition remains a challenging task due to the complexity of hand gestures and limited available datasets. However, most existing approaches address static gestures and dynamic sign phrases as separate recognition problems, often employing different feature representations and independent processing pipelines. This fragmentation increases system complexity and limits scalability for real-time sign language applications. This study proposes a unified hand-landmark-based deep learning framework for recognizing both static and dynamic VSL gestures within a single integrated system. The system begins by detecting hand landmarks through the MediaPipe hand tracking pipeline. Based on temporal motion analysis of landmark sequences, a gesture routing mechanism automatically determines whether the input corresponds to a static gesture or a dynamic sign phrase. Static gestures are classified using a convolutional neural network (CNN), while dynamic gestures are processed using a long short-term memory (LSTM) network to capture temporal dependencies. Experiments were conducted on a VSL dataset consisting of 23 static hand signs and 4 dynamic sign phrases collected from multiple participants with variations in hand shape and gesture execution style. The system performance is evaluated using accuracy, precision, recall, and F1-score. Experimental results demonstrate that the proposed framework achieves an average recognition accuracy of 92% for static gestures and 88.4% for dynamic sign phrases, outperforming traditional machine learning baselines. The proposed system provides a practical and efficient solution for VSL recognition and has potential applications in real-time assistive communication systems.

In this article, a real-time dynamic finger gesture recognition using a soft sensor embedded data glove is presented, which measures the metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joint angles of five fingers. In the gesture recognition field, a challenging problem is that of separating meaningful dynamic gestures from a continuous data stream. Unconscious hand motions or sudden tremors, which can easily lead to segmentation ambiguity, makes this problem difficult. Furthermore, the hand shapes and speeds of users differ when performing the same dynamic gesture, and even those made by one user often vary. To solve the problem of separating meaningful dynamic gestures, we propose a deep learning-based gesture spotting algorithm that detects the start/end of a gesture sequence in a continuous data stream. The gesture spotting algorithm takes window data and estimates a scalar value named gesture progress sequence (GPS). GPS is a quantity that represents gesture progress. Moreover, to solve the gesture variation problem, we propose a sequence simplification algorithm and a deep learning-based gesture recognition algorithm. The proposed three algorithms (gesture spotting algorithm, sequence simplification algorithm, and gesture recognition algorithm) are unified into the real-time gesture recognition system and the system was tested with 11 dynamic finger gestures in real-time. The proposed system took only 6 ms to estimate a GPS and no more than 12 ms to recognize the completed gesture in real-time.

Sign language recognition devices are effective approaches to breaking the communication barrier between signers and non-signers and exploring human-machine interactions. Wearable gloves have been developed for gesture recognition and virtual reality applications by employing flexible sensors for motion detection and machine learning for data analysis. However, most existing wearable devices present limited sign language translating capacity due to the sensors' design and distribution. Here, we propose a cost-effective dual-hand soft fiber optic glove system consisting of multimode soft liquid-core fiber optic sensors, gyroscopes, wireless printed circuit boards, and batteries for sign language translation. In combination with different deep learning techniques and recognition strategies, the glove system can recognize static gestures and dynamic gestures of American Sign Language, and deduce the meaning of sentences by the sequence of gestures. The soft glove system exhibits a broad sign language range (10 numbers, 26 alphabets, 18 words, and 5 sentences meaning prediction), and high recognition accuracy (98.6% for static gestures, 95% for dynamic gestures). The results also present the recognizing capacity for high-correlated gestures (e.g., “M” and “N”). Finally, we demonstrate its application for controlling the motion of a virtual character through 7 discrete commands in the VR interface.

Sign language serves as a vital way to communicate with individuals with hearing loss, deafness, or a speech disorder, yet accessibility remains limited, requiring technological advances to bridge the gap. This study presents the first real-time Greek Sign Language recognition system utilizing deep learning and embedded computers. The recognition system is implemented using You Only Look Once (YOLO11X-seg), an advanced object detection model, which is embedded in a Python-based framework. The model is trained to recognize Greek Sign Language letters and an expandable set of specific words, i.e., the model is capable of distinguishing between static hand shapes (letters) and dynamic gestures (words). The most important advantage of the proposed system is its mobility and scalable processing power. The data are recorded using a mobile IP camera (based on Raspberry Pi 4) via a Motion-Joint Photographic Experts Group (MJPEG) Stream. The image is transmitted over a private ZeroTier network to a remote powerful computer capable of quickly processing large sign language models, employing Moonlight streaming technology. Smaller models can run on an embedded computer. The experimental evaluation shows excellent 99.07% recognition accuracy, while real-time operation is supported, with the image frames processed in 42.7 ms (23.4 frames/s), offering remote accessibility without requiring a direct connection to the processing unit.

Sign language is a crucial communication tool for hearing and deaf communities, with each language containing unique hand positions, motion profiles, and shapes. To develop sign language recognition systems, common input data formats include RGB and depth, considering perspective, articulation distributions, joint visibility, and hand shape. However, difficulties endure in intricate and crowded settings, resulting in increased average errors. An improved deep learning method that makes use of Google's MediaPipe framework and combines Convolutional and Recurrent Neural Networks (RNNs) is suggested as a solution to these problems. While RNNs record temporal correlations in sign language gesture data, CNNs extract spatial characteristics from static images. MediaPipe is essential for preprocessing and feature extraction, as well as for supplying precise hand landmarks and enabling real-time processing. The proposed system shows a significant improvement in recognition accuracy and processing efficiency, with up to 5% improvement in recognition accuracy compared to traditional methods. This work contributes to the advancement of sign language recognition technologies, offering a more reliable and efficient solution for real-time sign language interpretation and text conversion.

Objective:The goal of this thesis is to design the interaction structure and framework of system to recognize sign language.BR Background: The sign language of meaningful individual gestures is combined to construct a sentence, so it is difficult to interpret and recognize the meaning of hand gesture for system, because of the sequence of continuous gestures. This being so, in order to interpret the meaning of individual gesture correctly, the interaction structure and framework are needed so that they can segment the indication of individual gesture.BR Method: We analyze 700 sign language words to structuralize the sign language gesture interaction. First of all, we analyze the transformational patterns of the hand gesture. Second, we analyze the movement of the transformational patterns of the hand gesture. Third, we analyze the type of other gestures except hands. Based on this, we design a framework for sign language interaction.BR Results: We elicited 8 patterns of hand gesture on the basis of the fact on whether the gesture has a change from starting point to ending point. And then, we analyzed the hand movement based on 3 elements: patterns of movement, direction, and whether hand movement is repeating or not. Moreover, we defined 11 movements of other gestures except hands and classified 8 types of interaction. The framework for sign language interaction, which was designed based on this mentioned above, applies to more than 700 individual gestures of the sign language, and can be classified as an individual gesture in spite of situation which has continuous gestures.BR Conclusion: This study has structuralized in 3 aspects defined to analyze the transformational patterns of the starting point and the ending point of hand shape, hand movement, and other gestures except hands for sign language interaction. Based on this, we designed the framework that can recognize the individual gestures and interpret the meaning more accurately, when meaningful individual gesture is input sequence of continuous gestures.BR Application: When we develop the system of sign language recognition, we can apply interaction framework to it. Structuralized gesture can be used for using database of sign language, inventing an automatic recognition system, and studying on the action gestures in other areas.

You have accessThe ASHA LeaderFeature1 Apr 2003Serving Clients Who Use Sign Language Susanne Scott, and James H. Lee Susanne Scott Google Scholar More articles by this author and James H. Lee Google Scholar More articles by this author https://doi.org/10.1044/leader.FTR1.08062003.6 SectionsAbout ToolsAdd to favorites ShareFacebookTwitterLinked In Many speech-language and hearing professionals now encounter clients who use sign languages as their primary mode of communication. Can these professionals serve these clients, or do they need to get interpreters or make referrals? ASHA is committed to meeting the needs of culturally and linguistically diverse populations. All too often “linguistically diverse” is assumed to refer to users of spoken languages other than English. However, individuals who use a signed language, whether American Sign Language (ASL) or some other sign system, are also linguistically diverse. It is difficult, if not impossible, to determine how many people in the United States use some form of sign language to communicate. It might be easier to try to determine the number of signing individuals that are seen in hearing and speech centers and in educational placements throughout the United States. In keeping with ASHA’s vision, it is important to ensure that there are an adequate number of speech-language and hearing professionals to serve this population. Although it may be necessary to use interpreters in some situations or locales or to make referrals to larger urban areas, programs, clinics, and schools throughout the country should be striving to better and more directly serve clients who use sign. The Population Clients who use a sign language/system are a very diverse population. Not only do these individuals differ in age, gender, race, ethnicity, national origin, religion, sexual orientation, and socio-economic status, but also in the sign language/system that they use. Often the sign language/system used is determined by the individual’s identity or, in the case of a child, the educational placement. An individual who identifies as a member of the deaf community/culture in the United States will most likely use American Sign Language. Individuals who depend on their residual hearing and identify as hard of hearing or hearing impaired, will most likely use an invented sign system (i.e., SEE II, Signed English) or more English-based sign. Children who are deaf or hard of hearing may use ASL, a sign system, or English-based sign as determined by the school’s educational/communication philosophy (see sidebar for definitions). Cultural and Linguistic Competence Cultural and linguistic competence can enhance clinical practice by: increasing the bond/relationship between the clinician and the client and the client’s significant others demonstrating respect and sensitivity enhancing service provision facilitating achievement of goals and carryover/generalization of skills And, importantly, it’s the right thing to do! The National Center for Cultural Competence ( www.georgetown.edu/ research/gucdc/ nccc/faqs.html) has identified six overarching reasons why cultural competence is important to health care providers. In brief, they are: 1. To respond to current and projected demographic changes in the United States 2. To eliminate long-standing disparities in the health status of people of diverse racial, ethnic, and cultural backgrounds 3. To improve the quality of services and health outcomes 4. To meet legislative, regulatory, and accreditation mandates 5. To gain a competitive edge in the market place 6. To decrease the likelihood of liability/malpractice claims Strategies Clinicians who are able to sign should apply the following strategies in order to meet the communication needs of their clients: Respect the communication mode of your clients. The clinician should try to match the client’s mode of communication. Be willing to admit when you do not understand. Try not to “bluff” or use a passive behavior. Speech-language-hearing professionals should model appropriate communication strategies such as confirmation and repair strategies when the clinician is not able to understand the client’s signs. Use facial expressions and body movements with signs. Facial expressions and body movements are fundamental features of ASL. In spoken English, speakers often signal a question by using a rising vocal intonation. In ASL, users do so by raising the eyebrows and widening the eyes. ASL users may ask a question by tilting their bodies forward while signaling with their eyes and eyebrows. Try to be conceptually correct in the use of signs. Choose signs that appropriately convey the information you are trying to communicate. Remember each sign in a signed language/system is composed of a specific, unique combination of a hand shape, movement, palm orientation, and location. Changing any one of these aspects of the sign changes the meaning of the sign. Always fingerspell technical terms (e.g., audiogram, frequency) first before assigning a “made up” sign (e.g., “A” hand shape circling in front of the ear + sign for graph = audiogram; “F” hand shape moving like a “wave” from left to right = frequency). Always use visual aids to supplement your signs. Use pictures of the ear and the “audiogram of familiar sounds” to help explain unfamiliar concepts (e.g., type of hearing loss; sounds the client may never have heard). Working With an Interpreter If necessary, it may be best to obtain the services of a certified interpreter for the deaf. Certified interpreters follow a strict Code of Ethics developed by the Registry of Interpreters of the Deaf (RID). Interpreters can hold different types of RID certification. For example, a CI (Certificate of Interpretation) certificate indicates the interpreter can sign from English to ASL and ASL to English. A CT (Certificate of Transliteration) certificate means the interpreter has been evaluated and is proficient in interpreting from spoken English to a more English-based sign and vice versa. Some RID interpreters hold both certificates. In addition, the National Association of the Deaf also certifies interpreters using a system with varying skill levels. Finally, some states also have certification programs and tests for interpreters. It is essential to find out what is required in your area and obtain the services of qualified interpreters. When working with an interpreter, speech-language or hearing professionals should remember that they are consumers of interpreting services. Be sure to establish what you and your signing client need from the interpreter. For example, some signing individuals have intelligible speech and the interpreter would not need to voice what the signer expresses. Other times the speech may not be intelligible enough and the interpreter may need to voice the intent of the signing client. Finally, it is important to remember that the interpreter is a conduit of information between the clinician and the client. The signing client is addressed directly. The client should not be addressed through the interpreter (i.e., “Tell her…”). Instead the professional should speak directly to the client and allow the interpreter to convey the information. To facilitate this, it is often helpful to position the interpreter next to the hearing or speech-language professional and across from the signing client. This allows for a good visual environment. The signing client can see the interpreter and the professional. During interactions as much eye contact as possible should be maintained with the client. The signing client will need to divide the visual attention between the professional and the interpreter but the professional should remain focused on the client. Definitions American Sign Language (ASL) ASL is a complete, complex language that incorporates the use of hand shapes, position, palm orientation and movement, facial expressions, body movements and postures, and gestures and other visual cues to form its words. It is completely separate from English. ASL has its own rules for grammar, morphology, and sentence order. It is the first language of many North Americans who are deaf and is the fourth most commonly used language in the United States (www.aslinfo.com). Deaf Community/Culture A multi-cultural group of individuals who identify themselves as Deaf, as an ethnic identity, and not a physical condition. The Deaf community is composed of people who use American Sign Language as their primary means of communication. The Deaf Community considers itself a linguistic minority group, a separate entity because of its unique culture, language, and social norms. It is separate from other disability consumer groups by the virtue of communication process, not physical disabilities (www.signmedia.com/info/adc.htm). Finger Spelling (Dactylology) and Invented Sign Systems Finger spelling uses hand shapes to represent the letters of the English alphabet. Invented Sign Systems (Signing Exact English [SEE] and Signed English [SE]) use manual codes (use of the hands) to represent the structure of English. English-Based Signing English-based signing entails signing a combination of both English and ASL features executed in English word order. The language proficiency of the signer will determine the use of the features from either language. It is different from the invented sign systems because it does not follow a clear-cut set of principles. In the past, this type of signing has been labeled pidgin signed English (PSE) and more recently referred to as sign English or conceptually accurate sign English (CASE). It was not intended to model English or be used in an educational context or to teach English. Its purpose is for social communication. Print Publications Resources Paul P. V. (2001). Language and Deafness (3rd ed.). San Diego, CA: Thomson Learning. Google Scholar Vold F. C., Kinsella-Meier M., & Hughes Hilley M. C. (1990). Signing with your client: A practical manual for audiologists and speech/language pathologists. Washington DC: Gallaudet University Press. Google Scholar Web www.nad.org/infocenter/infotogo/dcc/difference.html http://clerccenter.gallaudet.edu/InfoToGo/549.html www.oregon.gov/DHS/odhhs/ References Author Notes Susanne Scott, is a supervising aural rehabilitationist in the department of audiology and speech-language pathology at Gallaudet University. In addition to clinical supervision, she teaches the Clinical Applications of Sign Communication classes to students in the clinical doctorate of audiology program. Contact her by e-mail at [email protected]. James H. Lee, is a clinical supervisor in the department of audiology and speech-language pathology at Gallaudet University. In addition to clinical supervision, he teaches Clinical Applications of Sign Communication to speech-language pathology graduate students. Contact him by e-mail at [email protected]. Advertising Disclaimer | Advertise With Us Advertising Disclaimer | Advertise With Us Additional Resources FiguresSourcesRelatedDetails Volume 8Issue 6April 2003 Get Permissions Add to your Mendeley library History Published in print: Apr 1, 2003 Metrics Current downloads: 4,634 Topicsasha-topicsleader_do_tagasha-article-typesCopyright & Permissions© 2003 American Speech-Language-Hearing AssociationLoading ...

Recognizing dynamic gestures from a continuous data stream has been treated as a difficult task. One challenge is the gesture spotting problem, or segmentation of a gesture pattern from a data stream containing consecutive gestures. Although various types of gesture spotting strategies have been introduced so far, all methods has its own limitations to be applied to real world situations such as sign language conversation. One of the conditions that must be satisfied for practical applications is to detect repetitive gestures for emphasizing a certain meaning.In the view of gesture spotting, repetitive gestures are hard to be detected because similar starting and ending moments are repeated without constraints of the number of repetitions. We introduce a data glove-based real-time dynamic gesture recognition system that covers both repetitive and nonrepetitive gestures. The proposed system consists of three steps: Gesture spotting, gesture sequence compression, and gesture recognition. For gesture spotting, we define a quantity termed as a gesture progress sequence, which quantifies a progress of a gesture using numbers from 0 to 1. Gesture sequence compression removes variations including the number of repetitions that seek to impart the same message. At the gesture recognition step, the compressed gesture patterns are classified. The proposed system was evaluated using 17 American sign language (ASL) gestures and some ASL sentences.

Sign language is a primary mode of communication for individuals with hearing impairments, conveying meaning through hand shapes and hand movements. Contrary to spoken or written languages, sign language relies on the recognition and interpretation of hand gestures captured in video data. However, sign language datasets remain relatively limited compared to those of other languages, which hinders the training and performance of deep learning models. Additionally, the distinct word order of sign language, unlike that of spoken language, requires context-aware and natural sentence generation. To address these challenges, this study applies data augmentation techniques to build a Korean Sign Language dataset and train recognition models. Recognized words are then reconstructed into complete sentences. The sign recognition process uses OpenCV and MediaPipe to extract hand landmarks from sign language videos and analyzes hand position, orientation, and motion. The extracted features are converted into time-series data and fed into a Long Short-Term Memory (LSTM) model. The proposed recognition framework achieved an accuracy of up to 81.25%, while the sentence generation achieved an accuracy of up to 95%. The proposed approach is expected to be applicable not only to Korean Sign Language but also to other low-resource sign languages for recognition and translation tasks.

Sign languages is a critical requirement that helps deaf people to express their needs, feelings and emotions using a variety of hand gestures throughout their daily life. This language had evolved in parallel with spoken languages, however, it do not resemble its counterparts in the same way. Moreover, it is as complex as any other spoken language, as each sign language embodies hundreds of signs, that differs from the next by slight changes in hand shape, position, motion direction, face and body parts contributing to each sign. Unfortunately, sign languages are not globally standardized, where the language differs between countries and has its own vocabulary and varies although they might look similar. Furthermore, publicly available datasets are limited in quality and most of the available translation services are expensive, due to the required skilled human personnel. This paper proposes a deep learning approach for sign language detection that is finely tailored for the Egyptian sign language (special case of the generic sign language). The model is built to harnesses the power of convolutional and recurrent networks by integrating them together to better recognize the sign language spatio-temporal data-feed. In addition, the paper proposes the first Egyptian sign language dataset for emotion words and pronouns. The experimental results demonstrated the proposed approach promising results on the introduced dataset using combined CNN with RNN models.

Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as J and Z. The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8 percent accuracy, 97.4 percent precision, 97.2 percent recall, 97.3 percent F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.

Depression is a disease that destroys the whole life of a person. Through this, that person creates their own zone so that she/he is alone and cannot talk to anyone. Some of the most common types of depression are major bipolar disorder, persistent depressive disorder, depression, and seasonal depressive disorder. Two types of depression that are mainly detected in women are perinatal depression and PMDD or premenstrual dysphoric disorder. Depression makes people lose their identity. When dealing with audio signals, CNN-LSTM models can be used to recognize emotions in speech. CNNs process audio spectrograms to capture acoustic features, and LSTMs capture the dynamics of speech over time, achieving better emotion classification. For tasks like recognizing hand gestures or sign language, CNN-LSTM models can effectively capture both the static hand positions (using CNNs) and the dynamic gestures over time (using LSTMs), leading to improved accuracy. In this paper, we can merge two algorithms to improve the accuracy. Firstly, we can find the individual accuracy of 6 algorithms and compare them with each other, then we can use the proposed model. When combining CNN and LSTM networks, there are two primary methods: Using the CNN output as the input to the LSTM. In this approach, the output of the CNN is passed as input to the LSTM. By doing so, the LSTM can learn and extract features from the input data that have been learned by CNN. Alternatively, the output of the LSTM can be used as the input to the CNN.