• Home
  • Search
  • Lip-Reading Research Based on ShuffleNet and Attention-GRU
  • Cite Icon1
  • https://doi.org/10.54941/ahfe1004024Copy DOI Icon

Lip-Reading Research Based on ShuffleNet and Attention-GRU

Show More
  • Abstract
  • Literature Map
  • Citations
  • Similar Papers
Abstract

Human-computer interaction has seen a paradigm shift from textual or display-based control towards more intuitive control such as voice, gesture and mimicry. Particularly, speech recognition has attracted a lot of attention because it is the most prominent mode of communication. However, performance of speech recognition systems varies significantly according to sources of background noise, types of talkers and listener's hearing ability. Therefore, lip recognition technology which detects spoken words by tracking speaker's lip movements comes into being. It provides an alternative way for scenes with high background noise and people with hearing impaired problems. Also, lip reading technology has widespread application in public safety analysis, animation lip synthesis, identity authentication and other fields. Traditionally, most work in lipreading was based on hand-engineered features, that were usually modeled by HMM-based pipeline. Recently, deep learning methods are deployed either for extracting 'deep' features or for building end-to-end architectures. In this paper, we propose a neural network architecture combining convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) with a plug-in attention mechanism. The model consists of five parts: (1). Input: We use Dlib library for detecting 68 landmarks of the face, crop the lip area and extract 29 consecutive frames from the video sequence. The frames go through a simple C3D network for generic feature extraction. (2). CNN: with neural networks becoming deeper and deeper, computation complexity increases significantly as well, which motivated the appearance of the lightweight model architecture design. Then a lightweight CNN named ShuffleNet pre-trained on ImageNet dataset is used in our method to perform spatial downsampling of a single image. The ShuffleNet mainly uses two new operations, namely, pointwise group convolution and channel shuffle, which greatly reduce the computational cost without affecting recognition accuracy. (3) CBAM: In the field of image processing, a feature map contains a variety of important information. The traditional convolutional neural network performs convolution in the same way on all channels but importance of information varies greatly depending on different channels. To improve the performance of convolutional neural networks for feature extraction, we utilize an attention mechanism named Convolutional Block Attention Module (CBAM), which is a simple and effective attention module for feedforward convolutional neural networks and contains two independent sub-modules, namely, Channel Attention Module (CAM) and Spatial Attention Module (SAM), which perform Channel and Spatial Attention respectively. (4) RNN: The traditional Recurrent Neural Network (RNN) is mainly used to process sequential data, but with the extension of the RNN network, it may be unable to connect to all related information which may cause key information loss. It cannot solve the long-distance dependence problem and the performance may drop significantly. Due to this shortcoming of the traditional RNN network, we select the GRU network in this paper, which is a variant of the LSTM. It has a simpler structure and better performance than the LSTM neural network. (5) Outputs: Lastly, we pass the result of the backend to SoftMax for classifying the final word. In our experiment, we compare several model architectures and find that our model achieves a comparable accuracy to the current state-of-the-art model at a lower computational cost.

Similar Papers
  • PDF
  • Research Article
  • Citations36

One Spatio-Temporal Sharpening Attention Mechanism for Light-Weight YOLO Models Based on Sharpening Spatial Attention

  • Nov 28, 2021
  • Sensors (Basel, Switzerland)
  • Mengfan Xue +4
  • Research Article
  • Citations24

3D mineral prospectivity modeling in the Sanshandao goldfield, China using the convolutional neural network with attention mechanism

  • Dec 31, 2023
  • Ore Geology Reviews
  • Zhankun Liu +8
  • Conference Article
  • Citations2

Combining Position-aware CNN and RNN for Relation Extraction

  • Dec 20, 2019
  • Meng Ma +2
  • Conference Article

Emotion Recognition of Infant Crying Sounds Using Convolutional Recurrent Neural Network with Multi-Scale Joint Attention Mechanism

  • Jun 23, 2023
  • Yanhao Liu +3
  • Research Article
  • Citations7

SST-CRAM: spatial-spectral-temporal based convolutional recurrent neural network with lightweight attention mechanism for EEG emotion recognition.

  • Apr 30, 2024
  • Cognitive neurodynamics
  • Yingxiao Qiao +1
  • Conference Article
  • Citations5

MEDA: Multi-output Encoder-Decoder for Spatial Attention in Convolutional Neural Networks

  • Nov 01, 2019
  • Huayu Li +1
  • Research Article
  • Citations197

A CBAM Based Multiscale Transformer Fusion Approach for Remote Sensing Image Change Detection

  • Jan 01, 2022
  • IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
  • Wei Wang +3
  • Research Article
  • Citations76

Recurrent convolutional neural network for answer selection in community question answering

  • Apr 11, 2017
  • Neurocomputing
  • Xiaoqiang Zhou +3
  • Research Article
  • Citations226

EMG‐Based Estimation of Limb Movement Using Deep Learning With Recurrent Convolutional Neural Networks

  • Oct 25, 2017
  • Artificial Organs
  • Peng Xia +2
  • Research Article
  • Citations2

A HYBRID DEEP LEARNING APPROACH FOR BREAST CANCER DETECTION USING CNN AND RNN

  • Nov 04, 2024
  • The Bioscan
  • Dr J Gladson Maria Britto +2
  • Research Article
  • Citations85

Remaining useful life prediction for proton exchange membrane fuel cells using combined convolutional neural network and recurrent neural network

  • Oct 15, 2022
  • International Journal of Hydrogen Energy
  • Tabbi Wilberforce +5
  • Conference Article

Enhancing Convolutional Block Attention with Self-Attention on Agricultural Image Classification

  • Nov 03, 2025
  • Arnab Ghosh Chowdhury +2
  • Research Article
  • Citations3

Evaluation of Deep Learning Methods for Pulmonary Disease Classification.

  • Sep 08, 2025
  • Current medical imaging
  • Ajay Pal Singh +2
  • Research Article
  • Citations3

Pengaruh Metode Penyeimbangan Kelas Terhadap Tingkat Akurasi Analisis Sentimen pada Tweets Berbahasa Indonesia

  • Aug 11, 2020
  • Jurnal Teknik Informatika dan Sistem Informasi
  • Ivan Nathaniel Husada +1
  • Conference Article
  • Citations17

A Robust Deep Learning-based Approach for Network Traffic Classification using CNNs and RNNs

  • Mar 23, 2023
  • A Jenefa +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.