End-to-End Audiovisual Speech Recognition

Conventional convolutional neural networks (CNNs) predominantly emphasize spatial features of signals and often fall short in prioritizing sequential features. As the number of layers increases, they are prone to issues such as vanishing or exploding gradients, leading to training instability and subsequent erratic fluctuations in loss values and recognition rates. To address this issue, a novel hybrid model, termed one-dimensional (1D) residual network with attention mechanism and bidirectional gated recurrent unit (BGRU) is developed for rotating machinery fault classification. First, a novel 1D residual network with optimized structure is constructed to obtain spatial features and mitigate the gradient vanishing or exploding. Second, the attention mechanism (AM) is designed to catch important impact characteristics for fault samples. Next, temporal features are mined through the BGRU. Finally, feature information is summarized through global average pooling, and the fully connected layer is utilized to output the final classification result for rotating machinery fault diagnosis. The developed technique which is tested on one set of planetary gear data and three different sets of bearing data, has achieved classification accuracy of 98.5%, 100%, 100%, and 100%, respectively. Compared with other methods, including CNN, CNN-BGRU, CNN-AM, and CNN with an AM-BGRU, the proposed technique has the highest recognition rate and stable diagnostic performance.

The estimation of state of charge (SOC) is a critical issue in the energy management of electric vehicle (EV) power batteries. However, the current accuracy of SOC estimation methods does not meet the requirements of practical applications. Therefore, this study proposes an improved lithium-ion battery SOC estimation method that combines deep residual shrinkage network (DRSN) and bidirectional gated recurrent unit (BiGRU) to enhance the SOC estimation accuracy. First, we insert the bidirectional gated recurrent unit neural network between the global average pooling layer and the output fully connected layer of the deep residual shrinkage network. This improvement enhances the model’s expressiveness, robustness, and data learning effect. Second, we develop a new activation function called “∂_swish” to replace the original ReLU activation function in the deep residual shrinkage network. The ∂_swish activation function improves the accuracy of the deep network model and reduces the risk of overfitting by utilizing its regularization effect. Finally, we conduct experimental tests at three different temperatures using the FUDS driving cycle dataset and the DST-US06-FUDS continuous driving cycle dataset. The algorithm model’s convergence speed is verified by comparing it with other models. The results show that compared to other models, the proposed method significantly improves SOC estimation accuracy at three different temperatures. In addition, the method demonstrates a high convergence speed.

Human history is also the history of the fight against viral diseases. From the eradication of viruses to coexistence, advances in biomedicine have led to a more objective understanding of viruses and a corresponding increase in the tools and methods to combat them. More recently, antiviral peptides (AVPs) have been discovered, which due to their superior advantages, have achieved great impact as antiviral drugs. Therefore, it is very necessary to develop a prediction model to accurately identify AVPs. In this paper, we develop the iAVPs-ResBi model using k-spaced amino acid pairs (KSAAP), encoding based on grouped weight (EBGW), enhanced grouped amino acid composition (EGAAC) based on the N5C5 sequence, composition, transition and distribution (CTD) based on physicochemical properties for multi-feature extraction. Then we adopt bidirectional long short-term memory (BiLSTM) to fuse features for obtaining the most differentiated information from multiple original feature sets. Finally, the deep model is built by combining improved residual network and bidirectional gated recurrent unit (BiGRU) to perform classification. The results obtained are better than those of the existing methods, and the accuracies are 95.07, 98.07, 94.29 and 97.50% on the four datasets, which show that iAVPs-ResBi can be used as an effective tool for the identification of antiviral peptides. The datasets and codes are freely available at https://github.com/yunyunliang88/iAVPs-ResBi.

<p indent=0mm>To address the problem of insufficient detailed semantic information in current global features-based image captioning models, an image Chinese captioning model combining global and local features is proposed. The proposed model adopts the encoder-decoder framework. In the coding stage, the residual networks (ResNet) and Faster R-CNN are used to extract the global and local features of images respectively, improving the model ҆ s utilization of image features at different scales. A bi-directional gated recurrent unit (BiGRU) with embedded visual attention structure and residual connection structure is applied as the decoder (BiGRU with residual connection and attention, BiGRU-RA). The model can adaptively allocate image features and text weights, and improve the mapping relationship between image feature regions and context information. Additionally, the reinforcement learning-based policy gradient is added to improve the loss function of the model and optimize the evaluation criteria CIDEr directly. The training and experiments are conducted on the Chinese captioning dataset of AI challenger. The comparative results show that the proposed model obtained better scores and the generated caption are more accurate and detailed.

Many areas of image identification and classification for medical imaging diagnostics have greatly benefited from deep learning (DL). Diabetic retinopathy (DR) will become the most common cause of blindness worldwide, making diabetes a major threat to public health. This research proposes an automated identification system using deep recurrent neural networks (RNNs) to identify and classify four categories of diabetic eye diseases: DR, cataract, glaucoma, and diabetic macular edema (DME). We use three different model architectures based on RNN and their types, we called our proposed system RNN Diabetic framework. These models are combined with one of the commonly used architectures that support sufficient accuracy and speed for the model which is residual network (ResNet)152V2. The three model architectures are RNN+ResNet152V2, gated recurrent unit (GRU)+ResNet152V2, and bidirectional GRU (Bi-GRU)+ResNet152V2. The proposed models were assessed as collected datasets: DIARETDB0, DIARETDB1, messidor, HEI-MED, ocular, and retina. A full analysis and evaluation of these three deep RNN architectures are presented. The experiments showed that the Bi-GRU+ResNet152V2 model worked better than the other two proposed models. In addition, we compare these three proposed models with the previous studies and find that the proposed Bi-GRU+ResNet152V2 model achieves the highest results with accuracy equal to 99.8%, 98.1% sensitivity, 98.6% specificity, 99.8% precision, 99.8% F1 score, and 99.8% areas under the curve (AUC).

As an important indicator of vehicle systems, tire load is a key factor in the structural design and safety assessment of vehicles. Direct measurement methods for tire loads are expensive and complicated, while conventional load identification methods are limited by low accuracy and poor robustness. This study aimed to propose a radial load identification method for rubber-tired vehicles based on a one-dimensional convolutional neural network (1D CNN) and bidirectional gated recurrent unit (BiGRU). Considering a priori information of the radial load data of tires and based on the observability of the vehicle vibration system, the proposed method selected feature sets and then retained the effective feature subsets through feature selection to construct samples with multiple time steps as input and with a single time step as output for network training. In doing so, the load prediction results were obtained, and the theoretical model was modified by integrating prediction accuracy, generalization performance, and robustness. Compared with traditional algorithms, the proposed method could effectively reduce the error of load identification, improve adaptability under different operating conditions, and handle the measurement error of different noise levels, which are of practical application value in the engineering field.

Automatic Speech Recognition has penetrated deeply into our life. For well-resourced language, it can be considered as solved, but that’s not the case for under-resourced language like Bahasa. Although it’s the 7th most spoken language in the world, the research of speech recognition for Bahasa was still extremely limited, with setting still inconvenient for the real world and industry. This research is an attempt to make a speech recognition model that has applicability to the real world and industry, specifically that supports sentence level input with variable character length with end-to-end training. We built the model using the deep learning approach, specifically utilizing the residual networks and Bi-Directional Gated Recurrent Unit (Bi-GRU). To the best of our knowledge, this is the first Indonesian ASR model that can be trained in an end-to-end manner. Our model surpassed the baseline model on all metrics and achieve competitiveness with the current best result, which used the visual modal, for the dataset even with a more difficult and prone to noise modality like sound.

Sleep staging is the basis of sleep evaluation and a key step in the diagnosis of sleep-related diseases. Despite being useful, the existing sleep staging methods have several disadvantages, such as relying on artificial feature extraction, failing to recognize temporal sequence patterns in the long-term associated data, and reaching the accuracy upper limit of sleep staging. Hence, this paper proposes an automatic Electroencephalogram (EEG) sleep signal staging model, which based on Multi-scale Attention Residual Nets (MAResnet) and Bidirectional Gated Recurrent Unit (BiGRU). The proposed model is based on the residual neural network in deep learning. Compared with the traditional residual learning module, the proposed model additionally uses the improved channel and spatial feature attention units and convolution kernels of different sizes in parallel at the same position. Thus, multiscale feature extraction of the EEG sleep signals and residual learning of the neural networks is performed to avoid network degradation. Finally, BiGRU is used to determine the dependence between the sleep stages and to realize the automatic learning of sleep data staging features and sleep cycle extraction. According to the experiment, the classification accuracy and kappa coefficient of the proposed method on sleep-EDF data set are 84.24% and 0.78, which are respectively 0.24% and 0.21 higher than the traditional residual net. At the same time, this paper also verified the proposed method on UCD and SHHS data sets, and the figure of classification accuracy is 79.34% and 81.6%, respectively. Compared to related existing studies, the recognition accuracy is significantly improved, which validates the effectiveness and generalization performance of the proposed method.

Sleep staging is the basis of sleep evaluation and a key step in the diagnosis of sleep-related diseases. Despite being useful, the existing sleep staging methods have several disadvantages, such as relying on artificial feature extraction, failing to recognize temporal sequence patterns in the long-term associated data, and reaching the accuracy upper limit of sleep staging. Hence, this paper proposes an automatic Electroencephalogram (EEG) sleep signal staging model, which based on Multi-scale Attention Residual Nets (MAResnet) and Bidirectional Gated Recurrent Unit (BiGRU). The proposed model is based on the residual neural network in deep learning. Compared with the traditional residual learning module, the proposed model additionally uses the improved channel and spatial feature attention units and convolution kernels of different sizes in parallel at the same position. Thus, multiscale feature extraction of the EEG sleep signals and residual learning of the neural networks is performed to avoid network degradation. Finally, BiGRU is used to determine the dependence between the sleep stages and to realize the automatic learning of sleep data staging features and sleep cycle extraction. According to the experiment, the classification accuracy and kappa coefficient of the proposed method on sleep-EDF data set are 84.24% and 0.78, which are respectively 0.24% and 0.21 higher than the traditional residual net. At the same time, this paper also verified the proposed method on UCD and SHHS data sets, and the figure of classification accuracy is 79.34% and 81.6%, respectively. Compared to related existing studies, the recognition accuracy is significantly improved, which validates the effectiveness and generalization performance of the proposed method.

N6-methyladenine (6mA) is an important DNA methylation modification that serves a crucial function in various biological activities. Accurate prediction of 6mA sites is essential for elucidating its biological function and underlying mechanism. Although existing methods have achieved great success, there remains a pressing need for improved prediction accuracy and generalization cap ability across diverse species. This study aimed to develop a robust method to address these challenges. We proposed HD-6mAPred, a hybrid deep learning model that combines bidirectional gated recurrent unit (BiGRU), convolutional neural network (CNN) and attention mechanism, along with various DNA sequence coding schemes. Firstly, DNA sequences were encoded using four different ways: one-hot encoding, electron-ion interaction pseudo-potential (EIIP), enhanced nucleic acid composition (ENAC) and nucleotide chemical properties (NCP). Secondly, a hold-out search strategy was employed to identify the optimal features or feature combinations for both BiGRU and CNN. Finally, the attention mechanism was introduced to weigh the importance of features derived from the BiGRU and CNN. A series of experiments on the Rosaceae, rice and Arabidopsis datasets were conducted to demonstrate the superiority of HD-6mAPred. In Rosaceae, the HD-6mAPred model achieved excellent performance: accuracy (ACC) of 0.996, Matthew correlation coefficient (MCC) of 0.993, sensitivity (SN) and specificity (SP) of 0.995 and 0.998, respectively. In rice, the evaluation metrics are 0.952 (ACC), 0.905 (MCC), 0.955 (SN), and 0.949 (SP). In Arabidopsis, the corresponding metrics are 0.937 (ACC), 0.875 (MCC), 0.927 (SN), and 0.948 (SP). Compared to existing methods, these results demonstrate that HD-6mAPred achieves state-of-the-art performance in predicting 6mA sites across three plant species. Furthermore, HD-6mAPred not only improves the accuracy of 6mA site prediction, but also shows excellent generalization capability across species. The source code utilized in this study is publicly accessible at https://doi.org/10.5281/zenodo.15355131.

Depression detection in low-resource languages, such as Bangla, is an emerging challenge in mental health research due to the limited availability of labeled datasets and NLP resources. Bangla, the seventh most spoken language globally, is used by over 230 million people, yet it remains underrepresented in natural language processing (NLP) research compared to high-resource languages like English. In this study, we present a Bangla depression detection dataset collected from social media comprising 18,612 text samples, with 8556 labeled as depressive and the remaining as non-depressive. To effectively classify depressive content, we explore multiple deep learning architectures, including Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), Gated Recurrent Unit (GRU), and Bidirectional GRU (BiGRU). Additionally, we evaluate state-of-the-art transformer-based models such as BanglaBERT, BanglaBERT-Base, ELECTRA, RoBERTa, and BERT. Machine learning models are also employed as baselines for comparison. Our experimental results show that transformer-based models outperform both deep learning and machine learning approaches, with BanglaBERT achieving the highest accuracy of 89.23%, followed closely by BiLSTM with 88%. These findings highlight the effectiveness of transformer-based architectures in depression detection for Bangla text, contributing to improved mental health analysis in low-resource language settings. This study emphasizes the need for further NLP advancements in Bangla to develop AI-driven solutions for early depression detection and intervention.

Epilepsy is a chronic disease of transient brain dysfunction caused by sudden abnormal discharge of brain neurons. It is a common method to detect seizures by EEG signal, but it is cumbersome and easy to make mistakes. To solve the above problems, a two-stage multi-task learning method (residual attention and multi stream bidirectional recurrent network, RMNet) is proposed. In the first stage, the residual attention network composed of the residual neural network and the channel attention mechanism is used to mine the intrinsic correlation of EEG data, which enhances the ability of the network to obtain important information, it can better capture the local dependence of each channel and the short-term time pattern of EEG signal. In the second stage, bi-directional long short-term memory (Bi-LSTM) and bi-directional gate recurrent unit (Bi-GRU) are used for multi-scale fusion to form a multi stream bi-directional recurrent network, which can obtain different time characteristics, it not only improves the robustness of the network, but also improves the ability to extract the time characteristics and the information in the previous and future. Using CHB-MIT scalp EEG database data, the sensitivity, specificity and accuracy were 96.69%, 98.41% and 97.62% respectively. The results show that the method proposed in this paper is superior to the existing methods, and has a significant improvement in the automatic detection performance of epilepsy, which is more in line with the requirements of clinical application.KeywordsEpilepsy detectionEEGResidual attention networkMulti-stream bidirectional recurrent neural network

Lip-reading is a method that focuses on the observation and interpretation of lip movements to understand spoken language. Previous studies have exclusively concentrated on a single variation of residual networks (ResNets). This study primarily aimed to conduct a comparative analysis of several types of ResNets. This study additionally calculates metrics for several word structures included in the GRID dataset, encompassing verbs, colors, prepositions, letters, and numerals. This component has not been previously investigated in other studies. The proposed approach encompasses several stages, namely pre-processing, which involves face detection and mouth location, feature extraction, and classification. The architecture for feature extraction comprises a 3-dimensional convolutional neural network (3D-CNN) integrated with ResNets. The management of temporal sequences during the classification phase is accomplished through the utilization of the bidirectional gated recurrent units (Bi-GRU) model. The experimental results demonstrated a character error rate (CER) of 14.09% and a word error rate (WER) of 28.51%. The combination of 3D-CNN ResNet-34 and Bi-GRU yielded superior outcomes in comparison to ResNet-18 and ResNet-50. The correlation between increased network depth and enhanced performance in lip-reading models was not consistently observed. Nevertheless, the incorporation of additional trained parameters offers certain benefits. Moreover, it has demonstrated superior levels of precision in comparison to human professionals in the task of distinguishing diverse word structures. Doi: 10.28991/HIJ-2023-04-02-010 Full Text: PDF

Cardio-respiratory disorders account for approximately 30% of total mortality in India, with their incidence steadily rising due to factors such as increasing urban pollution, sedentary lifestyles, and delayed clinical diagnosis, as highlighted by the Indian Council of Medical Research. Conventional auscultation using stethoscopes is highly dependent on the clinician’s expertise and experience, which can lead to variability and potential inaccuracies in diagnosis. To address these limitations, this study presents an automated cardio-respiratory sound classification framework designed to enhance early detection and diagnostic consistency through machine learning. The proposed system incorporates a user-friendly Graphical User Interface (GUI) with role-based functionality, allowing administrators to manage model training while enabling end-users to perform real-time predictions. Audio signals are preprocessed using the Librosa library to extract informative acoustic features such as Mel-Frequency Cepstral Coefficients (MFCC), Chroma features, and Mel Spectrogram representations, ensuring comprehensive characterization of both heart and lung sounds. A range of machine learning models, including Quadratic Discriminant Analysis (QDA), Gradient Boosting Classifier (GBC), Naïve Bayes Classifier (NBC), and Logistic Regression Classifier (LRC), are developed and comparatively evaluated. Furthermore, a hybrid deep learning architecture combining a Bi-directional Convolutional Neural Network (BiCNN) with a Bi-directional Gated Recurrent Unit (BiGRU) is introduced to effectively capture both spatial (spectral) and temporal dependencies within the audio data. The system is capable of performing dual classification by identifying both heart sound types and lung sound types simultaneously. Its performance is rigorously assessed using standard evaluation metrics, including accuracy, precision, recall, F1-score, and Receiver Operating Characteristic–Area Under Curve (ROCAUC), demonstrating its potential as a reliable and efficient tool for clinical decision support.

BackgroundInformation transmission and processing in the nervous system has stochastic nature. Multiple factors contribute to neuronal trial-to-trial variability. Noise and variations are introduced by the processes at the molecular and cellular level (thermal noise, channel current noise, membrane potential variations, biochemical and diffusion noise at synapses etc). The stochastic processes are affected by different physical (temperature, electromagnetic field) and chemical (drugs) factors. The aim of this study was experimental investigation of hypotheses that increase in the noise level in the brain affects processing of visual information. Change in the noise level was introduced by an external factor producing excess noise in the brain.MethodsAn exposure to 450 MHz low-frequency modulated microwave radiation was applied to generate excess noise. Such exposure has been shown to increase diffusion, alter membrane resting potential, gating variables and intracellular Calcium efflux. Nine healthy volunteers passed the experimental protocol at the lower (without microwave) and the higher (with microwave) noise level. Two photos (visual stimuli) of unfamiliar, young male faces were presented to the subjects, one picture after another. The task was to identify later the photos from a group of six photos and to decide in which order they were presented. Each subject had a total of eight sessions at the lower and eight at the higher noise level. Each session consisted of 50 trials; altogether a subject made 800 trials, 400 at the lower and 400 at the higher noise level. Student t-test was applied for statistical evaluation of the results.ResultsCorrect recognition of both stimuli in the right order was better at the lower noise level. All the subjects under investigation showed higher numbers of right answers in trials at the lower noise level. Average number of correct answers from n=400 trials with microwave exposure was 50.3, without exposure 54.4, difference 7.5%, p<0.002. No difference between results at the lower and the higher noise level was revealed in the case of only partly correct or incorrect answers.ConclusionsOur experimental results showed that introduced excess noise reduced significantly ability of the nervous system in correct processing of visual information.