A Speech Recognition Model Building Method Combined Dynamic Convolution and Multi-Head Self-Attention Mechanism

Identifying drug-target interactions (DTIs) is an important step in the process of new drug discovery and drug repositioning. Accurate predictions for DTIs can improve the efficiency in the drug discovery and development. Although rapid advances in deep learning technologies have generated various computational methods, it is still appealing to further investigate how to design efficient networks for predicting DTIs. In this study, we propose an end-to-end deep learning method (called MHSADTI) to predict DTIs based on the graph attention network and multi-head self-attention mechanism. First, the characteristics of drugs and proteins are extracted by the graph attention network and multi-head self-attention mechanism, respectively. Then, the attention scores are used to consider which amino acid subsequence in a protein is more important for the drug to predict its interactions. Finally, we predict DTIs by a fully connected layer after obtaining the feature vectors of drugs and proteins. MHSADTI takes advantage of self-attention mechanism for obtaining long-dependent contextual relationship in amino acid sequences and predicting DTI interpretability. More effective molecular characteristics are also obtained by the attention mechanism in graph attention networks. Multiple cross validation experiments are adopted to assess the performance of our MHSADTI. The experiments on four datasets, human, C.elegans, DUD-E and DrugBank show our method outperforms the state-of-the-art methods in terms of AUC, Precision, Recall, AUPR and F1-score. In addition, the case studies further demonstrate that our method can provide effective visualizations to interpret the prediction results from biological insights.

Understanding gene expression processes necessitates the accurate classification and identification of transcription factors, which is supported by high-throughput sequencing technologies. However, these techniques suffer from inherent limitations such as time consumption and high costs. To address these challenges, the field of bioinformatics has increasingly turned to deep learning technologies for analyzing gene sequences. Nevertheless, the pursuit of improved experimental results has led to the inclusion of numerous complex analysis function modules, resulting in models with a growing number of parameters. To overcome these limitations, it is proposed a novel approach for analyzing DNA transcription factor sequences, which is named as DeepCAC. This method leverages deep convolutional neural networks with a multi-head self-attention mechanism. By employing convolutional neural networks, it can effectively capture local hidden features in the sequences. Simultaneously, the multi-head self-attention mechanism enhances the identification of hidden features with long-distant dependencies. This approach reduces the overall number of parameters in the model while harnessing the computational power of sequence data from multi-head self-attention. Through training with labeled data, experiments demonstrate that this approach significantly improves performance while requiring fewer parameters compared to existing methods. Additionally, the effectiveness of our approach is validated in accurately predicting DNA transcription factor sequences.

With the development of the Internet, an increasing number of applications and websites appearing on the Internet allow movie viewers to add comments for movies. If people want to know what per cent of comments are positive or negative, it will cost a large amount human resources and time to check each comment. However, with the help of the transformer model, it will save a large number of human resources and time to finish sentiment classification for long movie comments. The dataset ‘IMDb’ used to train the transformer model is a large Movie Review Dataset for binary sentiment classification of movie reviews. Furthermore, since sentiment classification for movie comments does not require the decoder in the transformer model to predict the next token, the transformer model only need to preserve the part of Positional Encoding, Encoder and Multi-head self-attention mechanism. This paper will investigate how three parameters (the number of layers in encoder, the amount of heads in multi-head self-attention, expansion factor in position-wise fully connected feed-forward network) affect the performance of the transformer model and which set of parameters could allow the transformer to have the best performance. After researching on three parameters, the transformer model used to do sentiment classification for movie comments has the best performance when there are sixteen heads in multi-head self-attention mechanism, four layers in Encoder, and expansion factor in feed-forward network which is position-wise fully connected is four.

Accurate runoff forecasting is of great significance for water resource allocation flood control and disaster reduction. However, due to the inherent strong randomness of runoff sequences, this task faces significant challenges. To address this challenge, this study proposes a new SMGformer runoff forecast model. The model integrates Seasonal and Trend decomposition using Loess (STL), Informer’s Encoder layer, Bidirectional Gated Recurrent Unit (BiGRU), and Multi-head self-attention (MHSA). Firstly, in response to the nonlinear and non-stationary characteristics of the runoff sequence, the STL decomposition is used to extract the runoff sequence’s trend, period, and residual terms, and a multi-feature set based on ‘sequence-sequence’ is constructed as the input of the model, providing a foundation for subsequent models to capture the evolution of runoff. The key features of the input set are then captured using the Informer’s Encoder layer. Next, the BiGRU layer is used to learn the temporal information of these features. To further optimize the output of the BiGRU layer, the MHSA mechanism is introduced to emphasize the impact of important information. Finally, accurate runoff forecasting is achieved by transforming the output of the MHSA layer through the Fully connected layer. To verify the effectiveness of the proposed model, monthly runoff data from two hydrological stations in China are selected, and eight models are constructed to compare the performance of the proposed model. The results show that compared with the Informer model, the 1th step MAE of the SMGformer model decreases by 42.2% and 36.6%, respectively; RMSE decreases by 37.9% and 43.6% respectively; NSE increases from 0.936 to 0.975 and from 0.487 to 0.837, respectively. In addition, the KGE of the SMGformer model at the 3th step are 0.960 and 0.805, both of which can maintain above 0.8. Therefore, the model can accurately capture key information in the monthly runoff sequence and extend the effective forecast period of the model.

Riding feeling recognition based on multi-head self-attention LSTM for driverless automobile

A High-Performance Customer Churn Prediction System Based on Self-Attention

Drug-target interactions provide insight into the drug-side effects and drug repositioning. However, wet-lab biochemical experiments are time-consuming and labor-intensive, and are insufficient to meet the pressing demand for drug research and development. With the rapid advancement of deep learning, computational methods are increasingly applied to screen drug-target interactions. Many methods consider this problem as a binary classification task (binding or not), but ignore the quantitative binding affinity. In this paper, we propose a new end-to-end deep learning method called DeepMHADTA, which uses the multi-head self-attention mechanism in a deep residual network to predict drug-target binding affinity. On two benchmark datasets, our method outperformed several current state-of-the-art methods in terms of multiple performance measures, including mean square error (MSE), consistency index (CI), , and PR curve area (AUPR). The results demonstrated that our method achieved better performance in predicting the drug–target binding affinity.

Traffic flow forecasting is an essential task of an intelligent transportation system (ITS), closely related to intelligent transportation management and resource scheduling. Dynamic spatial-temporal dependencies in traffic data make traffic flow forecasting to be a challenging task. Most existing research cannot model dynamic spatial and temporal correlations to achieve well-forecasting performance. The multi-head self-attention mechanism is a valuable method to capture dynamic spatial-temporal correlations, and combining it with graph convolutional networks is a promising solution. Therefore, we propose a multi-head self-attention spatiotemporal graph convolutional network (MSASGCN) model. It can effectively capture local correlations and potential global correlations of spatial structures, can handle dynamic evolution of the road network, and, in the time dimension, can effectively capture dynamic temporal correlations. Experiments on two real datasets verify the stability of our proposed model, obtaining a better prediction performance than the baseline algorithms. The correlation metrics get significantly reduced compared with traditional time series prediction methods and deep learning methods without using graph neural networks, according to MAE and RMSE results. Compared with advanced traffic flow forecasting methods, our model also has a performance improvement and a more stable prediction performance. We also discuss some problems and challenges in traffic forecasting.

Distant supervision for relation extraction is a task of recognizing semantic relations between entities in a large amount of plain text weakly supervised by external knowledge bases, which can benefit many NLP applications, such as knowledge graph completion and question answering. While it significantly alleviates the expensive cost for data labeling, it severely suffers from noisy labels. In this paper, we propose a Multi-head Self-attention Network (MSNet)-based label denoising method for relation extraction. More specifically, we encode the words, entities and their positions information into contextual embeddings via a multi-head self-attention mechanism, then extract the discriminative sentence features with max pooling operation. MSNet can capture the inherent structure of a sentence and model the relatedness between two words without regard to their distance. Moreover, we adopt a novel label confidence learning method to correct the noisy labels. A latent label is predicted step by step during training as the ground-truth according to a curriculum function of label confidence. This label denoising mechanism gradually incorporates the obtained latent label of easy relation patterns into later latent label prediction of hard patterns, which makes latent label consistent learning more reliable. To verify the effectiveness of our proposed method, in addition to the widely used PCNN-based architecture, we also perform the experiment on BiLSTM model as a comparison. The results demonstrate that our approach can outperform the state-of-the-art systems on the popular evaluation dataset.

Botnets are currently one of the main cyber security threats. In order to enhance the concealment, botnets usually use Domain Generation Algorithm (DGA) to establish communication between bots and command and control servers. Character-based deep learning methods are widely researched in the classification of DGA domains to detect botnets and have achieved good results. But the pronounceable DGA domain detection is still a challenge, since the linguistic statistical characteristics of the pronounceable DGA domains and benign domains are very similar. We propose a multi-head self-attention convolutional network method for DGA domain classification task. We use a shallow convolutional neural network to extract hidden features of domain characters. The multi-head self-attention mechanism with different input values is used to effectively obtain the relationship between the characters and the extracted implicit features, which will help us more effectively distinguish between pronounceable DGA domains and benign domains. Experiments on public data show that our model can effectively detect various types of DGA domains. Especially for the pronounceable DGA domains, our method is significantly better than other detection methods.

Airborne transient electromagnetic (ATEM) surveys provide a fast, flexible approach for identifying conductive metal deposits across a variety of intricate terrains. Nonetheless, the secondary electromagnetic response signals captured by ATEM systems frequently suffer from numerous noise interferences, which impede effective data processing and interpretation. Traditional denoising methods often fall short in addressing these complex noise backgrounds, leading to less-than-optimal signal extraction. To tackle this issue, a deep learning-based denoising network, called BA-ATEMNet, is introduced, using Bayesian learning alongside a multi-head self-attention mechanism to effectively denoise ATEM signals. The incorporation of a multi-head self-attention mechanism significantly enhances the feature extraction capabilities of the convolutional neural network, allowing for improved differentiation between signal and noise. Moreover, the combination of Bayesian learning with a weighted integration of prior knowledge and SNR enhances the model’s performance across varying noise levels, thereby increasing its adaptability to complex noise environments. Our experimental findings indicate that BA-ATEMNet surpasses other denoising models in both single and multiple noise conditions, achieving an average signal-to-noise ratio of 37.21 dB in multiple noise scenarios. This notable enhancement in SNR, compared to the next best model, which achieves an average SNR of 36.10 dB, holds substantial implications for ATEM-based mineral exploration and geological surveys.

Semantic segmentation of moving objects in dynamic backgrounds faces core challenges such as background interference and blurred target features. This study proposes an innovative architecture that integrates Generative Adversarial Network (GAN) with Transformers. The GAN module enhances adaptability to dynamic backgrounds through adversarial training, while the self-attention mechanism in the Transformer captures long-range semantic dependencies. A gated fusion strategy is designed to achieve dynamic balancing of multimodal features. The method employs a conditional GAN to generate dynamic background samples with variations in illumination and motion blur. A Transformer-based encoder-decoder structure is used to model global contextual relationships. A temporal attention module is introduced to incorporate motion vector fields, improving temporal consistency. Additionally, a KL-divergence (KL) constrained semantic consistency loss optimizes the plausibility of generated samples. Experiments are conducted on both a multi-dimensional simulated dataset and the real-world KITTI dataset. Results show that the proposed model achieves an average Intersection over Union (IoU) of 85.6% in standard dynamic scenes, outperforming DeepLabv3 + by 9.2% points. In low-light and high-speed motion scenarios, the robustness index reaches 92.0%, 8.5 points higher than baseline models. Ablation studies demonstrate that removing the Transformer leads to a 6.7% drop in mIoU, while excluding the feature fusion module reduces robustness by 4.0%, confirming the necessity of both components. Temporal analysis reveals that the model maintains a stable performance of 84.5–86.5% over 20-frame sequences, with fluctuation reduced by 63% compared to baseline. The adversarial training improves the model’s adaptability to lighting changes by 5.3%. The multi-head self-attention (MSA) mechanism reduces long-range misclassification by 6.7%. The gated fusion strategy lowers false positive rates in background-disturbed regions by 12.8%. This framework optimizes segmentation through a generator-segmenter feedback loop, effectively balancing dynamic background noise suppression and semantic fidelity. The contributions are threefold: (1) The first semantic segmentation framework to deeply integrate GANs and Transformers. (2) A theoretical model for dynamic feature gating and semantic consistency constraints. (3) A standardized evaluation system covering 10 dynamic background types and five illumination gradients. This study provides key technical support for real-time environmental perception in autonomous driving and intelligent surveillance, advancing both the theoretical and practical frontiers of dynamic scene understanding.

Recently, the Transformer becomes a state-of-the-art architecture in the filed of neural machine translation (NMT). A key point of its high-performance is the multi-head self-attention which is supposed to allow the model to independently attend to information from different representation subspaces. However, there is no explicit mechanism to ensure that different attention heads indeed capture different features, and in practice, redundancy has occurred in multiple heads. In this paper, we argue that using the same global attention in multiple heads limits multi-head self-attention’s capacity for learning distinct features. In order to improve the expressiveness of multi-head self-attention, we propose a novel Mixed Multi-Head Self-Attention (MMA) which models not only global and local attention but also forward and backward attention in different attention heads. This enables the model to learn distinct representations explicitly among multiple heads. In our experiments on both WAT17 English-Japanese as well as IWSLT14 German-English translation task, we show that, without increasing the number of parameters, our models yield consistent and significant improvements (0.9 BLEU scores on average) over the strong Transformer baseline.

Bearing is one of the most prone to failure components, and the most commonly used supporting parts in rotating machinery. There are important economic and safety significance for studying effective bearing fault diagnosis method. Deep learning has been widely applied to intelligent fault diagnosis of rotating machines. In particular, convolutional neural network (CNN) improves diagnostic accuracy due to automatic feature extraction ability. However, CNN cannot fully use the temporal information of the input signals and extract global features. This article presents a new depth model called two-stream multi-head self-attention convolutional neural network (TSMSCNN) for bearing fault diagnosis to solve the above problems. First, we obtain temporal information through the difference of data points in the input signal. Next, the temporal information and original signal are used as input for two CNN branches. Then, the multi-head self-attention (MS) mechanism adaptively aggregates the features and assigns weights to the features extracted by CNN. Finally, the learned high-level representations are fed into the full connect (FC) layer for fault diagnosis. Experimental results prove that the proposed TSMSCNN has higher accuracy than state-of-the-art methods and has excellent noise immunity and transfer performance.

Ship Time Headway (STH) is used in maritime navigation to describe the time interval between the arrivals of two consecutive ships in the same water area. This measurement may offer a straightforward way to gauge the frequency of ship traffic and the likelihood of congestion in a particular area. STH is an important factor in understanding and managing the dynamics of ship movements in busy waterways. This paper introduces a hybrid deep learning method for predicting STH in time domain. The method integrates the Seasonal-Trend Decomposition using Loess (STL), Multi-head Self-Attention (MSA) mechanism into Long Short-Term Memory (LSTM) neural network. The STH dataset was extracted from the Automatic Identification System (AIS) through ship trajectory spatial motion, and the seasonal, trend and residual components of the decomposition were then determined from the STH dataset using the STL algorithms. MSA-LSTM is adopted to comprehensively capture the evolving patterns of STH from the sequence. Comparison studies with existing methods demonstrate the accuracy and robustness of the predictions provided by this method, indicating that the proposed method outperforms other models in terms of prediction performance and learning capabilities. By predicting STH, the method offers potential to assist maritime traffic managers and navigators in assessing ship flow, thereby enabling them to make informed decisions on navigation safety and efficiency.