DeepCAC: a deep learning approach on DNA transcription factors classification based on multi-head self-attention and concatenate convolutional neural network

Drug-target interactions provide insight into the drug-side effects and drug repositioning. However, wet-lab biochemical experiments are time-consuming and labor-intensive, and are insufficient to meet the pressing demand for drug research and development. With the rapid advancement of deep learning, computational methods are increasingly applied to screen drug-target interactions. Many methods consider this problem as a binary classification task (binding or not), but ignore the quantitative binding affinity. In this paper, we propose a new end-to-end deep learning method called DeepMHADTA, which uses the multi-head self-attention mechanism in a deep residual network to predict drug-target binding affinity. On two benchmark datasets, our method outperformed several current state-of-the-art methods in terms of multiple performance measures, including mean square error (MSE), consistency index (CI), , and PR curve area (AUPR). The results demonstrated that our method achieved better performance in predicting the drug–target binding affinity.

2'-O-methylation (2OM) of ribose is a widespread RNA modification that significantly impacts RNA stability, structure, and function. Accurately predicting 2OM sites is crucial for understanding RNA's biological functions and related pathologies. Traditional detection methods pose challenges such as resource intensiveness, potential RNA sample damage, and high costs. However, recent advancements in machine learning, particularly deep learning techniques, offer rapid and cost-effective prediction solutions. In this study, we introduce DeepR2OM, a novel method integrating feature selection and deep learning for 2OM sites prediction. DeepR2OM encodes sequences using eight RNA descriptors, employs feature selection algorithms to reduce dimensions, and then utilizes a deep learning network for training. After evaluating various deep learning architectures, we selected Convolutional Neural Network (CNN), Multi-Head Self-Attention mechanism, and Deep Neural Network (DNN) as our final prediction models. Experimental results demonstrate DeepR2OM's effectiveness, achieving 87.1% accuracy (ACC), 85.5% recall rate (Recall), 87.9% precision (PRE), and a Matthews correlation coefficient (MCC) of 75.7% on an independent test set. This tool serves as a valuable resource for exploring the functional and bioinformatic aspects of 2OM sites.

Bearing is one of the most prone to failure components, and the most commonly used supporting parts in rotating machinery. There are important economic and safety significance for studying effective bearing fault diagnosis method. Deep learning has been widely applied to intelligent fault diagnosis of rotating machines. In particular, convolutional neural network (CNN) improves diagnostic accuracy due to automatic feature extraction ability. However, CNN cannot fully use the temporal information of the input signals and extract global features. This article presents a new depth model called two-stream multi-head self-attention convolutional neural network (TSMSCNN) for bearing fault diagnosis to solve the above problems. First, we obtain temporal information through the difference of data points in the input signal. Next, the temporal information and original signal are used as input for two CNN branches. Then, the multi-head self-attention (MS) mechanism adaptively aggregates the features and assigns weights to the features extracted by CNN. Finally, the learned high-level representations are fed into the full connect (FC) layer for fault diagnosis. Experimental results prove that the proposed TSMSCNN has higher accuracy than state-of-the-art methods and has excellent noise immunity and transfer performance.

Identifying drug-target interactions (DTIs) is an important step in the process of new drug discovery and drug repositioning. Accurate predictions for DTIs can improve the efficiency in the drug discovery and development. Although rapid advances in deep learning technologies have generated various computational methods, it is still appealing to further investigate how to design efficient networks for predicting DTIs. In this study, we propose an end-to-end deep learning method (called MHSADTI) to predict DTIs based on the graph attention network and multi-head self-attention mechanism. First, the characteristics of drugs and proteins are extracted by the graph attention network and multi-head self-attention mechanism, respectively. Then, the attention scores are used to consider which amino acid subsequence in a protein is more important for the drug to predict its interactions. Finally, we predict DTIs by a fully connected layer after obtaining the feature vectors of drugs and proteins. MHSADTI takes advantage of self-attention mechanism for obtaining long-dependent contextual relationship in amino acid sequences and predicting DTI interpretability. More effective molecular characteristics are also obtained by the attention mechanism in graph attention networks. Multiple cross validation experiments are adopted to assess the performance of our MHSADTI. The experiments on four datasets, human, C.elegans, DUD-E and DrugBank show our method outperforms the state-of-the-art methods in terms of AUC, Precision, Recall, AUPR and F1-score. In addition, the case studies further demonstrate that our method can provide effective visualizations to interpret the prediction results from biological insights.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Efficient jamming recognition capability is a prerequisite for radar anti-jamming and can enhance the survivability of radar in electronic warfare. Traditional recognition methods based on manually designed feature parameters have found it difficult to cope with the increasingly complex electromagnetic environment, and research combining deep learning to achieve jamming recognition is gradually increasing. However, existing research on radar jamming recognition based on deep learning can ignore the global representation in the jamming time–frequency domain data, while not paying enough attention to the problem of lightweighting the recognition network itself. Therefore, this paper proposes a lightweight jamming recognition network (JR-TFViT) that can fuse the global representation of jamming time–frequency domain data while combining the advantages of the Vision Transformer and a convolutional neural network (CNN). The global representation and local information in the jamming time–frequency data are fused with the assistance of the multi-head self-attention (MSA) mechanism in the transformer to improve the feature extraction capabilities of the recognition network. The model’s parameters are further decreased by modifying the standard convolutional operation mechanism and substituting the convolutional operation needed by the network with Ghost convolution, which has less parameters. The experimental results show that the JR-TFViT requires fewer model parameters while maintaining higher recognition performance than mainstream convolutional neural networks and lightweight CNNs. For 12 types of radar jamming, the JR-TFViT achieves 99.5% recognition accuracy at JNR = −6 dB with only 3.66 M model parameters. In addition, 98.9% recognition accuracy is maintained when the JR-TFViT parameter number is further compressed to 0.67 M.

Botnets are currently one of the main cyber security threats. In order to enhance the concealment, botnets usually use Domain Generation Algorithm (DGA) to establish communication between bots and command and control servers. Character-based deep learning methods are widely researched in the classification of DGA domains to detect botnets and have achieved good results. But the pronounceable DGA domain detection is still a challenge, since the linguistic statistical characteristics of the pronounceable DGA domains and benign domains are very similar. We propose a multi-head self-attention convolutional network method for DGA domain classification task. We use a shallow convolutional neural network to extract hidden features of domain characters. The multi-head self-attention mechanism with different input values is used to effectively obtain the relationship between the characters and the extracted implicit features, which will help us more effectively distinguish between pronounceable DGA domains and benign domains. Experiments on public data show that our model can effectively detect various types of DGA domains. Especially for the pronounceable DGA domains, our method is significantly better than other detection methods.

Seismic data suffer from under-sampling along spatial dimensions due to physical and economic limitations during acquisition, which can adversely impact subsequent imaging processes. Various approaches have been proposed to address this issue, including rank-reduction, transform, and deep learning (DL) methods for reconstructing missing traces in observed seismic data. While many DL-based reconstruction methods employ convolutional neural networks (CNNs) as their core components in supervised or unsupervised settings, the accuracy of CNN-based DL reconstruction methods can still be enhanced due to the inherent localization of convolutional operators. In this study, we present an innovative method, termed Attentive CNN (ACNet), that incorporates a transformer model with a multi-head self-attention mechanism into the CNN framework for unsupervised seismic data reconstruction. ACNet consists of an encoder built with the transformer model and a CNN decoder, enabling an iterative reconstruction process within the proposed unsupervised DL-based approach. We compare the performance of our method with that of the damped rank-reduction (DRR) method using synthetic and field data examples. The results demonstrate that our proposed method exhibits superior signal reconstruction and random noise suppression capabilities compared to the DRR method.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Egg defect detection is essential for ensuring food safety and quality in the egg production industry. This study introduces a novel approach for egg defect detection by integrating a multi-head self-attention (MHSA) mechanism into the AlexNet convolutional neural network (CNN) to overcome the limitations of traditional CNNs in capturing long-range dependencies and global context. A machine vision system equipped with a constant-speed rotating mechanism was developed to capture multi-view images of each egg’s surface. Each egg was imaged from six angles, generating a dataset of 2400 images categorized into four classes: bloodstained, cracked, dirty, and intact. The collected images were pre-processed through segmentation and cropping before being fed into the MHSA-AlexNet architecture. The extracted feature maps were flattened and passed through an MHSA layer with eight attention heads, followed by a fully connected layer and Softmax classifier. The proposed MHSA-AlexNet model achieved an average accuracy of 0.9847, outperforming the baseline AlexNet and surpassing other state-of-the-art CNNs. The model also surpassed conventional machine learning models, including Naïve Bayes, Random Forest, and Support Vector Machine, by 36.35–66.81%. Comparison with prior studies demonstrated that MHSA-AlexNet achieved superior accuracy, emphasizing the effectiveness of integrating self-attention mechanisms for egg defect detection.

Deep learning (DL) has shown explosive growth in its application to bioinformatics and has demonstrated thrillingly promising power to mine the complex relationship hidden in large-scale biological and biomedical data. A number of comprehensive reviews have been published on such applications, ranging from high-level reviews with future perspectives to those mainly serving as tutorials. These reviews have provided an excellent introduction to and guideline for applications of DL in bioinformatics, covering multiple types of machine learning (ML) problems, different DL architectures, and ranges of biological/biomedical problems. However, most of these reviews have focused on previous research, whereas current trends in the principled DL field and perspectives on their future developments and potential new applications to biology and biomedicine are still scarce. We will focus on modern DL, the ongoing trends and future directions of the principled DL field, and postulate new and major applications in bioinformatics.

PurposeThe aim of this study was to analyse and review deep learning convolutional neural networks for detecting and diagnosing early-stage dental caries on periapical radiographs.Materials and MethodsIn order to conduct this review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. Studies published from 2015 to 2021 under the keywords (deep convolutional neural network) AND (caries), (deep learning caries) AND (convolutional neural network) AND (caries) were systematically reviewed.ResultsWhen dental caries is improperly diagnosed, the lesion may eventually invade the enamel, dentin, and pulp tissue, leading to loss of tooth function. Rapid and precise detection and diagnosis are vital for implementing appropriate prevention and treatment of dental caries. Radiography and intraoral images are considered to play a vital role in detecting dental caries; nevertheless, studies have shown that 20% of suspicious areas are mistakenly diagnosed as dental caries using this technique; hence, diagnosis via radiography alone without an objective assessment is inaccurate. Identifying caries with a deep convolutional neural network-based detector enables the operator to distinguish changes in the location and morphological features of dental caries lesions. Deep learning algorithms have broader and more profound layers and are continually being developed, remarkably enhancing their precision in detecting and segmenting objects.ConclusionClinical applications of deep learning convolutional neural networks in the dental field have shown significant accuracy in detecting and diagnosing dental caries, and these models hold promise in supporting dental practitioners to improve patient outcomes.

Remote sensing image dehazing is a well-known remote sensing image processing task focused on restoring clean images from hazy images. The Transformer network, based on the self-attention mechanism, has demonstrated remarkable advantages in various image restoration tasks, due to its capacity to capture long-range dependencies within images. However, it is weak at modeling local context. Conversely, convolutional neural networks (CNNs) are adept at capturing local contextual information. Local contextual information could provide more details, while long-range dependencies capture global structure information. The combination of long-range dependencies and local context modeling is beneficial for remote sensing image dehazing. Therefore, in this paper, we propose a CNN-based adaptive local context enrichment module (ALCEM) to extract contextual information within local regions. Subsequently, we integrate our proposed ALCEM into the multi-head self-attention and feed-forward network of the Transformer, constructing a novel locally enhanced attention (LEA) and a local continuous-enhancement feed-forward network (LCFN). The LEA utilizes the ALCEM to inject local context information that is complementary to the long-range relationship modeled by multi-head self-attention, which is beneficial to removing haze and restoring details. The LCFN extracts multi-scale spatial information and selectively fuses them by the the ALCEM, which supplements more informative information compared with existing regular feed-forward networks with only position-specific information flow. Powered by the LEA and LCFN, a novel Transformer-based dehazing network termed LCEFormer is proposed to restore clear images from hazy remote sensing images, which combines the advantages of CNN and Transformer. Experiments conducted on three distinct datasets, namely DHID, ERICE, and RSID, demonstrate that our proposed LCEFormer achieves the state-of-the-art performance in hazy scenes. Specifically, our LCEFormer outperforms DCIL by 0.78 dB and 0.018 for PSNR and SSIM on the DHID dataset.

Deep learning (DL) is now generally acknowledged as the benchmark and evolution in machine learning (ML) fields. Further more, it has steadily suited the most extensively in computational techniques for ML, delivering excellent outcomes in several challenging intellectual works that have equal or still outperformed human ability. DL has the advantage of learning from enormous amounts of data. Here, the DL approach has been applied to classify thunderstorms and non-thunderstorms with a meteorological dataset. Daily observational hourly data sets from 2016 to 2021 have been used to classify the incidence of thunderstorms and non-thunderstorms. Three DL approaches are used: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM) to classify thunderstorm incidences. These DL techniques are compared with conventional ML techniques, showing that DL approaches outperformed ML. The CNN method provides the highest performance with 92.01% accuracy among all DL and ML approaches.

OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning