SMGformer: integrating STL and multi-head self-attention in deep learning model for multi-step runoff forecasting

Multimodal fusion: A study on speech-text emotion recognition with the integration of deep learning

The continuous increase in industrialized sustainable development and energy demand, particularly in the use of steam, highlights the critical importance of efficient energy forecasting for sustainability. While current deep learning models have proven effective, they often involve numerous hyperparameters that are challenging to control and optimize. To address these issues, this research presents an innovative deep learning model, automatically fine-tuned using an improved RIME optimization algorithm (IRIME), with the aim of enhancing accuracy in energy forecasting. Initially, the bidirectional gated recurrent unit (BiGRU) exhibited promising results in prediction tasks but encountered difficulties in handling the complexity of high-dimensional time-series data related to industrial steam. To overcome this limitation, a bidirectional temporal convolutional network (BiTCN) was introduced to more effectively capture long-term dependencies. Additionally, the integration of a multi-head self-attention (MSA) mechanism enabled the model to more accurately identify and predict key features within the data. The IRIME-BiTCN-BiGRU-MSA model achieved outstanding predictive performance, with an R2 of 0.87966, MAE of 0.25114, RMSE of 0.34127, and MAPE of 1.2178, outperforming several advanced forecasting methods. Although the model is computationally complex, its high precision and potential for automation offer a promising tool for high-precision forecasting of industrial steam emissions. This development supports broader objectives of enhancing energy efficiency and sustainability in industrial processes.

Accurate daily runoff forecast is crucial for water resources management, disaster reduction and power generation. However,traditional single-station runoff forecast lacks spatial information.In order to improve the accuracy of runoff forecast, this study presents a spatiotemporal graph convolutional network(STGCN) modeling with reservoir regulation(RR) and seasonal-trend decomposition procedures based on loess (STL) for Hanjiang River Basin(HRB) multi-station daily rrunoff forecast. First, the topological structure of the relationships between stations in the basin is extracted using a graph neural network. Then, a virtual node is constructed according to the principle of water balance, considering the influence of reservoir regulation and storage, and its runoff data is calculated. Finally, the runoff data of both virtual and real points are processed by STL to form a graphical dataset. This watershed runoff prediction model framework is applied to HRB, and various single and mixed models are benchmarked. The results show that the STGCN model outperforms traditional BP and LSTM models. Notably, the proposed STL-RR-STGCN model in this study significantly improves the accuracy of daily runoff predictions compared to the single STGCN model, especially for peak runoff predictions. For example, using the SMAPE index, the average value across 20 stations improved from 0.15 to 0.10, increasing the accuracy of the average measurement error in the HRB by 33.33%.

Accurate interval forecasting of wind power is crucial for ensuring the safe, stable, and cost-effective operation of power grids. In this paper, we propose a hybrid deep learning model for day-ahead wind power interval forecasting. The model begins by utilizing a Gaussian mixture model (GMM) to cluster daily data with similar distribution patterns. To optimize input features, a feature selection (FS) method is applied to remove irrelevant data. The empirical wavelet transform (EWT) is then employed to decompose both numerical weather prediction (NWP) and wind power data into frequency components, effectively isolating the high-frequency components that capture the inherent randomness and volatility of the data. A convolutional neural network (CNN) is used to extract spatial correlations and meteorological features, while the bidirectional gated recurrent unit (BiGRU) model captures temporal dependencies within the data sequence. To further enhance forecasting accuracy, a multi-head self-attention mechanism (MHSAM) is incorporated to assign greater weight to the most influential elements. This leads to the development of a day-ahead wind power interval forecasting model based on GMM-FS-EWT-CNN-BiGRU-MHSAM. The proposed model is validated through comparison with a benchmark forecasting model and demonstrates superior performance. Furthermore, a comparison with the interval forecasts generated using the NPKDE method shows that the new model achieves higher accuracy.

Accurate runoff forecasting is vital yet challenged by the increasing non-stationarity of hydrological systems, which often exceeds the capacity of traditional single models. Ensemble forecasting, as an effective approach, integrates multiple models’ information to enhance forecasting performance and assess uncertainty. However, existing methods (such as Bayesian Model Averaging and BMA) still have limitations in dealing with complex hydrological scenarios, particularly in the construction and optimization of forecast intervals. This paper proposes a novel hydrological ensemble interval forecasting method based on constrained multi-model weight optimization (Constrained Multi-Model Weight Optimization, CMWO). CMWO utilizes a set of heterogeneous deterministic models to generate members, assigns dynamic optimization weight intervals to enhance flexibility, and employs a multi-objective framework to minimize interval width and errors subject to a ≥95% coverage constraint. Taking the Huangjinxia Reservoir in the upper reaches of the Hanjiang River as a case study, the CMWO method was systematically applied and evaluated for decadal-scale runoff forecasting and comprehensively compared with widely used BMA methods and individual models. The results show that CMWO significantly outperforms in improving point forecast accuracy (measured by RMSE, KGE, etc.) and interval forecast quality (evaluated by PICP, PIAW, CRPS, etc.), especially in generating narrower, more informative prediction intervals while ensuring high reliability. The CMWO method proposed in this study provides a competitive new tool for the effective management of forecasting uncertainty in complex hydrological systems.

Accurate photovoltaic (PV) output power forecasting is essential for reliable power system operation, yet rapidly changing meteorological conditions often degrade forecasting accuracy. This study proposes an attention-enhanced bidirectional gated recurrent unit (BiGRU) optimized by an improved Marine Predators Algorithm (IMPA) for PV output power forecasting. Kernel Principal Component Analysis (KPCA) is first employed to extract compact nonlinear representations and suppress redundant features. Then, a dual multi-head self-attention mechanism is integrated before and after the BiGRU layer to strengthen temporal feature learning under fluctuating weather. Finally, the IMPA is designed to improve exploration–exploitation balance and automatically optimize key hyperparameters. Experiments under sunny, cloudy, and rainy conditions demonstrate that IMPA-Att-BiGRU reduces MAE and RMSE by 35.7–58.5% and 22.8–49.1% versus BiGRU, respectively, while increasing R2 by 2.2–4.1 percentage points. Against the best benchmark (LSTM), MAE and RMSE are further reduced by 38.1–49.5% and 33.8–52.4%. Moreover, in a cross-day rolling forecasting test with fivefold results, IMPA-Att-BiGRU achieves 62.4% MAE and 49.3% RMSE reductions over BiGRU, confirming robust performance under long-horizon error accumulation.

The Conformer enhanced Transformer by using convolution serial connected to the multi-head self-attention (MHSA). The method strengthened the local attention calculation and obtained a better effect in auto speech recognition. This paper proposes a hybrid attention mechanism which combines the dynamic convolution CNNs and multi-head self-attention. This study focuses on generating local attention by embedding DY-CNNs in MHSA, followed by parallel computation of the globe and local attention inside the attention layer. Finally, concatenate the result of global and local attention to the output. In the experiments, we use the Aishell-1 (178 hours) Chinese database for training. In the testing folder dev/test, 4.5%/4.8% CER was obtained. The proposed method shows better performance in computation speed and the number of experimental parameters. The results are extremely close to the best result (4.4%/4.7%) of the Conformer.

Understanding gene expression processes necessitates the accurate classification and identification of transcription factors, which is supported by high-throughput sequencing technologies. However, these techniques suffer from inherent limitations such as time consumption and high costs. To address these challenges, the field of bioinformatics has increasingly turned to deep learning technologies for analyzing gene sequences. Nevertheless, the pursuit of improved experimental results has led to the inclusion of numerous complex analysis function modules, resulting in models with a growing number of parameters. To overcome these limitations, it is proposed a novel approach for analyzing DNA transcription factor sequences, which is named as DeepCAC. This method leverages deep convolutional neural networks with a multi-head self-attention mechanism. By employing convolutional neural networks, it can effectively capture local hidden features in the sequences. Simultaneously, the multi-head self-attention mechanism enhances the identification of hidden features with long-distant dependencies. This approach reduces the overall number of parameters in the model while harnessing the computational power of sequence data from multi-head self-attention. Through training with labeled data, experiments demonstrate that this approach significantly improves performance while requiring fewer parameters compared to existing methods. Additionally, the effectiveness of our approach is validated in accurately predicting DNA transcription factor sequences.

With the development of the Internet, an increasing number of applications and websites appearing on the Internet allow movie viewers to add comments for movies. If people want to know what per cent of comments are positive or negative, it will cost a large amount human resources and time to check each comment. However, with the help of the transformer model, it will save a large number of human resources and time to finish sentiment classification for long movie comments. The dataset ‘IMDb’ used to train the transformer model is a large Movie Review Dataset for binary sentiment classification of movie reviews. Furthermore, since sentiment classification for movie comments does not require the decoder in the transformer model to predict the next token, the transformer model only need to preserve the part of Positional Encoding, Encoder and Multi-head self-attention mechanism. This paper will investigate how three parameters (the number of layers in encoder, the amount of heads in multi-head self-attention, expansion factor in position-wise fully connected feed-forward network) affect the performance of the transformer model and which set of parameters could allow the transformer to have the best performance. After researching on three parameters, the transformer model used to do sentiment classification for movie comments has the best performance when there are sixteen heads in multi-head self-attention mechanism, four layers in Encoder, and expansion factor in feed-forward network which is position-wise fully connected is four.

Identifying drug-target interactions (DTIs) is an important step in the process of new drug discovery and drug repositioning. Accurate predictions for DTIs can improve the efficiency in the drug discovery and development. Although rapid advances in deep learning technologies have generated various computational methods, it is still appealing to further investigate how to design efficient networks for predicting DTIs. In this study, we propose an end-to-end deep learning method (called MHSADTI) to predict DTIs based on the graph attention network and multi-head self-attention mechanism. First, the characteristics of drugs and proteins are extracted by the graph attention network and multi-head self-attention mechanism, respectively. Then, the attention scores are used to consider which amino acid subsequence in a protein is more important for the drug to predict its interactions. Finally, we predict DTIs by a fully connected layer after obtaining the feature vectors of drugs and proteins. MHSADTI takes advantage of self-attention mechanism for obtaining long-dependent contextual relationship in amino acid sequences and predicting DTI interpretability. More effective molecular characteristics are also obtained by the attention mechanism in graph attention networks. Multiple cross validation experiments are adopted to assess the performance of our MHSADTI. The experiments on four datasets, human, C.elegans, DUD-E and DrugBank show our method outperforms the state-of-the-art methods in terms of AUC, Precision, Recall, AUPR and F1-score. In addition, the case studies further demonstrate that our method can provide effective visualizations to interpret the prediction results from biological insights.

With the rapid development of smart agriculture and plant protection applications, unmanned aerial vehicles (UAVs) systems that integrate various advanced technologies have become increasingly important for enhancing agricultural production efficiency and reducing operational costs. In order to ensure the safe and stable operation of UAVs, fault diagnosis and reliability evaluation research are very important. Existing fault detection methods often have the problems of unsatisfactory accuracy in low-sample conditions, low computing efficiency and insufficient extraction of temporal characteristics, which limits their reliability and practical applicability. The challenge is further compounded by the scarcity of labeled fault data in real-world applications, underscoring the critical need for low-sample learning methods. To address these issues, this research proposes a multi-scale adaptive state-aware sequence learning framework. The framework adopts a Mamba-enhanced multi-scale temporal network (Mamba-MSTN), which integrates the Multi-Scale Temporal Feature Extraction (MSTFE) module, the adaptive state perception and screening module based on the selection state space model Mamba, and the global dependency modeling module based on Multi-Head Self-Attention (MHSA). Specifically, the MSTFE module hybridizes a 1D Residual Convolutional Neural Network (1D-RCNN) with Bidirectional Gated Recurrent Units (BiGRU) for jointly capturing transient details and long-term trends in flight data, thereby enabling effective multi-granularity temporal feature extraction. Mamba is introduced as an adaptive state perception and screening module. The module adopts an input-dependent state conversion mechanism to dynamically model flight timeseries data in a content-aware way, so as to realize key information filtering. The MHSA mechanism enhances the global dependency representation through parallel multi-subspace modeling adaptive attention to key segments, thus making up for the limitations of local modeling. Comprehensive experiments prove that the proposed method shows high sensitivity and robust generalization in binary and multi-class low-sample fault diagnosis tasks. It is superior to the current mainstream method in terms of accuracy, processing efficiency and resource consumption, and has strong practical application potential.

Reliability assessment plays a pivotal role in engineering: it prevents equipment failures, enables proactive maintenance planning, and sustains long-term, efficient operation. By assessing reliability, we gain deeper insight into performance characteristics, detect potential anomalies early, and act in advance-reducing downtime risk, improving productivity, and supporting sustainable operations. Traditional workflows relied on manual feature selection and extraction, which are labor-intensive. Recent advances in deep learning-such as Convolutional Neural Networks (CNNs)have provided powerful tools for feature extraction, yet they do not fully capture temporal dynamics or handle measurement noise. Moreover, many prior Health Index (HI) labeling schemes used linear, time-based definitions that do not faithfully reflect real degradation trajectories. This study proposes a new framework for HI prediction and lifetime estimation that integrates WaveletKernelNet (WKN), Bidirectional Gated Recurrent Units (BiGRU), Multi-Head Self-Attention (MSA), and a Simplified Swarm Optimization (SSO) algorithm. WKN-BiGRU-MSA extracts deep features from vibration signals and constructs the HI, while SSO optimizes the neural architecture to enhance generalization. We further introduce a novel HI labeling method-a two-stage label inspired by the bathtub curve-to better align model outputs with real-world degradation. Experiments on public rolling-bearing datasets validate the effectiveness of the proposed approach. Overall, our method extracts informative features from vibration data and yields reliable estimates.

Accurate runoff forecasting plays an important role in management and utilization of water resources. This paper investigates the accuracy of hybrid long short-term memory neural network and ant lion optimizer model (LSTM–ALO) in prediction of monthly runoff. As the parameters of long short-term memory neural network (LSTM) have influence on the prediction performance, the parameters of the LSTM are calibrated by using ant lion optimizer. Then the selection of suitable input variables of the LSTM–ALO is discussed for monthly runoff forecasting. Finally, we decompose root mean square error into three parts, which can help us better understanding the origin of differences between the observed and predicted runoff. To test the merits of the LSTM–ALO for monthly runoff forecasting, other models are employed to compare with the LSTM–ALO. The scatter-plots and box-plots are adopted for evaluating the performance of all models. In the case study, simulation results with the historical monthly runoff of the Astor River Basin show that the LSTM–ALO model has higher accuracy than that of other models. Therefore, the proposed LSTM–ALO provides an effective method for monthly runoff forecasting.

A High-Performance Customer Churn Prediction System Based on Self-Attention

Drug-target interactions provide insight into the drug-side effects and drug repositioning. However, wet-lab biochemical experiments are time-consuming and labor-intensive, and are insufficient to meet the pressing demand for drug research and development. With the rapid advancement of deep learning, computational methods are increasingly applied to screen drug-target interactions. Many methods consider this problem as a binary classification task (binding or not), but ignore the quantitative binding affinity. In this paper, we propose a new end-to-end deep learning method called DeepMHADTA, which uses the multi-head self-attention mechanism in a deep residual network to predict drug-target binding affinity. On two benchmark datasets, our method outperformed several current state-of-the-art methods in terms of multiple performance measures, including mean square error (MSE), consistency index (CI), , and PR curve area (AUPR). The results demonstrated that our method achieved better performance in predicting the drug–target binding affinity.