Industrial water withdrawal prediction using multi-head attention encoder model

To overcome the limitations of existing deep learning-based image recognition approaches for diagnosing the operating conditions of pumping wells–namely, slow processing speed and high parameter complexity–this paper proposes a lightweight diagnostic method based on a one-dimensional convolutional neural network (1D CNN). Unlike traditional image-based techniques, the proposed method eliminates the need to convert raw data into dynamometer card images, thereby reducing computational overhead and significantly decreasing the number of model parameters. The architecture integrates 1D CNN with a bidirectional long short-term memory (BiLSTM) network to effectively capture both local patterns and global temporal dependencies inherent in different operating states of pumping wells. Additionally, a multi-head attention (MHA) mechanism is incorporated to enhance the representation of critical time points, thereby improving the model's sensitivity to key features and its overall classification performance. Experiments conducted on an augmented dataset–generated via a simulation-based algorithm–demonstrate that the proposed 1DCNN-BiLSTM-MHA model achieves a classification accuracy of 99.22%. Experimental results demonstrate that the proposed model possesses strong feature learning capabilities. Comparative analysis with several baseline models highlights its superior diagnostic performance and reliability, along with reduced parameter count and improved computational speed.

To better understand and evaluate the GNSS positioning performance, it is convenient to adopt corresponding measures to reduce the impact of errors on positioning. A GNSS positioning error estimation scheme based on an improved multi-layer perceptron model is proposed. The multi-head attention mechanism and gating operation are integrated into the multi-layer perceptron model to adaptively select and filter features, enhancing the model’s ability to understand input features. First, the original positioning error of the satellite is obtained through the Kalman filter positioning method. The data are then preprocessed to extract available features. Finally, the features are input into the constructed model for training and testing to obtain the estimated positioning error value. Two types of comparative experiments were completed. The performance of the presented model is evaluated by the root mean square error. Experimental results show that the proposed method performs well in terms of performance indicators, and has obvious advantages over other state-of-the-art methods. In particular, the root mean square error of the presented method in the first dataset is 0.239 m, which is 39.2% and 17% lower than the current state-of-the-art long short-term memory network and convolutional neural network, respectively. The presented method can provide higher-precision estimated values for studying the GNSS positioning error estimation problem.

Knowledge reasoning technology is a method proposed to solve missing problems in knowledge graphs, which has been developing in recent years. In order to solve the problems of low accuracy, poor interpretability, and weak applicability in knowledge reasoning, a knowledge reasoning model called MLALogic is proposed, which combines HousE with multi-head latent attention mechanism. It consists of a rule generator with multi-head latent attention mechanism and a rule predictor with HousE embedding. The rule generator generates the rules required for reasoning and passes them into the predictor, which updates and then obtains scores for different rules, After that, the generator and predictor are continuously trained and optimised by the EM algorithm. Specifically, The model employs a multi-head latent attention mechanism to select more relevant relationships as rules, thereby improving model accuracy and reducing training costs. Additionally, the integration of HousE enhances flexibility in handling complex and multi-relational data.

This study proposes a deep learning method based on ResNet, Transformer, and Multi-Head Attention for food quality inspection. With the growing severity of food safety issues, traditional food quality inspection methods have significant shortcomings in efficiency and accuracy. Therefore, there is an urgent need to develop automated, efficient, and accurate food quality inspection technologies. Our method combines a deep convolutional neural network (ResNet), a self-attention mechanism (Transformer), and a multi-head attention mechanism (Multi-Head Attention). It effectively extracts detailed features from food images while enhancing attention to global image information and important regions, effectively identifying mold, contamination, and spoilage in food. Experimental results demonstrate that this method outperforms traditional deep learning models in metrics such as precision, recall, and F1 score. Ablation experiments also validate the contribution of each module to the performance improvement. This model provides a more accurate and automated solution for food quality inspection and has broad application prospects.

Data-driven forecasting framework for daily reservoir inflow time series considering the flood peaks based on multi-head attention mechanism

RPConvformer: A novel Transformer-based deep neural networks for traffic flow prediction

As drilling exploration targets become increasingly complex, the rising incidence of lost circulation necessitates rapid and reliable early warning systems for operational safety. However, existing models often suffer from limited long-term prediction capabilities, severe time lags, and insufficient accuracy. To address these challenges, this study proposes a lost circulation warning model (PatchTST+DT), which fuses PatchTST (Patch Time Series Transformer) with a physics-informed dynamic threshold model. First, real-time logging data from South China Sea Block X was preprocessed, and eight key features were selected via L1-regularized logistic regression. Then, PatchTST’s unique patching design and multi-head attention mechanism were leveraged to efficiently capture global dependencies in time series data; the physics-informed threshold model was incorporated as a constraint to boost model accuracy. Finally, the proposed model was comprehensively compared with baseline models including LSTM (Long Short-Term Memory), Transformer, and standalone PatchTST. Based on cross-validation results, the proposed PatchTST+DT outperforms all baseline models and exhibits the best performance in long-term prediction (80 steps), with an average ACC (Accuracy) of 0.9320, FPR (False Positive Rate) of 0.0629, FNR (False Negative Rate) of 0.0773, F 1 + score of 0.9065 and PR-AUC (Precision–Recall Area Under Curve) of 0.8950. All metrics show a standard deviation within ±0.03, which confirms the model’s robust generalization ability and stability across different data subsets. In field validation, PatchTST+DT successfully issues early warnings for lost circulation events up to 6 min in advance, markedly improving alert timeliness and reliability while enhancing drilling operational safety.

Pine wilt disease (PWD) can cause destructive death in many species of pine trees within a short period. The recognition of infected pine trees in unmanned aerial vehicle (UAV) forest images is a key technology for automatic monitoring and early warning of pests. This paper collected UAV visible and multispectral images of Korean pines (Pinus koraiensis) and Chinese pines (P. tabulaeformis) infected by PWD and divided the PWD infection into early, middle, and late stages. With the open-source annotation tool, LabelImg, we labeled the category of infected pine trees at each stage. After coordinate-correction preprocessing of the ground truth, the Korean pine and Chinese pine datasets were established. As a means of detecting infected pine trees of PWD and determining different infection stages, a multi-band image-fusion infected pine tree detector (MFTD) based on deep learning was proposed. Firstly, the Halfway Fusion mode was adopted to fuse the network based on four YOLOv5 variants. Simultaneously, the Backbone network was initially designed as a dual branching network that includes visible and multispectral subnets. Moreover, the features of visible and multispectral images were extracted. To fully utilize the features of visible and multispectral images, a multi-band feature fusion transformer (MFFT) with a multi-head attention mechanism and a feed-forward network was constructed to enhance the information correlation between visible and multispectral feature maps. Finally, following the MFFT module, the two feature maps were fused and input into Neck and Head to predict the categories and positions of infected pine trees. The best-performing MFTD model achieved the highest detection accuracy with mean average precision values (mAP@50) of 88.5% and 86.8% on Korean pine and Chinese pine datasets, respectively, which improved by 8.6% and 10.8% compared to the original YOLOv5 models trained only with visible images. In addition, the average precision values (AP@50) are 87.2%, 93.5%, and 84.8% for early, middle, and late stages on the KP dataset and 81.2%, 92.9%, and 86.2% on the CP dataset. Furthermore, the largest improvement is observed in the early stage with 14.3% and 11.6%, respectively. The results show that MFTD can accurately detect the infected pine trees, especially those at the early stage, and improve the early warning ability of PWD.

As one of the core tasks of natural language processing technology, text classification methods general-ly face the problems of insufficient global semantic capture and limited feature focusing ability when processing long texts or complex semantics. To address this issue, a deep learning model that integrates improved convolutional neural networks, unbalanced bidirectional long short-term memory networks, and multi-head attention mechanisms is proposed. Utilizing an improved bidirectional long short-term memory network to capture global semantic information, while dynamically focusing on key features through a multi head attention mechanism to enhance the model's adaptability to classification tasks. The performance of the model is validated through experiments on AG News (short text) and IMDb (long text) datasets. The results show that in short text classification, the proposed method has an accu-racy rate of 96% and a classification error rate of only 1.46%. In the task of long text classification, the method proposed in the study has a product under the curve of 0.98. In adversarial attack testing, the accuracy rates of adversarial samples generated by different methods are 92.85% and 90.63%, respec-tively, with the lowest robustness degradation rates of 3.72% and 5.49%, respectively. In cross domain generalization testing, it shows the least classification errors and superior cross domain adaptability. These results validate the high performance, robustness, and wide applicability of the method. The re-search indicates that this approach can validly improve the performance of text classification and pro-vide new solutions for natural language processing related tasks in long text and multi-category scenar-ios.

This study delves into the efficacy of various machine learning and statistical models that have captured the attention of financial analysts. Two of them, Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are variations of Recurrent Neural Networks while the Autoregressive Integrated Moving Average (ARIMA) is a statistical model. These models will be used to forecast stock market data across different economic sectors. In the dynamic landscape of financial markets, accurate forecasting is crucial. This research paper contributes to quantitative finance by conducting a comprehensive comparative analysis of these models on historical stock market data from three sectors: extraction, manufacturing, and service (which are considered the primary, secondary, and tertiary sectors of the economy respectively). The models' performances are evaluated using mean squared error (MSE) on six selected stocks representing these sectors. Results reveal the power of recurrent neural networks in capturing intricate patterns. Moreover, the results will explore whether or not the efficacy of each model is impacted by the sector of the economy that it is forecasting data for.

In the last few years, the application of adhesive joints has grown significantly. Adhesive joints are often affected by a specific type of defect known as weak adhesion, which can only be effectively detected through destructive tests. In this paper, we propose nondestructive testing techniques to detect weak adhesion. These are based on Lamb wave (LW) data and artificial intelligence algorithms. A dataset consisting of simulated LW time series extracted from single-lap joints (SLJs) subjected to multiple levels of weak adhesion was generated. The raw time series were pre-processed to avoid numerical saturation and to remove outliers. The processed data were then used as the input to different artificial intelligence algorithms, namely feedforward neural networks (FNNs), long short-term memory (LSTM) networks, gated recurrent unit (GRU) networks, and convolutional neural networks (CNNs), for their training and testing. The results showed that all algorithms were capable of detecting up to 20 different levels of weak adhesion in SLJs, with an overall accuracy between 97% and 99%. Regarding the training time, the FNN emerged as the most-appropriate. On the other hand, the GRU showed overall faster learning, being able to converge in less than 50 epochs. Therefore, the FNN and GRU presented the best accuracy and had relatively acceptable convergence times, making them the most-suitable choices. The proposed approach constitutes a new framework allowing the creation of standardized data and optimal algorithm selection for further work on nondestructive damage detection and localization in adhesive joints.

How to enable the computer to accurately analyze the emotional information and story background of characters in Qin opera is a problem that needs to be studied. To promote the artistic inheritance and cultural emotion color dissemination of Qin opera, an emotion analysis model of Qin opera based on attention residual network (ResNet) is presented. The neural network is improved and optimized from the perspective of the model, learning rate, network layers, and the network itself, and then multi-head attention is added to the ResNet to increase the recognition ability of the model. The convolutional neural network (CNN) is optimized from the internal depth, and the fitting ability and stability of the model are enhanced through the ResNet model. Combined with the attention mechanism, the expression of each weight information is strengthened. The multi-head attention mechanism is introduced in the model and a multi-head attention ResNet, namely, MHAtt_ResNet, is proposed. The network structure can effectively identify the features of the spectrogram, improve the weight information of spectrogram features, and deepen the relationship between distant information in long-time series. Through experiments, the proposed model has high emotional classification accuracy for Qin opera, and with the increase of the number of data sets, the model will train a better classification effect.

The development of social media has changed the way the public expresses political opinions, especially regarding the evaluation of President Joko Widodo’s (Jokowi) leadership after his term. Platform X (formerly Twitter) has become the primary source of public opinion data, but the use of informal language and sarcasm makes accurate sentiment analysis challenging. This study creates a sentiment analysis model that uses deep learning with a CNN-BiLSTM structure and a multi-head attention mechanism. The dataset consists of 52,643 tweets that have been labeled and embedded using IndoBERT. To address class imbalance, the SMOTE method was applied to the training data, enabling the model to better learn from minority classes. The results indicate that the model achieves a high accuracy of 98.78%, with an average precision, recall, and F1-score of 0.98. These findings indicate that the model is not only accurate but also reliable in distinguishing each sentiment class. A comparison with other model variants suggests that the complete combination of CNN-BiLSTM and Multi-Head Attention delivers the best performance, although the improvement is relatively small.

With the number of all kinds of self-media articles explosive growth, it is of great research significance and practical application value to recommend high-quality articles to users. This paper explores how to combine named entities of knowledge graph and multi-head attention mechanism with quality article identification, not only the contents of articles. For the article classification task of CCKS 2022, we proposed an incorporating named entity information approach based on the NEZHA model with continual pre-training combined, multi-head attention and BiGRU. Five-fold cross-validation solved the problem of less training data, warm-up strategy handled the problem of unstable effect at the beginning of training and focal loss treated the unbalanced samples. Then we selected the top three results in terms of F1 scores for voting fusion. Finally, data augmentation was applied in accordance with certain rules in the fusion results. Our method achieves F1 score of 86.51% on the test set of the task and ranks the second place in the competition.KeywordsMulti-head attentionNamed entityKnowledge graph

Obtaining robust feature representations from multi-position wearable sensory data is challenging in human activity recognition (HAR) since data from different positions can have unordered implicit correlations. Graph neural networks (GNNs) represent data as structured graphs by mining complex relationships and interdependency via message passing between the nodes of graphs. This paper proposes a novel framework (MhaGNN) that combines GNNs and the multi-head attention mechanism, aiming to learn more informative representations for multi-position HAR tasks. The MhaGNN framework takes the sensor channels from multiple wearing positions as nodes to construct graph-structured data from the spatial dimension. Besides, the multi-head attention mechanism is introduced to complete the message passing and aggregation of the graphs for spatial-temporal feature extraction. The MhaGNN learns correlations among sensor channels that can be used as compensatory features together with the captured features from each single sensor channel to enhance HAR. Experimental evaluations on three publicly available HAR datasets and a ground-truth dataset demonstrate that our proposed MhaGNN achieves state-of-the-art recognition performance with the captured rich features, including PAMAP2, OPPORTUNITY, MHAEATH and MPWHAR.