MLALogic:Knowledge Reasoning Model Combining Knowledge Representation with Multi-head Latent Attention Mechanism

To better understand and evaluate the GNSS positioning performance, it is convenient to adopt corresponding measures to reduce the impact of errors on positioning. A GNSS positioning error estimation scheme based on an improved multi-layer perceptron model is proposed. The multi-head attention mechanism and gating operation are integrated into the multi-layer perceptron model to adaptively select and filter features, enhancing the model’s ability to understand input features. First, the original positioning error of the satellite is obtained through the Kalman filter positioning method. The data are then preprocessed to extract available features. Finally, the features are input into the constructed model for training and testing to obtain the estimated positioning error value. Two types of comparative experiments were completed. The performance of the presented model is evaluated by the root mean square error. Experimental results show that the proposed method performs well in terms of performance indicators, and has obvious advantages over other state-of-the-art methods. In particular, the root mean square error of the presented method in the first dataset is 0.239 m, which is 39.2% and 17% lower than the current state-of-the-art long short-term memory network and convolutional neural network, respectively. The presented method can provide higher-precision estimated values for studying the GNSS positioning error estimation problem.

To overcome the limitations of existing deep learning-based image recognition approaches for diagnosing the operating conditions of pumping wells–namely, slow processing speed and high parameter complexity–this paper proposes a lightweight diagnostic method based on a one-dimensional convolutional neural network (1D CNN). Unlike traditional image-based techniques, the proposed method eliminates the need to convert raw data into dynamometer card images, thereby reducing computational overhead and significantly decreasing the number of model parameters. The architecture integrates 1D CNN with a bidirectional long short-term memory (BiLSTM) network to effectively capture both local patterns and global temporal dependencies inherent in different operating states of pumping wells. Additionally, a multi-head attention (MHA) mechanism is incorporated to enhance the representation of critical time points, thereby improving the model's sensitivity to key features and its overall classification performance. Experiments conducted on an augmented dataset–generated via a simulation-based algorithm–demonstrate that the proposed 1DCNN-BiLSTM-MHA model achieves a classification accuracy of 99.22%. Experimental results demonstrate that the proposed model possesses strong feature learning capabilities. Comparative analysis with several baseline models highlights its superior diagnostic performance and reliability, along with reduced parameter count and improved computational speed.

Knowledge graphs, powerful tools that explicitly transfer knowledge to machines, have significantly advanced new knowledge inferences. Discovering unknown relationships between diseases and genes/proteins in biomedical knowledge graphs can lead to the identification of disease development mechanisms and new treatment targets. Generating high-quality representations of biomedical entities is essential for successfully predicting disease-gene/protein associations. We developed a computational model that predicts disease-gene/protein associations using the Precision Medicine Knowledge Graph, a biomedical knowledge graph. Embeddings of biomedical entities were generated using two different methods-a large language model (LLM) and the knowledge graph embedding (KGE) algorithm. The LLM utilizes information obtained from massive amounts of text data, whereas the KGE algorithm relies on graph structures. We developed a disease-gene/protein association prediction model, "Cross-Modal Embedding Integrator (CMEI)," by integrating embeddings from different modalities using a multi-head attention mechanism. The area under the receiver operating characteristic curve of CMEI was 0.9662 (± 0.0002) in predicting disease-gene/protein associations. In conclusion, we developed a computational model that effectively predicts disease-gene/protein associations. CMEI may contribute to the identification of disease development mechanisms and new treatment targets.

This study proposes a deep learning method based on ResNet, Transformer, and Multi-Head Attention for food quality inspection. With the growing severity of food safety issues, traditional food quality inspection methods have significant shortcomings in efficiency and accuracy. Therefore, there is an urgent need to develop automated, efficient, and accurate food quality inspection technologies. Our method combines a deep convolutional neural network (ResNet), a self-attention mechanism (Transformer), and a multi-head attention mechanism (Multi-Head Attention). It effectively extracts detailed features from food images while enhancing attention to global image information and important regions, effectively identifying mold, contamination, and spoilage in food. Experimental results demonstrate that this method outperforms traditional deep learning models in metrics such as precision, recall, and F1 score. Ablation experiments also validate the contribution of each module to the performance improvement. This model provides a more accurate and automated solution for food quality inspection and has broad application prospects.

With the number of all kinds of self-media articles explosive growth, it is of great research significance and practical application value to recommend high-quality articles to users. This paper explores how to combine named entities of knowledge graph and multi-head attention mechanism with quality article identification, not only the contents of articles. For the article classification task of CCKS 2022, we proposed an incorporating named entity information approach based on the NEZHA model with continual pre-training combined, multi-head attention and BiGRU. Five-fold cross-validation solved the problem of less training data, warm-up strategy handled the problem of unstable effect at the beginning of training and focal loss treated the unbalanced samples. Then we selected the top three results in terms of F1 scores for voting fusion. Finally, data augmentation was applied in accordance with certain rules in the fusion results. Our method achieves F1 score of 86.51% on the test set of the task and ranks the second place in the competition.KeywordsMulti-head attentionNamed entityKnowledge graph

Purpose Path information in knowledge graphs can provide explicit explanations for recommendation decisions, thus becoming a focus in explainable recommendation research. However, limited studies about explainable recommendation problems result in low model transparency and poor persuasiveness, affecting user experience. Therefore, the goal is to provide accurate and interpretable recommendations for recommendations. Design/methodology/approach This study proposes a recommendation reasoning method based on knowledge graph and reinforcement learning. To alleviate the noise problem in the state space, a multi-head attention mechanism is used to learn state expressions. A dual critic network is used to optimize long-term and short-term rewards simultaneously, achieving path reasoning in the knowledge graph to provide short-term and long-term value explanations for recommendation decisions. Findings We conduct extensive experiments on the real-world benchmark dataset and the domain dataset to validate the effectiveness of our method on the recommendation and explanation tasks, which proves the method's ability to generate high-quality and interpretable course recommendations. Originality/value Developing explainable recommendation methods based on the combination of knowledge graph and reinforcement learning is crucial to overcome the current limitations. A recommendation system that integrates knowledge reasoning, autonomous learning and interpretability may meet the needs of the modern education field for explainable recommendation systems.

Obtaining robust feature representations from multi-position wearable sensory data is challenging in human activity recognition (HAR) since data from different positions can have unordered implicit correlations. Graph neural networks (GNNs) represent data as structured graphs by mining complex relationships and interdependency via message passing between the nodes of graphs. This paper proposes a novel framework (MhaGNN) that combines GNNs and the multi-head attention mechanism, aiming to learn more informative representations for multi-position HAR tasks. The MhaGNN framework takes the sensor channels from multiple wearing positions as nodes to construct graph-structured data from the spatial dimension. Besides, the multi-head attention mechanism is introduced to complete the message passing and aggregation of the graphs for spatial-temporal feature extraction. The MhaGNN learns correlations among sensor channels that can be used as compensatory features together with the captured features from each single sensor channel to enhance HAR. Experimental evaluations on three publicly available HAR datasets and a ground-truth dataset demonstrate that our proposed MhaGNN achieves state-of-the-art recognition performance with the captured rich features, including PAMAP2, OPPORTUNITY, MHAEATH and MPWHAR.

BackgroundWith the rapid expansion of biomedical literature, biomedical information extraction has attracted increasing attention from researchers. In particular, relation extraction between 2 entities is a long-term research topic.ObjectiveThis study aimed to perform 2 multiclass relation extraction tasks of Biomedical Natural Language Processing Workshop 2019 Open Shared Tasks: relation extraction of Bacteria-Biotope (BB-rel) task and binary relation extraction of plant seed development (SeeDev-binary) task. In essence, these 2 tasks are aimed at extracting the relation between annotated entity pairs from biomedical texts, which is a challenging problem.MethodsTraditional research methods adopted feature- or kernel-based methods and achieved good performance. For these tasks, we propose a deep learning model based on a combination of several distributed features, such as domain-specific word embedding, part-of-speech embedding, entity-type embedding, distance embedding, and position embedding. The multi-head attention mechanism is used to extract the global semantic features of an entire sentence. Meanwhile, we introduced a dependency-type feature and the shortest dependency path connecting 2 candidate entities in the syntactic dependency graph to enrich the feature representation.ResultsExperiments show that our proposed model has excellent performance in biomedical relation extraction, achieving F1 scores of 65.56% and 38.04% on the test sets of the BB-rel and SeeDev-binary tasks. Especially in the SeeDev-binary task, the F1 score of our model is superior to that of other existing models and achieves state-of-the-art performance.ConclusionsWe demonstrated that the multi-head attention mechanism can learn relevant syntactic and semantic features in different representation subspaces and different positions to extract comprehensive feature representation. Moreover, syntactic dependency features can improve the performance of the model by learning dependency relation between the entities in biomedical texts.

How to enable the computer to accurately analyze the emotional information and story background of characters in Qin opera is a problem that needs to be studied. To promote the artistic inheritance and cultural emotion color dissemination of Qin opera, an emotion analysis model of Qin opera based on attention residual network (ResNet) is presented. The neural network is improved and optimized from the perspective of the model, learning rate, network layers, and the network itself, and then multi-head attention is added to the ResNet to increase the recognition ability of the model. The convolutional neural network (CNN) is optimized from the internal depth, and the fitting ability and stability of the model are enhanced through the ResNet model. Combined with the attention mechanism, the expression of each weight information is strengthened. The multi-head attention mechanism is introduced in the model and a multi-head attention ResNet, namely, MHAtt_ResNet, is proposed. The network structure can effectively identify the features of the spectrogram, improve the weight information of spectrogram features, and deepen the relationship between distant information in long-time series. Through experiments, the proposed model has high emotional classification accuracy for Qin opera, and with the increase of the number of data sets, the model will train a better classification effect.

Industrial water withdrawal prediction is the cornerstone of water resources monitoring and early warning, as well as a key task in implementing rigid constraints on water resources. Although many factors may influence water withdrawal, they are difficult to obtain. Extracting the features in the historical water withdrawal data itself is a more convenient alternative way to achieve accurate prediction. However, due to the lack of significant regularity and periodicity in industrial water data, feature extraction is challenging. In this paper, a multi-head attention encoder (MAEN) model consisting of multi-head attention mechanisms and feed-forward neural networks is proposed to enable more convenient and accurate forecasting. The multi-head attention mechanism allows to focus on different parts of the data simultaneously, detecting complex dependencies, and extracting key features from historical data to enable more efficient feature extraction. Applied to water withdrawal data from real factories, the proposed model outperforms common time series prediction models including artificial neural network, long short-term memory, and gate recurrent unit. Compared to the best-performing comparison model, MAEN shows a reduction of mean squared error by 3.2% for 1 day ahead predictions, and 8.9% for 7 days ahead predictions, illustrating its superior performance in industrial water withdrawal prediction.

The development of social media has changed the way the public expresses political opinions, especially regarding the evaluation of President Joko Widodo’s (Jokowi) leadership after his term. Platform X (formerly Twitter) has become the primary source of public opinion data, but the use of informal language and sarcasm makes accurate sentiment analysis challenging. This study creates a sentiment analysis model that uses deep learning with a CNN-BiLSTM structure and a multi-head attention mechanism. The dataset consists of 52,643 tweets that have been labeled and embedded using IndoBERT. To address class imbalance, the SMOTE method was applied to the training data, enabling the model to better learn from minority classes. The results indicate that the model achieves a high accuracy of 98.78%, with an average precision, recall, and F1-score of 0.98. These findings indicate that the model is not only accurate but also reliable in distinguishing each sentiment class. A comparison with other model variants suggests that the complete combination of CNN-BiLSTM and Multi-Head Attention delivers the best performance, although the improvement is relatively small.

As one of the core tasks of natural language processing technology, text classification methods general-ly face the problems of insufficient global semantic capture and limited feature focusing ability when processing long texts or complex semantics. To address this issue, a deep learning model that integrates improved convolutional neural networks, unbalanced bidirectional long short-term memory networks, and multi-head attention mechanisms is proposed. Utilizing an improved bidirectional long short-term memory network to capture global semantic information, while dynamically focusing on key features through a multi head attention mechanism to enhance the model's adaptability to classification tasks. The performance of the model is validated through experiments on AG News (short text) and IMDb (long text) datasets. The results show that in short text classification, the proposed method has an accu-racy rate of 96% and a classification error rate of only 1.46%. In the task of long text classification, the method proposed in the study has a product under the curve of 0.98. In adversarial attack testing, the accuracy rates of adversarial samples generated by different methods are 92.85% and 90.63%, respec-tively, with the lowest robustness degradation rates of 3.72% and 5.49%, respectively. In cross domain generalization testing, it shows the least classification errors and superior cross domain adaptability. These results validate the high performance, robustness, and wide applicability of the method. The re-search indicates that this approach can validly improve the performance of text classification and pro-vide new solutions for natural language processing related tasks in long text and multi-category scenar-ios.

The fault diagnosis of cargo UAVs (Unmanned Aerial Vehicles) is crucial to ensure the safety of logistics distribution. In the context of smart logistics, the new trend of utilizing knowledge graph (KG) for fault diagnosis is gradually emerging, bringing new opportunities to improve the efficiency and accuracy of fault diagnosis in the era of Industry 4.0. The operating environment of cargo UAVs is complex, and their faults are typically closely related to it. However, the available data only considers faults and maintenance data, making it difficult to diagnose faults accurately. Moreover, the existing KG suffers from the problem of confusing entity boundaries during the extraction process, which leads to lower extraction efficiency. Therefore, a fault diagnosis knowledge graph (FDKG) for cargo UAVs constructed based on multi-domain fusion and incorporating an attention mechanism is proposed. Firstly, the multi-domain ontology modeling is realized based on the multi-domain fault diagnosis concept analysis expression model and multi-dimensional similarity calculation method for cargo UAVs. Secondly, a multi-head attention mechanism is added to the BERT-BILSTM-CRF network model for entity extraction, relationship extraction is performed through ERNIE, and the extracted triples are stored in the Neo4j graph database. Finally, the DJI cargo UAV failure is taken as an example for validation, and the results show that the new model based on multi-domain fusion data is better than the traditional model, and the precision rate, recall rate, and F1 value can reach 87.52%, 90.47%, and 88.97%, respectively.

AISE: Attending to Intent and Slots Explicitly for better spoken language understanding

Code classification and code clone detection are crucial for understanding and maintaining large software systems. Although deep learning surpasses traditional techniques in capturing the features of source code, existing models suffer from low processing power and high complexity. We propose a novel source code representation method based on the multi-head attention mechanism (SCRMHA). SCRMHA captures the vector representation of entire code segments, enabling it to focus on different positions of the input sequence, capture richer semantic information, and simultaneously process different aspects and relationships of the sequence. Moreover, it can calculate multiple attention heads in parallel, speeding up the computational process. We evaluate SCRMHA on both the standard dataset and an actual industrial dataset, and analyze the differences between these two datasets. Experiment results in code classification and clone detection tasks show that SCRMHA consumes less time and reduces complexity by about one-third compared with traditional source code feature representation methods. The results demonstrate that SCRMHA reduces the computational complexity and time consumption of the model while maintaining accuracy.