Seq2Emoji: A hybrid sequence generation model for short text emoji prediction

With the rapid development of information technology and the Internet, text information on various media platforms, such as blogs, microblogs, and news, has exploded. In order to process these text data more efficiently, text classification technology has gradually become a research focus. In view of the common problems of incomplete structure and sparse features in short texts, this paper designs a short text classification model with multi-level semantic feature fusion to effectively extract key word features and sentence features from short texts. The model first uses BERT for preliminary semantic feature extraction, and then combines the first-layer Bi-GRU and the word-level self-attention mechanism to enhance the weight of important semantic information in the text. Next, the second-layer Bi-GRU and the sentence-level self-attention mechanism are used to achieve feature fusion at the word vector and sentence vector levels. The maximum pooling strategy of CNN is used to screen the best features and predict the category probability. In addition, the loss function is adjusted, a threshold is introduced to screen and continuously train samples that are difficult to classify, and an adaptive adjustment factor is added in the back propagation to adaptively attenuate the learning rate and accelerate the model fitting process. In the experiment of 20 Newsgroups dataset, compared with the baseline model, the classification accuracy of this model increased by 3.51% to 88.74%, which effectively proved the effectiveness of this model in improving the classification accuracy of Chinese short texts.

Keyphrase greatly provides summarized and valuable information. This information can help us not only understand text semantics, but also organize and retrieve text content effectively. The task of automatically generating it has received considerable attention in recent decades. From the previous studies, we can see many workable solutions for obtaining keyphrases. One method is to divide the content to be summarized into multiple blocks of text, then we rank and select the most important content. The disadvantage of this method is that it cannot identify keyphrase that does not include in the text, let alone get the real semantic meaning hidden in the text. Another approach uses recurrent neural networks to generate keyphrases from the semantic aspects of the text, but the inherently sequential nature precludes parallelization within training examples, and distances have limitations on context dependencies. Previous works have demonstrated the benefits of the self-attention mechanism, which can learn global text dependency features and can be parallelized. Inspired by the above observation, we propose a keyphrase generation model, which is based entirely on the self-attention mechanism. It is an encoder-decoder model that can make up the above disadvantage effectively. In addition, we also consider the semantic similarity between keyphrases, and add semantic similarity processing module into the model. This proposed model, which is demonstrated by empirical analysis on five datasets, can achieve competitive performance compared to baseline methods.

Information and communication technology (ICT) has been widely accepted in education since the COVID-19 outbreak. Today, the convenience that ICT provides in education makes learning independent of time and place. However, compared to face-to-face learning, ICT online learning has the difficulty of finding student questions efficiently. One of the ways to solve this problem is through finding their questions from the online discussion content. With online learning, teachers and students usually send out questions and receive answers on a discussion board without the limitations of time or place. However, because liquid learning is quite convenient, people tend to solve problems in short online texts with a lack of detailed information to express ideas in an online environment. Therefore, the ICT online education environment may result in misunderstandings between teachers and students. For teachers and students to better understand each other’s views, this study aims to classify discussions into a hierarchical structure, named a question map, with several types of learning questions to clarify the views of teachers and students. In addition, this study attempts to extend the description of possible omissions in short texts by using external resources prior to classification. In brief, by applying short text hierarchical classification, this study constructs a question map that can highlight each student’s learning problems and inform the instructor where the main focus of the future course should be, thus improving the ICT education environment.

Emojis are a succinct and visual way to express feelings, emotions, and thoughts during text conversations. Owing to the increase in the use of social media, the usage of emojis has increased drastically. There are various techniques for automating emoji prediction, which use contextual information, temporal information, and user-based features. However, the problem of personalised and dynamic recommendations of emojis persists. This paper proposes personalised emoji recommendations using the time and location parameters. It presents a new annotated conversational dataset and investigates the impact of time and location for emoji prediction. The methodology comprises a hybrid model that uses neural networks and score-based metrics: semantic and cosine similarity. Our approach differs from existing studies and improves the accuracy of emoji prediction up to 73.32% using BERT. KeywordsNLPEmoji predictionBERT

As one of the key tasks in natural language processing, the accuracy of short text classification will directly affect the performance of subsequent downstream tasks. Most of the existing short text classification algorithms usually do not understand the deep semantic information related to the medical field and lack the feature extraction of medical proprietary vocabulary. Traditional models often extract features from context information, which is limited by the lack of access to global and deep semantic information. This paper presents a short text classification algorithm for Chinese clinical medicine combining ALBERT pre-training model and graph attention network. The ALBERT pre-training model is used to represent the feature vectors of the short text of the problem, and then the graphical structure data is generated from the feature vectors. GAT is used to assign different weights to each node in a uniform neighborhood. Finally, a map-level semantic representation for category prediction is generated. The experimental results show that the accuracy of the model is 83.27% on Chinese clinical problem datasets, which effectively improves the model performance.

The network intrusion detection system (NIDS) plays a critical role in maintaining network security. However, traditional NIDS relies on a large volume of samples for training, which exhibits insufficient adaptability in rapidly changing network environments and complex attack methods, especially when facing novel and rare attacks. As attack strategies evolve, there is often a lack of sufficient samples to train models, making it difficult for traditional methods to respond quickly and effectively to new threats. Although existing few-shot network intrusion detection systems have begun to address sample scarcity, these systems often fail to effectively capture long-range dependencies within the network environment due to limited observational scope. To overcome these challenges, this paper proposes a novel elevated few-shot network intrusion detection method based on self-attention mechanisms and iterative refinement. This approach leverages the advantages of self-attention to effectively extract key features from network traffic and capture long-range dependencies. Additionally, the introduction of positional encoding ensures the temporal sequence of traffic is preserved during processing, enhancing the model's ability to capture temporal dynamics. By combining multiple update strategies in meta-learning, the model is initially trained on a general foundation during the training phase, followed by fine-tuning with few-shot data during the testing phase, significantly reducing sample dependency while improving the model's adaptability and prediction accuracy. Experimental results indicate that this method achieved detection rates of 99.90% and 98.23% on the CICIDS2017 and CICIDS2018 datasets, respectively, using only 10 samples.

Vulnerable smart contract function locating based on Multi-Relational Nested Graph Convolutional Network

Individual cattle identification is a prerequisite and foundation for precision livestock farming. Existing methods for cattle identification require radio frequency or visual ear tags, all of which are prone to loss or damage. Here, we propose and implement a new unified deep learning approach to cattle identification using video analysis. The proposed deep learning framework is composed of a Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) with a self-attention mechanism. More specifically, the Inception-V3 CNN was used to extract features from a cattle video dataset taken in a feedlot with rear-view. Extracted features were then fed to a BiLSTM layer to capture spatio-temporal information. Then, self-attention was employed to provide a different focus on the features captured by BiLSTM for the final step of cattle identification. We used a total of 363 rear-view videos from 50 cattle at three different times with an interval of 1 month between data collection periods. The proposed method achieved 93.3% identification accuracy using a 30-frame video length, which outperformed current state-of-the-art methods (Inception-V3, MLP, SimpleRNN, LSTM, and BiLSTM). Furthermore, two different attention schemes, namely, additive and multiplicative attention mechanisms were compared. Our results show that the additive attention mechanism achieved 93.3% accuracy and 91.0% recall, greater than multiplicative attention mechanism with 90.7% accuracy and 87.0% recall. Video length also impacted accuracy, with video sequence length up to 30-frames enhancing identification performance. Overall, our approach can capture key spatio-temporal features to improve cattle identification accuracy, enabling automated cattle identification for precision livestock farming.

In this paper, we propose a novel fusion framework based on a dense network with channel and spatial attention (CSpA-DN) for PET and MR images. In our approach, an encoder composed of the densely connected neural network is constructed to extract features from source images, and a decoder network is leveraged to yield the fused image from these features. Simultaneously, a self-attention mechanism is introduced in the encoder and decoder to further integrate local features along with their global dependencies adaptively. The extracted feature of each spatial position is synthesized by a weighted summation of those features at the same row and column with this position via a spatial attention module. Meanwhile, the interdependent relationship of all feature maps is integrated by a channel attention module. The summation of the outputs of these two attention modules is fed into the decoder and the fused image is generated. Experimental results illustrate the superiorities of our proposed CSpA-DN model compared with state-of-the-art methods in PET and MR images fusion according to both visual perception and objective assessment.

Introduction: Alzheimer's disease (AD) is still one of the most challenging neurodegenerative disorders with no available therapeutics. The identification of drug targets is important for effective treatment development. This research introduces an innovative and improved deep learning prediction framework, BiLSTM-AD, aimed toward therapeutic target indication in crucial, complicated illnesses, particularly Alzheimer's disease. Methods: This paper proposes a deep learning framework, BiLSTM-AD, to predict drug-target indications for Alzheimer's Disease using Protein-Protein Interaction (PPI) datasets from String-DB and InTAct databases. The model utilizes Dual Mode Self-Attention to better capture local and global dependencies in the PPI data. This attention mechanism enables the model to learn from the most significant interactions, which has resulted in better prediction accuracy. Results: The BiLSTM-AD model is evaluated with Precision, Recall, and F1-Score, among other metrics, and compared to reactions-based baseline models such as ANN, RNN, CNN, LSTM, and GNN of the same architecture. Testing on PPI datasets, compared to those baseline methods, BiLSTM-AD has achieved 96% and 97.3% prediction accuracy and provides a more robust solution to infer AD-associated targets for potential drugs. Discussion: The model's self-attention mechanism integrates max and average pooling to capture and emphasize both dominating local interactions and global context. Using a dual perspective greatly enhances the model's ability to discover possible therapeutic targets in important proteinprotein interactions. Precision, Recall, F1-Score, AUC, AUPR, and MSE show that the BiLSTMAD model outperforms other benchmark models in prediction accuracy. With 96% accuracy and 0.93 AUC, our technique surpasses strong baselines (CNN, LSTM, GNN). Conclusion: Integrating the BiLSTM and dual mode self-attention mechanism is an especially effective approach to improve the model performance and enables a practical focus on interaction among drug candidates.

Runoff forecasting is important for water resource management. Although deep learning models have substantially improved the accuracy of runoff prediction, the temporal and feature dependencies between rainfall–runoff time series elements have not been effectively exploited. In this work, we propose a new hybrid deep learning model to predict hourly streamflow: SA-CNN-LSTM (self-attention, convolutional neural network, and long short-term memory network). The advantages of CNN and LSTM in terms of data extraction from time series data are combined with the self-attention mechanism. By considering interdependences of the rainfall–runoff sequence between timesteps and between features, the prediction performance of the model is enhanced. We explored the performance of the model in the Mazhou Basin, China; we compared its performance with the performances of LSTM, CNN, ANN (artificial neural network), RF (random forest), SA-LSTM, and SA-CNN. Our analysis demonstrated that SA-CNN-LSTM demonstrated robust prediction with different flood magnitudes and different lead times; it was particularly effective within lead times of 1–5 h. Additionally, the performance of the self-attention mechanism with LSTM and CNN alone, respectively, was improved at some lead times; however, the overall performance was unstable. In contrast, the hybrid model integrating CNN, LSTM, and the self-attention mechanism exhibited better model performance and robustness. Overall, this study considers the importance of temporal and feature dependencies in hourly runoff prediction, then proposes a hybrid deep learning model to improve the performances of conventional models in runoff prediction.

As the battery cycles between charging and discharging, the working conditions or improper operations such as overcharge and over discharge will aggravate the negative reaction inside the battery, generate irreversible chemical substances, and reduce the number of active substances involved in the electrochemical reaction, resulting in a decrease in battery capacity. Batteries that lose 20% of their capacity can be considered to have failed. A failed battery shows that the battery capacity and power decay faster, and the electrical characteristics, stability, and safety of the battery will drop significantly. As a means of improving the machine learning model’s accuracy and generalization for RUL prediction of zinc-ion batteries, this paper mainly discusses about the design of the encoder–decoder model structure and the application of optimization methods. Then, the method of neural network hyperparameter optimization is studied. Finally, the validity of the research work done in this paper is verified by a series of comparative experiments.

To more effectively capture the periodic and dynamic changes in urban traffic flow and the spatiotemporal correlation of complex road networks, a new traffic flow prediction method, the Enhanced Spatiotemporal Graph Convolutional Network Encoder-Decoder Model (ESGCN-EDM), is proposed. The model achieves a significant enhancement in prediction accuracy through the introduction of the attention-based Personalized-enhanced Fusion Graph Convolutional Network (aPFGCN) and the Temporal Convolutional Bidirectional Long Short-Term Memory (TCBiL) module. The aPFGCN module effectively reduces the dimensionality of features and decreases model complexity to obtain the final node feature representation by personalizing the adjustment of node influence coefficients and applying Fourier transform and inverse transform techniques. Additionally, by incorporating an attention mechanism, it enhances the model’s ability to focus on important information and effectively captures the spatial topological relationships within the traffic network. The TCBiL module integrates 1D convolution with BiLSTM to form a unified temporal feature extraction module. The 1D convolution is utilized to extract local features from the time series, while the BiLSTM captures long-term dependencies within the time series. This allows for simultaneous feature extraction and temporal modeling, thereby enhancing the model’s efficiency and performance, and strengthening its ability to model time series. In the encoder part of the ESGCN-EDM, the aPFGCN is combined with the TCBiL to handle the spatiotemporal coupling interactions of the road network. The decoder part then performs multi-step predictions based on spatiotemporal sequences using the TCBiL and CNN, generating high-dimensional representations. Extensive experiments conducted on two real-world road traffic datasets demonstrate that the ESGCN-EDM model consistently outperforms other benchmark models in 1-hour, 30-minute, and 15-minute traffic flow predictions. Specifically, on the PeMSD8 dataset, the model achieves reductions in MAE, RMSE, and SMAPE by 7.9%, 2.1%, and 16.9%, respectively, compared to the AMRGCN model for 1-hour predictions. Similarly, on the PeMSD4 dataset, the model reduces MAE, RMSE, and MAPE by 1.8%, 1.1%, and 3.0%, respectively. These results validate the efficacy of the proposed model and its ability to significantly enhance the accuracy of traffic flow forecasting.

Surface roughness prediction based on fusion of dynamic-static data

Sleep staging serves as a fundamental assessment for sleep quality measurement and sleep disorder diagnosis. Although current deep learning approaches have successfully integrated multimodal sleep signals, enhancing the accuracy of automatic sleep staging, certain challenges remain, as follows: 1) optimizing the utilization of multi-modal information complementarity, 2) effectively extracting both long- and short-range temporal features of sleep information, and 3) addressing the class imbalance problem in sleep data. To address these challenges, this paper proposes a two-stream encode-decoder network, named TSEDSleepNet, which is inspired by the depth sensitive attention and automatic multi-modal fusion (DSA2F) framework. In TSEDSleepNet, a two-stream encoder is used to extract the multiscale features of electrooculogram (EOG) and electroencephalogram (EEG) signals. And a self-attention mechanism is utilized to fuse the multiscale features, generating multi-modal saliency features. Subsequently, the coarser-scale construction module (CSCM) is adopted to extract and construct multi-resolution features from the multiscale features and the salient features. Thereafter, a Transformer module is applied to capture both long- and short-range temporal features from the multi-resolution features. Finally, the long- and short-range temporal features are restored with low-layer details and mapped to the predicted classification results. Additionally, the Lovász loss function is applied to alleviate the class imbalance problem in sleep datasets. Our proposed method was tested on the Sleep-EDF-39 and Sleep-EDF-153 datasets, and it achieved classification accuracies of 88.9% and 85.2% and Macro-F1 scores of 84.8% and 79.7%, respectively, thus outperforming conventional traditional baseline models. These results highlight the efficacy of the proposed method in fusing multi-modal information. This method has potential for application as an adjunct tool for diagnosing sleep disorders.