Graph neural networks embedded with domain knowledge for cyber threat intelligence entity and relationship mining.

The continuous improvement of the cyber threat intelligence sharing mechanism provides new ideas to deal with Advanced Persistent Threats (APT). Extracting attack behaviors, i.e., Tactics, Techniques, Procedures (TTP) from Cyber Threat Intelligence (CTI) can facilitate APT actors’ profiling for an immediate response. However, it is difficult for traditional manual methods to analyze attack behaviors from cyber threat intelligence due to its heterogeneous nature. Based on the Adversarial Tactics, Techniques and Common Knowledge (ATT&CK) of threat behavior description, this paper proposes a threat behavioral knowledge extraction framework that integrates Heterogeneous Text Network (HTN) and Graph Convolutional Network (GCN) to solve this issue. It leverages the hierarchical correlation relationships of attack techniques and tactics in the ATT&CK to construct a text network of heterogeneous cyber threat intelligence. With the help of the Bidirectional Encoder Representation from Transformers (BERT) pretraining model to analyze the contextual semantics of cyber threat intelligence, the task of threat behavior identification is transformed into a text classification task, which automatically extracts attack behavior in CTI, then identifies the malware and advanced threat actors. The experimental results show that F1 achieve 94.86% and 92.15% for the multi-label classification tasks of tactics and techniques. Extend the experiment to verify the method’s effectiveness in identifying the malware and threat actors in APT attacks. The F1 for malware and advanced threat actors identification task reached 98.45% and 99.48%, which are better than the benchmark model in the experiment and achieve state of the art. The model can effectively model threat intelligence text data and acquire knowledge and experience migration by correlating implied features with a priori knowledge to compensate for insufficient sample data and improve the classification performance and recognition ability of threat behavior in text.

The increasing sophistication of cyber threats originating from the Dark Web has necessitated the development of advanced threat intelligence frameworks capable of detecting and predicting malicious activities in real time. Traditional cybersecurity approaches often struggle to process the vast, unstructured, and linguistically diverse data generated on underground forums and illicit marketplaces. Transformer-based natural language processing (NLP) models, such as Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT), have demonstrated exceptional capabilities in understanding and generating contextualized textual representations, making them highly effective for Dark Web analysis and cybercrime prediction. This chapter explores the integration of transformer-based models in cyber threat intelligence workflows, emphasizing their ability to automate the identification of emerging threats, detect cybercriminal activities, and forecast evolving attack patterns. Key challenges, including data scarcity, adversarial linguistic variations, and ethical considerations in Dark Web monitoring, are analyzed alongside potential solutions leveraging AI-driven methodologies. , the chapter discusses the implications of transformer-based threat intelligence frameworks for real-time cybersecurity applications and future research directions aimed at enhancing cyber resilience. The insights presented contribute to the advancement of AI-driven cyber threat intelligence, enabling proactive threat mitigation strategies in an increasingly complex digital threat landscape.

The relationship between food and disease holds significant importance in traditional medicine and healthcare practices. However, when it comes to disseminating dietary and health-related information, the news media often falls short in terms of authority and accuracy. To address this issue, this paper introduces a method for extracting relationships among food, herbals, diseases, and symptoms leveraging BERT (Bidirectional Encoder Representations from Transformers) framework. The method utilizes Transfer Learning to fine-tune the BERT pretraining model on an existing dataset to enable the model to recognize entities such as food, herbals, diseases, symptoms from unstructured text found in various literature sources. Following entity recognition, an entity embedding masking process is employed to construct a tailored training dataset specifically designed for entity relationship extraction. By combining entity recognition and relationship extraction, this method effectively uncovers the connections between food, herbals, diseases, and symptoms. The experimental results: Named Entity Recognition precision rate of 93.51%, a recall of 92.22%, and F1 of 92.86%. Relation Extraction accuracy of 91.99%, F1 of 93.88%. This method addresses the scarcity of annotated data in this domain, offering a method for automatic extraction of entity relationships, even without the need for expert annotations.

Our study focuses on Traditional Chinese Medical (TCM) named entity recognition (NER), which involves identifying and extracting specific entity names from TCM record. This task has significant implications for doctors and researchers, as it enables the automated identification of relevant TCM terms, ultimately enhancing research efficiency and accuracy. However, the current Bidirectional Encoder Representations from Transformers-Long Short Term Memory-Conditional Random Fields (BERT-LSTM-CRF) model for TCM NER is constrained by a traditional structure, limiting its capacity to fully harness the advantages provided by Bidirectional Encoder Representations from Transformers (BERT) and long short term memory (LSTM) models. Through comparative experiments, we also observed that the straightforward superimposition of models actually leads to a decrease in recognition results. To optimize the structure of the traditional BERT-BiLSTM-CRF model and obtain more effective text representations, we propose the Dyn-Att Net model, which introduces dynamic attention and a parallel structure. By integrating BERT and LSTM models with the dynamic attention mechanism, our model effectively captures semantic, contextual, and sequential relations within text sequences, resulting in high accuracy. To validate the effectiveness of our model, we compared it with nine other models in TCM dataset namely the publicly available PaddlePaddle dataset. Our Dyn-Att Net model, based on BERT, outperforms the other models, achieving an F1 score of 81.91%, accuracy of 92.06%, precision of 80.26%, and recall of 83.76%. Furthermore, its robust generalization capability is substantiated through validation on the APTNER, MSRA, and EduNER datasets. Overall, the Dyn-Att Net model not only enhances NER accuracy within the realm of traditional Chinese medicine, but also showcases considerable potential for cross-domain generalization. Moreover, the Dyn-Att Net model's parallel architecture facilitates efficient computation, contributing to time-saving efforts in NER tasks.

Fenomena bullying di media sosial, khususnya di Twitter, telah menjadi isu yang semakin memprihatinkan dengan dampak signifikan terhadap kesehatan mental pengguna. Dalam rangka mengatasi masalah ini, deteksi otomatis tweet yang mengandung konten bullying menjadi sangat penting. Penelitian ini bertujuan untuk membandingkan performa dua model pemrosesan bahasa alami terbaru, yaitu BERT (Bidirectional Encoder Representations from Transformers) dan XLNet, dalam klasifikasi tweet yang mengandung bullying. Metodologi penelitian ini melibatkan pengumpulan dataset tweet yang telah dilabeli sebagai bullying atau non-bullying. Proses preprocessing teks dilakukan untuk membersihkan dan menyiapkan data sebelum digunakan dalam pelatihan model. Kedua model, BERT dan XLNet, dilatih dan diuji menggunakan dataset yang sama. Evaluasi performa dilakukan dengan menggunakan metrik akurasi, presisi, recall, dan F1-score. Hasil penelitian menunjukkan bahwa kedua model memiliki kemampuan yang baik dalam mengidentifikasi tweet bullying, akan tetapi XLNet menunjukkan performa yang lebih unggul dibandingkan BERT dengan tingkat akurasi sebesar 95%. Dengan nilai presisi = 100%, recall = 0,87%, dan F1-score = 0,88%. XLNet mampu menangkap konteks dan nuansa bahasa yang lebih kompleks dalam tweet, yang berkontribusi pada akurasi klasifikasi yang lebih tinggi. Penelitian ini memberikan kontribusi penting dalam bidang deteksi bullying di media sosial dengan menunjukkan bahwa penggunaan model XLNet lebih efektif dibandingkan BERT. Temuan ini dapat membantu platform seperti Twitter dalam mengidentifikasi dan mencegah konten bullying, sehingga menciptakan lingkungan online yang lebih aman bagi pengguna, serta dapat digunakan sebagai dasar untuk pengembangan sistem deteksi bullying yang lebih canggih dan efisien di masa depan. Abstract The phenomenon of bullying on social media, particularly on Twitter, has become an increasingly concerning issue with significant impacts on users' mental health. In order to address this issue, automatic detection of tweets containing bullying content is crucial. This study aims to compare the performance of two recent natural language processing models, namely BERT (Bidirectional Encoder Representations from Transformers) and XLNet, in the classification of tweets containing bullying. The research methodology involves collecting a dataset of tweets that have been labelled as bullying or non-bullying. Text preprocessing is done to clean and prepare the data before it is used in model training. Both models, BERT and XLNet, were trained and tested using the same dataset. Performance evaluation was conducted using accuracy, precision, recall, and F1-score metrics. The results show that both models have a good ability to identify bullying tweets, but XLNet shows superior performance compared to BERT with an accuracy rate of 95%. With precision = 100%, recall = 0.87%, and F1-score = 0.88%. XLNet is able to capture more complex context and language nuances in tweets, which contributes to higher classification accuracy. This research makes an important contribution to the field of bullying detection on social media by showing that the use of the XLNet model is more effective than BERT. These findings can help platforms like Twitter identify and prevent bullying content, thereby creating a safer online environment for users, and can be used as a basis for the development of more sophisticated and efficient bullying detection systems in the future.

With the tremendous growth in the number of scientific papers being published, searching for references while writing a scientific paper is a time-consuming process. A technique that could add a reference citation at the appropriate place in a sentence will be beneficial. In this perspective, context-aware citation recommendation has been researched upon for around two decades. Many researchers have utilized the text data called the context sentence, which surrounds the citation tag, and the metadata of the target paper to find the appropriate cited research. However, the lack of well-organized benchmarking datasets and no model that can attain high performance has made the research difficult. In this paper, we propose a deep learning based model and well-organized dataset for context-aware paper citation recommendation. Our model comprises a document encoder and a context encoder, which uses Graph Convolutional Networks (GCN) layer and Bidirectional Encoder Representations from Transformers (BERT), which is a pre-trained model of textual data. By modifying the related PeerRead dataset, we propose a new dataset called FullTextPeerRead containing context sentences to cited references and paper metadata. To the best of our knowledge, This dataset is the first well-organized dataset for context-aware paper recommendation. The results indicate that the proposed model with the proposed datasets can attain state-of-the-art performance and achieve a more than 28% improvement in mean average precision (MAP) and recall@k.

In this paper, we propose an efficient approach for named entity recognition (NER). This paper applies the broad learning system (BLS) to NER task, which includes the feature learning and incremental learning processes. In the feature learning of BLS, the calculation of each new feature node involves only matrix multiplication, and can effectively reduce the amount of calculation to improve efficiency in obtaining rich features. In the incremental learning of BLS, the feature nodes are calculated by active function and ridge regression in order to obtain incremental nodes. In this paper, we combine the BLS with bidirectional encoder representations from transformers (BERT) and conditional random fields (CRF) algorithms to conduct NER of English corpus. Firstly, we extract features through BERT. Secondly, we use BLS to calculate the extracted features. Finally, we apply CRF to decoding features. In numerical experiments, NER of English corpus based on BIO schema is used as benchmark, And the proposed method shows higher accuracy and faster training than other baseline methods.

In recent years, as cybersecurity threats have become increasingly severe and cyberattacks have occurred frequently, higher requirements have been put forward for cybersecurity protection. Therefore, the Named Entity Recognition (NER) technique, which is the cornerstone of Cyber Threat Intelligence (CTI) analysis, is particularly important. However, most existing NER studies are limited to recognizing single-layer flat entities, ignoring the possible nested entities in CTI. On the other hand, most of the existing studies focus on English CTIs, and the existing models performed poorly in a limited number of Chinese CTI studies. Given the above challenges, we propose in this paper a novel unified model, RBTG, which aims to identify flat and nested entities in Chinese CTI effectively. To overcome the difficult boundary recognition problem and the direction-dependent and distance-dependent properties in Chinese CTI NER, we use Global Pointer as the decoder and TENER as the encoder layer, respectively. Specifically, the Global Pointer layer solves the problem of the insensitivity of general NER methods to entity boundaries by utilizing the relative position information and the multiplicative attention mechanism. The TENER layer adapts to the Chinese CTI NER task by introducing an attention mechanism with direction awareness and distance awareness. Meanwhile, to cope with the complex feature capture of hierarchical structure and dependencies among Chinese CTI nested entities, the TENER layer solves the problem by following the structure of multiple self-attention layers and feed-forward network layers superimposed on each other in the Transformer. In addition, to fill the gap in the Chinese CTI nested entity dataset, we further apply the Large Language Modeling (LLM) technique and domain knowledge to construct a high-quality Chinese CTI nested entity dataset, CDTinee, which consists of six entity types selected from STIX, including nearly 4000 entity types extracted from more than 3000 threatening sentences. In the experimental session, we conduct extensive experiments on multiple datasets, and the results show that the proposed model RBTG outperforms the baseline model in both flat NER and nested NER.

Information about bacteria biotopes (BB) is important for fundamental research and applications in microbiology. BB task at BioNLP-OST 2019 focuses on the extraction of locations and phenotypes of microorganisms from PubMed abstracts and full-text excerpts. The subtask BB-rel+ner aims to recognize relevant entities and extract interrelationships about BBs. The corresponding corpus owns some distinctive features (e.g. nested entities) which are challenging to deal with. Therefore, previous methods achieved low performance on entity and relation extraction and limited the mutual effect between named entity recognition and relation extraction. There is still much room for improvement. We propose a span-based model to extract entities and relations jointly from biomedical text regarding the BBs. For alleviating the problem of annotated data deficiency in domain-specific task, we employ a BERT (Bidirectional Encoder Representations from Transformers) model pre-trained on the domain-specific corpus to encode sentences. Our model considers all spans in a sentence as potential entity mentions and computes relation scores between the most confident entity spans based on representations of spans and contexts between spans. Experiments on the BB-rel+ner 2019 corpus demonstrate that our model achieves significantly better performance than the state-of-the-art method, with a reduction of 21.6% slot error rate (SER) for extracting relations. Our model is also effective in recognizing nested entities. Furthermore, the model can be applied to the CHEMPROT corpus for joint extraction of chemical-protein entities and relations, achieving state-of-the-art performance. Our source code is available at https://github.com/zmmzGitHub/SpanMB_BERT. Supplementary data are available at Bioinformatics online.

Recently, people used social media to convey their emotions in the text. Text classification is required for group a text containing emotion or not and identify the type of emotion. Most of the research in emotion classification use machine learning methods. So this research offers the use of deep learning for the classification of emotions in text. One of the algorithms that is widely used for deep learning is BERT (Bidirectional Encoder Representations from Transformers). For the experiment, classification is divided into two phases, namely the first phase groups emotion or not emotion tweets, and the second phase groups emotion tweets into five types of emotions, namely happy, angry, sad, scared and surprised. The experiment uses three data scenarios, namely the commuter line, transjakarta and combination data. The BERT method was compared with the BiLSTM and LSTM methods for emotion classification. The result of this study is BERT generates the highest accuracy compared to other methods for the three data scenarios with values of 87.24%, 86.27%, and 86.99% in the first phase. Then, the next accuracy value followed by obtained BiLSTM, and the last is LSTM. In the second phase, BERT also obtained the highest accuracy values for the three data scenarios with values of 94.26%, 93.71 %, and 93.98%. Then, the evaluation using Precision, Recall and F1-Measure results that BERT provides the best overall method. In this study, BERT has good performance for emotion classification.

We study the problem of incorporating prior knowledge into a deep Transformer-based model,i.e.,Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.

BackgroundThe unstructured data of Chinese cancer electronic health records (EHRs) contains valuable medical expertise. Accurate medical entity recognition is crucial for building a medical-assisted decision system. Named entity recognition (NER) in cancer EHRs typically uses general models designed for English medical records. There is a lack of specialized handling for cancer-specific records and limited application to Chinese medical records.ObjectiveThis study aims to propose a specific NER model to enhance the recognition of medical entities in Chinese cancer EHRs.MethodsDesensitized inpatient EHRs related to breast cancer were collected from a leading hospital in Beijing. Building upon the MC Bidirectional Encoder Representations from Transformers (BERT) foundation, the study further incorporated a Chinese cancer corpus for pretraining, resulting in the construction of the ChCancerBERT pretrained model. In conjunction with dilated-gated convolutional neural networks, bidirectional long short-term memory, multihead attention mechanism, and a conditional random field, this model forms a multimodel, multilevel integrated NER approach.ResultsThis approach effectively extracts medical entity features related to symptoms, signs, tests, treatments, and time in Chinese breast cancer EHRs. The entity recognition performance of the proposed model surpasses that of the baseline model and other models compared in the experiment. The F1-score reached 86.93%, precision reached 87.24%, and recall reached 86.61%. The model introduced in this study demonstrates exceptional performance on the CCKS2019 dataset, attaining a precision rate of 87.26%, a recall rate of 87.27%, and an impressive F1-score of 87.26%, surpassing that of existing models.ConclusionsThe experiments demonstrate that the approach proposed in this study exhibits excellent performance in NER within breast cancer EHRs. This advancement will further contribute to clinical decision support for cancer treatment and research. In addition, the study reveals that incorporating domain-specific corpora in clinical NER tasks can further enhance the performance of BERT models in specialized domains.

In this paper, a tagging tool is developed to streamline the process of locating tags for each term and manually selecting the target term. It directly extracts the terms to be tagged from sentences and displays it to the user. It also increases tagging efficiency by allowing users to reflect candidate categories in untagged terms. It is based on annotations automatically generated using machine learning. Subsequently, this architecture is fine-tuned using Bidirectional Encoder Representations from Transformers (BERT) to enable the tagging of terms that cannot be captured using Named-Entity Recognition (NER). The tagged text data extracted using the proposed tagging tool can be used as an additional training dataset. The tagging tool, which receives and saves new NE annotation input online, is added to the NER and RE web interfaces using BERT. Annotation information downloaded by the user includes the category (e.g., diseases, genes/proteins) and the list of words associated to the named entity selected by the user. The results reveal that the RE and NER results are improved using the proposed web service by collecting more NE annotation data and fine-tuning the model using generated datasets. Our application programming interfaces and demonstrations are available to the public at via the website link provided in this paper.

The traditional topic-based auditing model lacks the ability of multi-object and multi-abnormal correlation auditing, which makes it impossible to solve the multi-scenario and multi-factor correlation-based auditing problem. This paper designs an intelligent power marketing audit model based on a knowledge graph. First, an entity identification and relationship extraction method for power marketing business based on NLP (natural language processing) and sequence annotation technology is proposed, and the description content is imported into the knowledge graph database; then, semantic disambiguation and knowledge are carried out by using bidirectional encoder representation from transformers (BERT). Link to build a knowledge map of business audit rules: Finally, an experimental analysis is carried out by taking the copying and receiving business with a large business volume in the marketing audit work as an example, and it is verified that the proposed model can effectively improve the information analysis ability and the audit accuracy of the audit work.

In this paper we focus on an Natural Language Inference task. Being given two sentences, we classify their relation as NEUTRAL, ENTAILMENT or CONTRADICTION. Considering the achievements of BERT (Bidirectional Encoder Representations from Transformers) in many Natural Language Processing tasks, we use BERT features to create our base model for this task. However, several questions arise: can other features improve the performance obtained with BERT? If we are able to predict the situations in which BERT will fail, can we improve the performance by providing alternative models for these situations? We test several strategies and models, as alternatives to the standalone BERT model in the possible failure situations, and we take advantage of semantic features extracted from Discourse Representation Structures.