Exploiting Vulnerabilities of Machine Learning Models on Medical Text via Generative Adversarial Attacks

In this study, we explore the potential of machine learning (ML) models after different text representation methods on the balanced IMDB dataset, which is widely regarded as a gold standard in sentiment analysis, one of the Natural Language processing (NLP) tasks. On the open source IMDB movie reviews dataset, we first undertake data cleaning and text representation with data preprocessing steps. Then, we apply sentiment classification using different ML models. In order to evaluate the models, we used precision (P), recall (R), F1-score (F1), and area under curve (AUC), as well as receiver operating characteristic (ROC). It is worth noting that text feature extraction with Bidirectional Encoder Representations from Transformers (BERT) provided the highest performance in all models, with the SVM model offering particularly promising results. In this model, we observed the following results: ACC 0.9033, F1 0.9308, R 0.9015, R 0.9015, P 0.9072, AUC 0.9638, and ROC 0.96. These findings suggest that NLP techniques and, in particular, machine learning models that employ BERT may offer high levels of accuracy and reliability in text classification problems. It would be beneficial for future studies to validate these findings using BERT on different NLP tasks. This would help to evaluate the effectiveness and applicability of the models in practice.

The problem of the existence of news article that does not match with content, called clickbait, has seriously interfered readers from getting the information they expect. The number of clickbait news continues significantly increased in recent years. According to this problem, a clickbait detector is required to automatically identify news article headlines that include clickbait and non-clickbait. Additionally, many currently existing solutions use handcrafted features and traditional machine learning methods, which limit the generalization. Therefore, this study fine-tunes the Bidirectional Encoder Representations from Transformers (BERT) and uses the Indonesian news headlines dataset CLICK-ID to predict clickbait (BERT). In this research, we use IndoBERT as the pre-trained model, a state-of-the-art BERT-based language model for Indonesian. Then, the usefulness of BERT-based classifiers is then assessed by comparing the performance of IndoBERT classifiers with different pre-trained models with that of two word-vectors-based approaches (i.e., bag-of-words and TF-IDF) and five machine learning classifiers (i.e., NB, KNN, SVM, DT, and RF). The evaluation results indicate that all fine-tuned IndoBERT classifiers outperform all word-vectors-based machine learning classifiers in classifying clickbait and non-clickbait Indonesian news headlines. The IndoBERTBASE using the two training phases model gets the highest accuracy of 0.8247, which is 0.064 (6%), outperforming the SVM classifier's accuracy with the bag-of-words model 0.7607.

With the development of Internet technology and the transformation of news media to informatization, news images, texts, and other information have exploded. The news image and text classification can effectively solve the disorder problem of news information. The early news image text classification is to establish artificial classifiers according to specific classification rules, but the classification error rate is high and the classification speed is slow. Later, machine learning technology replaced manual classification of news image texts. Although the classification efficiency has been greatly improved, the classification time is still a bit long. The bidirectional encoder representations from transformers (BERT) model uses transformer and encoder to pretrain news image text to improve classification efficiency. By comparing the differences between machine learning and BERT models in news image text classification, the experiments showed that the average precision, recall, and F1 values of the news image text classification algorithm using the BERT model were 96.6%, 95.7%, and 96.1%, respectively. All three evaluation criteria were about 5% more than the classification algorithm of the machine learning model. The classification speed of the news image text classification algorithm using the BERT model was 1.8 times that of the news image text classification algorithm based on support vector machine. Therefore, the news image text classification algorithm using the BERT model can improve the classification accuracy and efficiency of news image text.

Abstract—This paper presents a hybrid sentiment analysis system that integrates lexicon-based, traditional machine learning, and deep learning techniques to classify textual data into positive, negative, or neutral sentiments. The system leverages Python-based libraries such as Scikit-learn, NLTK, and Transformers to pre- process text, extract features, and apply models in- cluding Support Vector Machine (SVM), Bidirectional Encoder Representations from Transformers (BERT), Linear Regression, and Valence Aware Dictionary and sEntiment Reasoner (VADER). The framework pro- cesses customer reviews from e-commerce platforms (Amazon, Flipkart) and social media (Instagram) using web scraping and provides actionable insights through sentiment summarization and visualizations (bar and pie charts). Experimental results demonstrate BERT’s superior performance with 92.3% accuracy, followed by SVM (85.6%), Linear Regression (81.2%), and VADER (76.8%). The system addresses challenges like sarcasm, class imbalance, and scalability, offering a scalable, user-friendly solution for real-world applications in e- commerce, social media analytics, and brand reputation management. Index Terms—Sentiment Analysis, Machine Learning, Deep Learning, Natural Language Processing, BERT, SVM, VADER, Text Classification, Data Visualization.

Communication-based train control (CBTC) systems are automated train control systems based on continuous and bidirectional train-ground communication. As one of the critical subsystems of the safety-demanding CBTC systems, the Data Communication System (DCS) is physically exposed to the environment. DCS faults may lead to delayed trains, stranded passengers, suspension of service, or even catastrophic losses of lives or assets. Therefore, it is quite desirable to identify the DCS faults and accordingly take effective maintenance measures. Most fault classification works are based on traditional Machine Learning (ML) techniques, such as Decision Tree (DT) and Bayesian Network. A substantial amount of fault information can be lost when we manually extract features from the system logs. This paper proposes a Deep Hybrid Learning (DHL) method to identify the DCS faults. A pre-trained Chinese Bidi-rectional Encoder Representation from Transformers (BERT) Deep Learning (DL) model is used to extract fault features from raw text logs automatically, and several traditional ML models are used to classify the system faults by the extracted features. Our study highlights the advantages of the DHL model in DCS fault diagnosis tasks. The intense simulation results illustrate that the proposed DHL model achieves substantial accuracy compared with traditional ML algorithms.

Fenomena bullying di media sosial, khususnya di Twitter, telah menjadi isu yang semakin memprihatinkan dengan dampak signifikan terhadap kesehatan mental pengguna. Dalam rangka mengatasi masalah ini, deteksi otomatis tweet yang mengandung konten bullying menjadi sangat penting. Penelitian ini bertujuan untuk membandingkan performa dua model pemrosesan bahasa alami terbaru, yaitu BERT (Bidirectional Encoder Representations from Transformers) dan XLNet, dalam klasifikasi tweet yang mengandung bullying. Metodologi penelitian ini melibatkan pengumpulan dataset tweet yang telah dilabeli sebagai bullying atau non-bullying. Proses preprocessing teks dilakukan untuk membersihkan dan menyiapkan data sebelum digunakan dalam pelatihan model. Kedua model, BERT dan XLNet, dilatih dan diuji menggunakan dataset yang sama. Evaluasi performa dilakukan dengan menggunakan metrik akurasi, presisi, recall, dan F1-score. Hasil penelitian menunjukkan bahwa kedua model memiliki kemampuan yang baik dalam mengidentifikasi tweet bullying, akan tetapi XLNet menunjukkan performa yang lebih unggul dibandingkan BERT dengan tingkat akurasi sebesar 95%. Dengan nilai presisi = 100%, recall = 0,87%, dan F1-score = 0,88%. XLNet mampu menangkap konteks dan nuansa bahasa yang lebih kompleks dalam tweet, yang berkontribusi pada akurasi klasifikasi yang lebih tinggi. Penelitian ini memberikan kontribusi penting dalam bidang deteksi bullying di media sosial dengan menunjukkan bahwa penggunaan model XLNet lebih efektif dibandingkan BERT. Temuan ini dapat membantu platform seperti Twitter dalam mengidentifikasi dan mencegah konten bullying, sehingga menciptakan lingkungan online yang lebih aman bagi pengguna, serta dapat digunakan sebagai dasar untuk pengembangan sistem deteksi bullying yang lebih canggih dan efisien di masa depan. Abstract The phenomenon of bullying on social media, particularly on Twitter, has become an increasingly concerning issue with significant impacts on users' mental health. In order to address this issue, automatic detection of tweets containing bullying content is crucial. This study aims to compare the performance of two recent natural language processing models, namely BERT (Bidirectional Encoder Representations from Transformers) and XLNet, in the classification of tweets containing bullying. The research methodology involves collecting a dataset of tweets that have been labelled as bullying or non-bullying. Text preprocessing is done to clean and prepare the data before it is used in model training. Both models, BERT and XLNet, were trained and tested using the same dataset. Performance evaluation was conducted using accuracy, precision, recall, and F1-score metrics. The results show that both models have a good ability to identify bullying tweets, but XLNet shows superior performance compared to BERT with an accuracy rate of 95%. With precision = 100%, recall = 0.87%, and F1-score = 0.88%. XLNet is able to capture more complex context and language nuances in tweets, which contributes to higher classification accuracy. This research makes an important contribution to the field of bullying detection on social media by showing that the use of the XLNet model is more effective than BERT. These findings can help platforms like Twitter identify and prevent bullying content, thereby creating a safer online environment for users, and can be used as a basis for the development of more sophisticated and efficient bullying detection systems in the future.

Sentiment analysis of news media articles is essential for understanding the dynamics of conflict and cooperation in transboundary rivers. However, it is not known which machine learning model(s) can best meet the requirement of sentiment analysis for transboundary rivers. This study presents a comparative examination of ten machine learning models commonly used in the field of text sentiment analysis, including K-Nearest Neighbors, Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, Gradient Boosting Decision Tree, Extreme Gradient Boosting, Multilayer Perceptron, Long Short-Term Memory and Bidirectional Encoder Representations from Transformers, for five-class sentiment classification of 9382 news articles (1977–2022) attending to transboundary water conflict and cooperation. By evaluating their performance in terms of accuracy, precision, recall and F1-score, the Bidirectional Encoder Representations from Transformers (BERT) model demonstrated good overall performance and prediction capabilities for news articles with conflictive sentiments. By comparing with the AFINN sentiment dictionary, BERT showed superior performance in the prediction and identification of conflictive sentiment labels. And by validating against historical water events in the three river basins, BERT performed best in the Indus River basin. The findings of this study hold significant implications for government agencies in transboundary rivers, allowing them to promptly assess and respond to public sentiment, thereby preventing water conflict and promoting water cooperation.

People post their emotional expressions about specific events through social media platforms, such as praise and complaints about city bus travel services. In this paper, we propose a social media-based method to analyze bus service satisfaction. Firstly, we collect microblogs related to bus service from Sina Weibo, and pre-process the collected microblogs. Then the Bidirectional Encoder Representations from Transformers (BERT) pre-trained model is combined with Bi-directional Long Short-Term Memory (BiLSTM) model to form BERT- BiLSTM model, using the pre-processed data to train the BERT-BiLSTM model to realize the classification of text data into positive and negative sentiments, and finally using the method of this paper to analyze and verify the satisfaction of Guangzhou bus services system in 2010. Comparing the method of this paper with traditional machine learning and deep learning methods, the experiments show the effectiveness of the BERT-BiLSTM model. KeywordsSocial mediaBus serviceSentiment analysisBERT-BiLSTMBinary classification

Recently, people used social media to convey their emotions in the text. Text classification is required for group a text containing emotion or not and identify the type of emotion. Most of the research in emotion classification use machine learning methods. So this research offers the use of deep learning for the classification of emotions in text. One of the algorithms that is widely used for deep learning is BERT (Bidirectional Encoder Representations from Transformers). For the experiment, classification is divided into two phases, namely the first phase groups emotion or not emotion tweets, and the second phase groups emotion tweets into five types of emotions, namely happy, angry, sad, scared and surprised. The experiment uses three data scenarios, namely the commuter line, transjakarta and combination data. The BERT method was compared with the BiLSTM and LSTM methods for emotion classification. The result of this study is BERT generates the highest accuracy compared to other methods for the three data scenarios with values of 87.24%, 86.27%, and 86.99% in the first phase. Then, the next accuracy value followed by obtained BiLSTM, and the last is LSTM. In the second phase, BERT also obtained the highest accuracy values for the three data scenarios with values of 94.26%, 93.71 %, and 93.98%. Then, the evaluation using Precision, Recall and F1-Measure results that BERT provides the best overall method. In this study, BERT has good performance for emotion classification.

The rising complexity of cyberattacks poses significant challenges to traditional intrusion detection system (IDS). Approaches that rely mainly on feature engineering and rule matching often fail to address new and complex attack patterns effectively. To address these challenges, this paper proposes a network IDS based on bidirectional encoder representations from transformers (BERT), which aims to leverage recent advances in the field of natural language processing (NLP) to enhance the performance of IDS. The core of our approach lies in converting network traffic data into textual representations that can be processed by applying advanced NLP techniques. Specifically, we first convert various types of information in network traffic into a natural-language-like sequence representation, which allows us to utilize a pre-trained BERT model for feature extraction. The BERT model, with its bidirectional encoder structure, captures complex contextual information in sequences, producing high-quality feature representations. After feature extraction is completed, we further perform classification using a classifier based on a bidirectional gated recurrent unit (GRU) network, which is able to efficiently process sequence data and reduce computational complexity while capturing temporal dependencies in the sequences. By training the classifier on the basis of the extracted features, we are able to achieve an accurate distinction between normal and abnormal traffic. To validate the effectiveness of our approach, we conducted experiments on several public datasets, and the experimental results show that the BERT-based network IDS significantly outperforms traditional machine learning (ML) and deep learning (DL) methods in terms of detection performance. The BERT model enhances detection accuracy and effectively captures complex attack patterns, compared to traditional methods. Our research not only demonstrates the great potential of NLP technology in the field of network security but also provides a new research direction for cross-domain applications. By treating network traffic as a “language,” our approach resolves issues that traditional methods struggle with while being scalable and adaptable to various network environments and attack patterns.

We study the problem of incorporating prior knowledge into a deep Transformer-based model,i.e.,Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.

Mild cognitive impairment (MCI) is a clinical condition characterized by a decline in cognitive ability and progression of cognitive impairment. It is often considered a transitional stage between normal aging and Alzheimer's disease (AD). This study aimed to compare deep learning (DL) and traditional machine learning (ML) methods in predicting MCI using plasma proteomic biomarkers. A total of 239 adults were selected from the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort along with a pool of 146 plasma proteomic biomarkers. We evaluated seven traditional ML models (support vector machines (SVMs), logistic regression (LR), naïve Bayes (NB), random forest (RF), k-nearest neighbor (KNN), gradient boosting machine (GBM), and extreme gradient boosting (XGBoost)) and six variations of a deep neural network (DNN) model-the DL model in the H2O package. Least Absolute Shrinkage and Selection Operator (LASSO) selected 35 proteomic biomarkers from the pool. Based on grid search, the DNN model with an activation function of "Rectifier With Dropout" with 2 layers and 32 of 35 selected proteomic biomarkers revealed the best model with the highest accuracy of 0.995 and an F1 Score of 0.996, while among seven traditional ML methods, XGBoost was the best with an accuracy of 0.986 and an F1 Score of 0.985. Several biomarkers were correlated with the APOE-ε4 genotype, polygenic hazard score (PHS), and three clinical cerebrospinal fluid biomarkers (Aβ42, tTau, and pTau). Bioinformatics analysis using Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) revealed several molecular functions and pathways associated with the selected biomarkers, including cytokine-cytokine receptor interaction, cholesterol metabolism, and regulation of lipid localization. The results showed that the DL model may represent a promising tool in the prediction of MCI. These plasma proteomic biomarkers may help with early diagnosis, prognostic risk stratification, and early treatment interventions for individuals at risk for MCI.

Bidirectional Encoder Representations from Transformers (BERT) models achieve state-of-the-art performance on a number of Natural Language Processing tasks. However, their model size on disk often exceeds 1 GB and the process of fine-tuning them and using them to run inference consumes significant hardware resources and runtime. This makes them hard to deploy to production environments. This paper fine-tunes DistilBERT, a lightweight deep learning model, on medical text for the named entity recognition task of Protected Health Information (PHI) and medical concepts. This work provides a full assessment of the performance of DistilBERT in comparison with BERT models that were pre-trained on medical text. For Named Entity Recognition task of PHI, DistilBERT achieved almost the same results as medical versions of BERT in terms of F1 score at almost half the runtime and consuming approximately half the disk space. On the other hand, for the detection of medical concepts, DistilBERT's F1 score was lower by 4 points on average than medical BERT variants.

In this paper we focus on an Natural Language Inference task. Being given two sentences, we classify their relation as NEUTRAL, ENTAILMENT or CONTRADICTION. Considering the achievements of BERT (Bidirectional Encoder Representations from Transformers) in many Natural Language Processing tasks, we use BERT features to create our base model for this task. However, several questions arise: can other features improve the performance obtained with BERT? If we are able to predict the situations in which BERT will fail, can we improve the performance by providing alternative models for these situations? We test several strategies and models, as alternatives to the standalone BERT model in the possible failure situations, and we take advantage of semantic features extracted from Discourse Representation Structures.

Complex Social Engineering (SE) attacks by manipulating people psychologically to mislead them and to weaken the security systems have become quite common. This research investigates the patterns of SE attacks through concepts of behavioral psychology integrated with protection frameworks driven by AI (Artificial Intelligence). This study evaluates significant weaknesses in the behaviour of human beings which hackers use against them by analysing the strategies of attackers such as gathering data, phishing techniques and exploitation of trust. We provide an enhanced protection architecture that includes BERT (Bidirectional Encoder Representations from Transformers) to deal with such attacks. Through the evaluation of text semantics in messages, emails, and websites, BERT's deep learning (DL) ability helps them detect phishing content, suspicious patterns of language, and fraudulent messages. Our approach decreases false positives and enhances contextual knowledge thereby improving traditional models used for detection. Experimental outcomes prove that BERT is more accurate in identifying harmful content than conventional Machine Learning (ML) approaches. This research focuses on the fact that the knowledge of behavioral psychology and AI-driven approaches can make cybersecurity systems more efficient and decrease the risk of SE attacks.