Sentiment Analysis of News Headlines Using Simple Transformers

With the development of Internet technology and the transformation of news media to informatization, news images, texts, and other information have exploded. The news image and text classification can effectively solve the disorder problem of news information. The early news image text classification is to establish artificial classifiers according to specific classification rules, but the classification error rate is high and the classification speed is slow. Later, machine learning technology replaced manual classification of news image texts. Although the classification efficiency has been greatly improved, the classification time is still a bit long. The bidirectional encoder representations from transformers (BERT) model uses transformer and encoder to pretrain news image text to improve classification efficiency. By comparing the differences between machine learning and BERT models in news image text classification, the experiments showed that the average precision, recall, and F1 values of the news image text classification algorithm using the BERT model were 96.6%, 95.7%, and 96.1%, respectively. All three evaluation criteria were about 5% more than the classification algorithm of the machine learning model. The classification speed of the news image text classification algorithm using the BERT model was 1.8 times that of the news image text classification algorithm based on support vector machine. Therefore, the news image text classification algorithm using the BERT model can improve the classification accuracy and efficiency of news image text.

Imbalanced aspect categorization using bidirectional encoder representation from transformers

Fenomena bullying di media sosial, khususnya di Twitter, telah menjadi isu yang semakin memprihatinkan dengan dampak signifikan terhadap kesehatan mental pengguna. Dalam rangka mengatasi masalah ini, deteksi otomatis tweet yang mengandung konten bullying menjadi sangat penting. Penelitian ini bertujuan untuk membandingkan performa dua model pemrosesan bahasa alami terbaru, yaitu BERT (Bidirectional Encoder Representations from Transformers) dan XLNet, dalam klasifikasi tweet yang mengandung bullying. Metodologi penelitian ini melibatkan pengumpulan dataset tweet yang telah dilabeli sebagai bullying atau non-bullying. Proses preprocessing teks dilakukan untuk membersihkan dan menyiapkan data sebelum digunakan dalam pelatihan model. Kedua model, BERT dan XLNet, dilatih dan diuji menggunakan dataset yang sama. Evaluasi performa dilakukan dengan menggunakan metrik akurasi, presisi, recall, dan F1-score. Hasil penelitian menunjukkan bahwa kedua model memiliki kemampuan yang baik dalam mengidentifikasi tweet bullying, akan tetapi XLNet menunjukkan performa yang lebih unggul dibandingkan BERT dengan tingkat akurasi sebesar 95%. Dengan nilai presisi = 100%, recall = 0,87%, dan F1-score = 0,88%. XLNet mampu menangkap konteks dan nuansa bahasa yang lebih kompleks dalam tweet, yang berkontribusi pada akurasi klasifikasi yang lebih tinggi. Penelitian ini memberikan kontribusi penting dalam bidang deteksi bullying di media sosial dengan menunjukkan bahwa penggunaan model XLNet lebih efektif dibandingkan BERT. Temuan ini dapat membantu platform seperti Twitter dalam mengidentifikasi dan mencegah konten bullying, sehingga menciptakan lingkungan online yang lebih aman bagi pengguna, serta dapat digunakan sebagai dasar untuk pengembangan sistem deteksi bullying yang lebih canggih dan efisien di masa depan. Abstract The phenomenon of bullying on social media, particularly on Twitter, has become an increasingly concerning issue with significant impacts on users' mental health. In order to address this issue, automatic detection of tweets containing bullying content is crucial. This study aims to compare the performance of two recent natural language processing models, namely BERT (Bidirectional Encoder Representations from Transformers) and XLNet, in the classification of tweets containing bullying. The research methodology involves collecting a dataset of tweets that have been labelled as bullying or non-bullying. Text preprocessing is done to clean and prepare the data before it is used in model training. Both models, BERT and XLNet, were trained and tested using the same dataset. Performance evaluation was conducted using accuracy, precision, recall, and F1-score metrics. The results show that both models have a good ability to identify bullying tweets, but XLNet shows superior performance compared to BERT with an accuracy rate of 95%. With precision = 100%, recall = 0.87%, and F1-score = 0.88%. XLNet is able to capture more complex context and language nuances in tweets, which contributes to higher classification accuracy. This research makes an important contribution to the field of bullying detection on social media by showing that the use of the XLNet model is more effective than BERT. These findings can help platforms like Twitter identify and prevent bullying content, thereby creating a safer online environment for users, and can be used as a basis for the development of more sophisticated and efficient bullying detection systems in the future.

Public opinion is important for decision-making on numerous occasions for national growth in democratic countries like Bangladesh, the USA, and India. Sentiment analysis is a technique used to determine the polarity of opinions expressed in a text. The more complex stage of sentiment analysis is known as Aspect-Based Sentiment Analysis (ABSA), where it is possible to ascertain both the actual topics being discussed by the speakers as well as the polarity of each opinion. Nowadays, people leave comments on a variety of websites, including social networking sites, online news sources, and even YouTube video comment sections, on a wide range of topics. ABSA can play a significant role in utilizing these comments for a variety of objectives, including academic, commercial, and socioeconomic development. In English and many other popular European languages, there are many datasets for ABSA, but the Bengali language has very few of them. As a result, ABSA research on Bengali is relatively rare. In this paper, we present a Bengali dataset that has been manually annotated with five aspects and their corresponding sentiment. A baseline evaluation was also carried out using the Bidirectional Encoder Representations from Transformers (BERT) model, with 97% aspect detection accuracy and 77% sentiment classification accuracy. For aspect detection, the F1-score was 0.97 and for sentiment classification, it was 0.77.

Sentiment analysis of news media articles is essential for understanding the dynamics of conflict and cooperation in transboundary rivers. However, it is not known which machine learning model(s) can best meet the requirement of sentiment analysis for transboundary rivers. This study presents a comparative examination of ten machine learning models commonly used in the field of text sentiment analysis, including K-Nearest Neighbors, Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, Gradient Boosting Decision Tree, Extreme Gradient Boosting, Multilayer Perceptron, Long Short-Term Memory and Bidirectional Encoder Representations from Transformers, for five-class sentiment classification of 9382 news articles (1977–2022) attending to transboundary water conflict and cooperation. By evaluating their performance in terms of accuracy, precision, recall and F1-score, the Bidirectional Encoder Representations from Transformers (BERT) model demonstrated good overall performance and prediction capabilities for news articles with conflictive sentiments. By comparing with the AFINN sentiment dictionary, BERT showed superior performance in the prediction and identification of conflictive sentiment labels. And by validating against historical water events in the three river basins, BERT performed best in the Indus River basin. The findings of this study hold significant implications for government agencies in transboundary rivers, allowing them to promptly assess and respond to public sentiment, thereby preventing water conflict and promoting water cooperation.

As social media and online social networking sites have emerged as essential channels for everyday social interactions, researchers are increasingly turning to conversational social media as a valuable alternative source of insights into public sentiment and behavior during times of crisis. In a similar vein, our study investigates the utility of using conversational social media data for analyzing public sentiment related to the COVID-19 pandemic. Our research question is, to what extent can machine learning models based on conversational social media data help gauge public sentiment during a pandemic? To evaluate the performance of various machine learning algorithms in sentiment analysis tasks, we used a Twitter dataset and compared their effectiveness using eight different models (i.e., Decision Tree (DT), Random Forest (RF), Naïve Bayes (NB), Extreme Gradient Boosting (XGBoost), Logistic Regression, Support Vector Machine (SVM), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representations from Transformers (BERT)). The results show that among the eight models, the top three models are BERT, LSTM, and SVM, achieving overall accuracies of 87.44%, 81.94%, and 80.02%, respectively. Employing sentiment analysis models on pandemic-related social media posts allows us to gain insights into of human emotions and sentiments during public health crises like a pandemic.

Given the foundational role of system requirements in design projects, designers can benefit from classifying, comparing, and observing connections between requirements. Manually undertaking these processes, however, can be laborious and time-consuming. Previous studies have employed Bidirectional Encoder Representations from Transformers (BERT), a state-of-the-art natural language processing (NLP) deep neural network model, to automatically analyze written requirements. Yet, it remains unclear whether BERT can sufficiently capture the nuances that differentiate requirements between and within design documents. This work evaluates BERT’s performance on two requirement classification tasks (one inter- document and one intra-document) executed on a corpus of 1,303 requirements sourced from five system design projects. First, in the “parent document classification” task, a BERT model is fine-tuned to classify requirements according to their originating project. A separate BERT model is then fine-tuned on a “functional classification” task where each requirement is classified as either functional or nonfunctional. Our results also include a comparison with a baseline model, Word2Vec, and demonstrate that our model achieves higher classification accuracy. When evaluated on test sets, the former model receives a Matthews correlation coefficient (MCC) of 0.95, while the latter receives an MCC of 0.82, indicating BERT’s ability to reliably distinguish requirements. This work then explores the application of BERT’s representations, known as embeddings, to identify similar requirements and predict requirement change.

We study the problem of incorporating prior knowledge into a deep Transformer-based model,i.e.,Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.

To develop and evaluate domain-specific and pretrained bidirectional encoder representations from transformers (BERT) models in a transfer learning task on varying training dataset sizes to annotate a larger overall dataset. The authors retrospectively reviewed 69 095 anonymized adult chest radiograph reports (reports dated April 2020-March 2021). From the overall cohort, 1004 reports were randomly selected and labeled for the presence or absence of each of the following devices: endotracheal tube (ETT), enterogastric tube (NGT, or Dobhoff tube), central venous catheter (CVC), and Swan-Ganz catheter (SGC). Pretrained transformer models (BERT, PubMedBERT, DistilBERT, RoBERTa, and DeBERTa) were trained, validated, and tested on 60%, 20%, and 20%, respectively, of these reports through fivefold cross-validation. Additional training involved varying dataset sizes with 5%, 10%, 15%, 20%, and 40% of the 1004 reports. The best-performing epochs were used to assess area under the receiver operating characteristic curve (AUC) and determine run time on the overall dataset. The highest average AUCs from fivefold cross-validation were 0.996 for ETT (RoBERTa), 0.994 for NGT (RoBERTa), 0.991 for CVC (PubMedBERT), and 0.98 for SGC (PubMedBERT). DeBERTa demonstrated the highest AUC for each support device trained on 5% of the training set. PubMedBERT showed a higher AUC with a decreasing training set size compared with BERT. Training and validation time was shortest for DistilBERT at 3 minutes 39 seconds on the annotated cohort. Pretrained and domain-specific transformer models required small training datasets and short training times to create a highly accurate final model that expedites autonomous annotation of large datasets.Keywords: Informatics, Named Entity Recognition, Transfer Learning Supplemental material is available for this article. ©RSNA, 2022See also the commentary by Zech in this issue.

Natural Language Processing (NLP) is a pivotal area in artificial intelligence, aiming to make computers capable of understanding and generating human language. This study evaluates and compares three prominent NLP models—the Generative Pre-trained Transformer (GPT) model, Bidirectional Encoder Representations from Transformers (BERT) model, and Generalized Autoregressive Pretraining for Language Understanding (XLNet)—to determine their strengths, limitations, and suitability for various tasks. The research involves a comprehensive analysis of these models, utilizing well-established datasets such as the Stanford Question Answering Dataset (SQuAD), General Language Understanding Evaluation (GLUE), Reading Comprehension from Examinations (RACE), and the Situations with Adversarial Generations (SWAG). The study explores each model's architecture, pre-training, and fine-tuning processes: GPT’s unidirectional approach is assessed for its language generation and handling of long-range dependencies; Bidirectional encoding is examined for its effectiveness in context understanding, and XLNet permutation-based training is analyzed for its robust contextual comprehension. The experimental results reveal that GPT excels in generative tasks but is constrained by its unidirectional nature. BERT achieves superior accuracy in comprehension tasks but is computationally demanding and susceptible to pre-training bias. XLNet outperforms both GPT and BERT in accuracy and contextual understanding, though at the cost of increased complexity. The results offer a significant understanding of the effectiveness and applicability of these models, suggesting future research directions such as hybrid models and improvements in efficiency.

The omnipresence of smartphones helps people to declare an emergency in realtime. In times of crisis, Twitter has become a big communication platform. As a result, more organizations are interested in tracking Twitter programmatically. Although governments and emergency management agencies work together through their respective national system for response to disasters, the sentiments of the people affected during and after the catastrophe decide the effectiveness of the disaster response and the recovery process. In recent years, sentiment analysis via Twitter-based machine learning has become a common subject. However, the detection of such tweets was often difficult due to the tweets’ language structure’s uncertainty. Thus, it is not always apparent if the words of a person announce a catastrophe. BERT (Bidirectional Encoder Representations by Transformers) is a profound learning model developed by Google. Since Google opened it, several scientists and companies have embraced it and have applied it to many text classification tasks. Therefore, we use BERT in this paper to some dataset disaster tweets. This research will help rescue and emergency responders establish effective knowledge management techniques for a rapidly evolving disaster environment.

Text-based sentiment analytics is an important research direction in the field of natural language processing, and it is widely applied in identifying emotional tendencies. The proliferation of e-commerce platforms has underscored the criticality of emotional tendencies and user experiences delineated in product reviews for gauging product quality and user satisfaction. This study presents a developed text-based sentiment analysis system tailored to product reviews on e-commerce platforms. Three distinct text-based sentiment classification models based on Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representations from Transformers (BERT) were constructed and implemented. Subsequently, sentiment classification and comparative experiments were conducted utilizing real-world product review data. The experimental outcomes underscore that the BERT model outperforms others in terms of the Area Under the Curve (AUC) metric, yielding superior results in the domain of text sentiment analysis. This paper extensively deliberates on the fusion of deep learning methodologies into sentiment analytics, encapsulating the entire spectrum of data collection, preprocessing, and model analysis, with an overarching goal of providing a reference point for research endeavors in analogous realms.

Recently, people used social media to convey their emotions in the text. Text classification is required for group a text containing emotion or not and identify the type of emotion. Most of the research in emotion classification use machine learning methods. So this research offers the use of deep learning for the classification of emotions in text. One of the algorithms that is widely used for deep learning is BERT (Bidirectional Encoder Representations from Transformers). For the experiment, classification is divided into two phases, namely the first phase groups emotion or not emotion tweets, and the second phase groups emotion tweets into five types of emotions, namely happy, angry, sad, scared and surprised. The experiment uses three data scenarios, namely the commuter line, transjakarta and combination data. The BERT method was compared with the BiLSTM and LSTM methods for emotion classification. The result of this study is BERT generates the highest accuracy compared to other methods for the three data scenarios with values of 87.24%, 86.27%, and 86.99% in the first phase. Then, the next accuracy value followed by obtained BiLSTM, and the last is LSTM. In the second phase, BERT also obtained the highest accuracy values for the three data scenarios with values of 94.26%, 93.71 %, and 93.98%. Then, the evaluation using Precision, Recall and F1-Measure results that BERT provides the best overall method. In this study, BERT has good performance for emotion classification.

In this paper, we present our experiments with BERT (Bidirectional Encoder Representations from Transformers) models in the task of sentiment analysis, which aims to predict the sentiment polarity for the given text. We trained an ensemble of BERT models from a large self-collected movie reviews dataset and distilled the knowledge into a single production model. Moreover, we proposed an improved BERT’s pooling layer architecture, which outperforms standard classification layer while enables per-token sentiment predictions. We demonstrate our improvements on a publicly available dataset with Czech movie reviews.

使用社交媒体数据发现旅游目的地：基于 bert 的情感分析。研究目的这项工作的主要目的是使用情感分析技术和来自 Twitter 和 Instagram 的数据来分析旅游目的地, 以便找到最具代表性的实体（或地点）和用户的感知（或方面）。研究设计/方法/途径我们使用 90,725 个 Instagram 帖子和 235,755 个 Twitter 推文来分析格拉纳达（西班牙）的旅游业, 以确定旅行者在两个社交媒体网站上提到的重要地点和看法。我们使用了几种方法对英语和西班牙语文本进行情感分类, 包括深度学习模型。研究发现测试集中的最佳结果是使用来自Transformers (BERT) 模型的双向编码器表示 (BERT) 用于西班牙语文本和Tweeteval 用于英语文本, 这些结果随后用于分析我们的数据集。然后可以确定最重要的实体和方面, 这反过来又为研究人员、从业人员、旅行者和旅游管理者提供了有趣的见解, 从而可以改进服务并制定更好的营销策略。研究局限性我们提出了一个用于执行情感分类的西班牙旅游 BERT 模型, 以及通过主题标签找到地点并揭示每个地点的重要负面方面的过程。实践意义该研究使管理人员和从业人员能够使用我们发布的西班牙旅游数据集实施西班牙-BERT 模型, 以便在应用程序中采用该数据集, 以找到正面和负面的看法。研究原创性本研究提出了一种如何在旅游领域应用情感分析的新方法。首先, 介绍了评估不同现有模型和工具的方法; 其次, 使用 BERT（深度学习模型）训练模型; 第三, 提出了如何通过标签识别目的地地点的接受度的方法, 最后通过实体和方面的识别来评估用户表达积极性（消极性）的原因。