Imbalanced aspect categorization using bidirectional encoder representation from transformers

Fenomena bullying di media sosial, khususnya di Twitter, telah menjadi isu yang semakin memprihatinkan dengan dampak signifikan terhadap kesehatan mental pengguna. Dalam rangka mengatasi masalah ini, deteksi otomatis tweet yang mengandung konten bullying menjadi sangat penting. Penelitian ini bertujuan untuk membandingkan performa dua model pemrosesan bahasa alami terbaru, yaitu BERT (Bidirectional Encoder Representations from Transformers) dan XLNet, dalam klasifikasi tweet yang mengandung bullying. Metodologi penelitian ini melibatkan pengumpulan dataset tweet yang telah dilabeli sebagai bullying atau non-bullying. Proses preprocessing teks dilakukan untuk membersihkan dan menyiapkan data sebelum digunakan dalam pelatihan model. Kedua model, BERT dan XLNet, dilatih dan diuji menggunakan dataset yang sama. Evaluasi performa dilakukan dengan menggunakan metrik akurasi, presisi, recall, dan F1-score. Hasil penelitian menunjukkan bahwa kedua model memiliki kemampuan yang baik dalam mengidentifikasi tweet bullying, akan tetapi XLNet menunjukkan performa yang lebih unggul dibandingkan BERT dengan tingkat akurasi sebesar 95%. Dengan nilai presisi = 100%, recall = 0,87%, dan F1-score = 0,88%. XLNet mampu menangkap konteks dan nuansa bahasa yang lebih kompleks dalam tweet, yang berkontribusi pada akurasi klasifikasi yang lebih tinggi. Penelitian ini memberikan kontribusi penting dalam bidang deteksi bullying di media sosial dengan menunjukkan bahwa penggunaan model XLNet lebih efektif dibandingkan BERT. Temuan ini dapat membantu platform seperti Twitter dalam mengidentifikasi dan mencegah konten bullying, sehingga menciptakan lingkungan online yang lebih aman bagi pengguna, serta dapat digunakan sebagai dasar untuk pengembangan sistem deteksi bullying yang lebih canggih dan efisien di masa depan. Abstract The phenomenon of bullying on social media, particularly on Twitter, has become an increasingly concerning issue with significant impacts on users' mental health. In order to address this issue, automatic detection of tweets containing bullying content is crucial. This study aims to compare the performance of two recent natural language processing models, namely BERT (Bidirectional Encoder Representations from Transformers) and XLNet, in the classification of tweets containing bullying. The research methodology involves collecting a dataset of tweets that have been labelled as bullying or non-bullying. Text preprocessing is done to clean and prepare the data before it is used in model training. Both models, BERT and XLNet, were trained and tested using the same dataset. Performance evaluation was conducted using accuracy, precision, recall, and F1-score metrics. The results show that both models have a good ability to identify bullying tweets, but XLNet shows superior performance compared to BERT with an accuracy rate of 95%. With precision = 100%, recall = 0.87%, and F1-score = 0.88%. XLNet is able to capture more complex context and language nuances in tweets, which contributes to higher classification accuracy. This research makes an important contribution to the field of bullying detection on social media by showing that the use of the XLNet model is more effective than BERT. These findings can help platforms like Twitter identify and prevent bullying content, thereby creating a safer online environment for users, and can be used as a basis for the development of more sophisticated and efficient bullying detection systems in the future.

With the rate at which the data is being generated, it is vital to use it and get some insights from it. When reporting on events, news expresses the opinions of news entities like people, locations, and things. In this paper, we obtain the sentiment of the news headlines using a new technique called transformers, in particular simple transformers that have been a significant advancement in the natural language processing field. Sentiment Analysis has been performed using the four transformers model. These models are pre-trained on extensive data, and we have fine-tuned them by training them on our own news headlines dataset. For our sentiment analysis task, classification models (specific simple transformer model) are used to classify news headlines as negative, neutral, positive. The idea behind taking four different models that are Bidirectional Encoder Representations from Transformers (BERT) base-cased [1], Robustly optimized BERT approach (RoBERTa) base [1], Distilled BERT (DistilBERT) base-cased [1], XLNet base-cased [1], is the different dataset on which they are pre-trained, parameters used by them, and different method used by them, which improve their performance significantly in comparison to different machine learning classifiers and prior deep learning models. The model that performed the best is bert-base-cased with Matthews Correlation Coefficient (MCC) score of 90.1%, an F1 score of 93.6%, and an Accuracy score of 93.6%.

Twitter has emerged as a critical platform for capturing public sentiment, offering a valuable source for sentiment analysis. This study presents a comparative evaluation of two BERT (Bidirectional Encoder Representations from Transformers) models—baseline and fine-tuned—targeted at analyzing Indonesian-language tweets. Employing the CRISP-DM framework, the methodology encompasses automated data crawling, comprehensive text pre-processing (including case folding, cleaning, tokenisation, normalisation, and data augmentation), and model development using the IndoBERT-base-p1 architecture. The experimental results reveal that the fine-tuned BERT model achieves significantly improved performance over the non-optimized model, with accuracy, precision, recall, and F1-score values reaching 91%, 0.91, 0.90, and 0.91, respectively. These findings indicate the fine-tuned model's superior ability to capture linguistic subtleties and contextual sentiment features within informal social media text. Furthermore, the model is deployed in a web-based application for real-time sentiment classification, demonstrating its practical applicability. This study underscores the effectiveness of Fine-tuning in enhancing BERT-based sentiment analysis for low-resource languages. It highlights its potential for informing decisionmaking in digital communication, marketing, and policy research contexts.

The omnipresence of smartphones helps people to declare an emergency in realtime. In times of crisis, Twitter has become a big communication platform. As a result, more organizations are interested in tracking Twitter programmatically. Although governments and emergency management agencies work together through their respective national system for response to disasters, the sentiments of the people affected during and after the catastrophe decide the effectiveness of the disaster response and the recovery process. In recent years, sentiment analysis via Twitter-based machine learning has become a common subject. However, the detection of such tweets was often difficult due to the tweets’ language structure’s uncertainty. Thus, it is not always apparent if the words of a person announce a catastrophe. BERT (Bidirectional Encoder Representations by Transformers) is a profound learning model developed by Google. Since Google opened it, several scientists and companies have embraced it and have applied it to many text classification tasks. Therefore, we use BERT in this paper to some dataset disaster tweets. This research will help rescue and emergency responders establish effective knowledge management techniques for a rapidly evolving disaster environment.

Recently, people used social media to convey their emotions in the text. Text classification is required for group a text containing emotion or not and identify the type of emotion. Most of the research in emotion classification use machine learning methods. So this research offers the use of deep learning for the classification of emotions in text. One of the algorithms that is widely used for deep learning is BERT (Bidirectional Encoder Representations from Transformers). For the experiment, classification is divided into two phases, namely the first phase groups emotion or not emotion tweets, and the second phase groups emotion tweets into five types of emotions, namely happy, angry, sad, scared and surprised. The experiment uses three data scenarios, namely the commuter line, transjakarta and combination data. The BERT method was compared with the BiLSTM and LSTM methods for emotion classification. The result of this study is BERT generates the highest accuracy compared to other methods for the three data scenarios with values of 87.24%, 86.27%, and 86.99% in the first phase. Then, the next accuracy value followed by obtained BiLSTM, and the last is LSTM. In the second phase, BERT also obtained the highest accuracy values for the three data scenarios with values of 94.26%, 93.71 %, and 93.98%. Then, the evaluation using Precision, Recall and F1-Measure results that BERT provides the best overall method. In this study, BERT has good performance for emotion classification.

We study the problem of incorporating prior knowledge into a deep Transformer-based model,i.e.,Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.

Public opinion is important for decision-making on numerous occasions for national growth in democratic countries like Bangladesh, the USA, and India. Sentiment analysis is a technique used to determine the polarity of opinions expressed in a text. The more complex stage of sentiment analysis is known as Aspect-Based Sentiment Analysis (ABSA), where it is possible to ascertain both the actual topics being discussed by the speakers as well as the polarity of each opinion. Nowadays, people leave comments on a variety of websites, including social networking sites, online news sources, and even YouTube video comment sections, on a wide range of topics. ABSA can play a significant role in utilizing these comments for a variety of objectives, including academic, commercial, and socioeconomic development. In English and many other popular European languages, there are many datasets for ABSA, but the Bengali language has very few of them. As a result, ABSA research on Bengali is relatively rare. In this paper, we present a Bengali dataset that has been manually annotated with five aspects and their corresponding sentiment. A baseline evaluation was also carried out using the Bidirectional Encoder Representations from Transformers (BERT) model, with 97% aspect detection accuracy and 77% sentiment classification accuracy. For aspect detection, the F1-score was 0.97 and for sentiment classification, it was 0.77.

Bali is one of the provinces with the most abundant natural and cultural wealth in Indonesia. One commodity that supports it is coffee. Bali Coffee is not only a gastronomic identity, but also a cultural identity which makes it have added value to be developed into various business lines. One business derivative that is quite promising is a coffee shop. However, these favorable conditions also need to be maintained to ensure good quality reaches consumers. One thing that can do is analyze reviews from customers. One of the most popular methods is Sentiment Analysis. This technique allows business to analyze customer reviews on social media. It can be a feedback to maintaining and improving quality and good relationships with customers. This research aims to create a machine learning model to analyze customer reviews at several coffee shops in Bali which are divided into three labels, namely: positive, negative and neutral. The methods used are: scraping, cleaning, stopword removal, embedding, undersampling, and modeling. The algorithms used are Bidirectional Encoder Representation from Transformer (BERT) and Generative Pre-trained Transformers (GPT). The performance metrics used in this research are precision, recall, accuracy and loss. This research succeeded in creating a sentiment analysis model for coffee shop customers in Bali. The BERT model obtained an accuracy value of 78% without undersampling with a loss in the 10th iteration of 0.27. Meanwhile, the BERT model with undersampling obtained an accuracy value of 32.85% with a loss in the 10th iteration of 0.16. The GPT2 model without undersampling gets an accuracy of 78% with a loss in the 10th iteration of 0.25. Meanwhile, the GPT model with undersampling obtained an accuracy value of 32.85% with a loss in the 10th iteration of 0.15.

Sentiment analysis of news media articles is essential for understanding the dynamics of conflict and cooperation in transboundary rivers. However, it is not known which machine learning model(s) can best meet the requirement of sentiment analysis for transboundary rivers. This study presents a comparative examination of ten machine learning models commonly used in the field of text sentiment analysis, including K-Nearest Neighbors, Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, Gradient Boosting Decision Tree, Extreme Gradient Boosting, Multilayer Perceptron, Long Short-Term Memory and Bidirectional Encoder Representations from Transformers, for five-class sentiment classification of 9382 news articles (1977–2022) attending to transboundary water conflict and cooperation. By evaluating their performance in terms of accuracy, precision, recall and F1-score, the Bidirectional Encoder Representations from Transformers (BERT) model demonstrated good overall performance and prediction capabilities for news articles with conflictive sentiments. By comparing with the AFINN sentiment dictionary, BERT showed superior performance in the prediction and identification of conflictive sentiment labels. And by validating against historical water events in the three river basins, BERT performed best in the Indus River basin. The findings of this study hold significant implications for government agencies in transboundary rivers, allowing them to promptly assess and respond to public sentiment, thereby preventing water conflict and promoting water cooperation.

Text-based sentiment analytics is an important research direction in the field of natural language processing, and it is widely applied in identifying emotional tendencies. The proliferation of e-commerce platforms has underscored the criticality of emotional tendencies and user experiences delineated in product reviews for gauging product quality and user satisfaction. This study presents a developed text-based sentiment analysis system tailored to product reviews on e-commerce platforms. Three distinct text-based sentiment classification models based on Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representations from Transformers (BERT) were constructed and implemented. Subsequently, sentiment classification and comparative experiments were conducted utilizing real-world product review data. The experimental outcomes underscore that the BERT model outperforms others in terms of the Area Under the Curve (AUC) metric, yielding superior results in the domain of text sentiment analysis. This paper extensively deliberates on the fusion of deep learning methodologies into sentiment analytics, encapsulating the entire spectrum of data collection, preprocessing, and model analysis, with an overarching goal of providing a reference point for research endeavors in analogous realms.

In this paper we focus on an Natural Language Inference task. Being given two sentences, we classify their relation as NEUTRAL, ENTAILMENT or CONTRADICTION. Considering the achievements of BERT (Bidirectional Encoder Representations from Transformers) in many Natural Language Processing tasks, we use BERT features to create our base model for this task. However, several questions arise: can other features improve the performance obtained with BERT? If we are able to predict the situations in which BERT will fail, can we improve the performance by providing alternative models for these situations? We test several strategies and models, as alternatives to the standalone BERT model in the possible failure situations, and we take advantage of semantic features extracted from Discourse Representation Structures.

As the online shopping industry continues to grow at a staggering rate, e-commerce channels have also begun the use of automated systems for sentiment analysis so as to be able to assess comprehension of user reviews. Over the past few years, machine learning (ML) has come of age with the introduction of deep learning models such as Convolutional Neural Networks (CNNs) and Bidirectional Encoder Representations from Transformers (BERT) which have greatly improved the levels of accuracy achieved in sentiment analysis. This paper aims to address this problem by proposing a hybrid model, based on networks of convolutional neural networks (CNNs) and BERT, for the purposes of e-commerce sentiment classification. Posting more plausible situations, academic findings, and perspectives for future research, we focus more on practical issues. When we compared traditional approaches with our hybrid, it became clear that the model performs better in many dimensions: accuracy, precision, and progressively recall

The internet’s prevalence has increased the use of social media platforms for sharing peoples’ views, reviews, opinions, and sentiments. Consequently, sentiment analysis has become valuable to marketing, brand management, public policy, healthcare, and e-commerce. Panic detection systems refer to a system that utilizes sentiment analysis techniques to identify, quantify, and potentially track expressions of panic, fear, or related strong negative emotions within digital content. Sentiment analysis application can indicate a person’s attitude or cultural influences. The rise of Arabic digital content on social platforms presents unique challenges and opportunities for sentiment analysis. There are fewer Arabic sentiment lexicons, emotional connotations, and annotated corpora than in English. Therefore, this study applies machine learning and deep learning to identify sentiments within the IRAQI-Arabic panic detection systems. The proposed model is based on Residual Network (ResNet), MobileNet, and Convolutional Neural Network (CNN). A web scraping API collected COVID-19-related news from Google, Facebook, Twitter, and BBC News. The model used was retrained on the images and comments based on training, public, and private sub-sets. Bidirectional Encoder Representations from Transformers (BERT) were applied across SMOTE, Random oversampling, five-folds, and ten folds. The results show that the accuracy of BERT is 91%. The proposed MobileNet model had an accuracy of 0.98, recall of 0.98, and MAE of 0.032. The model’s performance confirms that it is suitable for critical applications that require high-level precision. The study offers a novel model for Arabic panic systems. Also, it presents an Iraqi-Arabic dataset tailored for panic detection.

Deep learning approaches, utilizing Bidirectional Encoder Representation from Transformers (BERT) and advanced fine-tuning techniques, have achieved state-of-the-art accuracies in the domain of term extraction from texts. However, BERT presents some limitations in that it primarily captures the semantic context relative to the surrounding text without considering how relevant or central a token is to the overall document content. There has also been research on the application of sequence labeling on contextualized embeddings; however, the existing methods often rely solely on local context for extracting key phrases from texts. To address these limitations, this study proposes a centrality-weighted BERT model for key phrase extraction from text using sequence labelling (CenBERT-SEQ). The proposed CenBERT-SEQ model utilizes BERT to represent terms with various contextual embedding architectures, and introduces a centrality-weighting layer that integrates document-level context into BERT. This layer leverages document embeddings to influence the importance of each term based on its relevance to the entire document. Finally, a linear classifier layer is employed to model the dependencies between the outputs, thereby enhancing the accuracy of the CenBERT-SEQ model. The proposed CenBERT-SEQ model was evaluated against the standard BERT base-uncased model using three Computer Science article datasets, namely, SemEval-2010, WWW, and KDD. The experimental results show that, although the CenBERT-SEQ and BERT-base models achieved higher and close comparable accuracy, the proposed CenBERT-SEQ model achieved higher precision, recall, and F1-score than the BERT-base model. Furthermore, a comparison of the proposed CenBERT-SEQ model to that of related studies revealed that the proposed CenBERT-SEQ model achieved a higher accuracy, precision, recall, and F1-score of 95%, 97%, 91%, and 94%, respectively, than related studies, showing the superior capabilities of the CenBERT-SEQ model in keyphrase extraction from scientific documents.

Text classification algorithms investigate the intricate relationships between words or phrases and attempt to deduce the document's interpretation. In the last few years, these algorithms have progressed tremendously. Transformer architecture and sentence encoders have proven to give superior results on natural language processing tasks. But a major limitation of these architectures is their applicability for text no longer than a few hundred words. In this paper, we explore hierarchical transfer learning approaches for long document classification. We employ pre-trained Universal Sentence Encoder (USE) and Bidirectional Encoder Representations from Transformers (BERT) in a hierarchical setup to capture better representations efficiently. Our proposed models are conceptually simple where we divide the input data into chunks and then pass this through base models of BERT and USE. Then output representation for each chunk is then propagated through a shallow neural network comprising of LSTMs or CNNs for classifying the text data. These extensions are evaluated on 6 benchmark datasets. We show that USE + CNN/LSTM performs better than its stand-alone baseline. Whereas the BERT + CNN/LSTM performs on par with its stand-alone counterpart. However, the hierarchical BERT models are still desirable as it avoids the quadratic complexity of the attention mechanism in BERT. Along with the hierarchical approaches, this work also provides a comparison of different deep learning algorithms like USE, BERT, HAN, Longformer, and BigBird for long document classification. The Longformer approach consistently performs well on most of the datasets.