Eliminating Sentiment Bias for Aspect-Level Sentiment Classification with Unsupervised Opinion Extraction

Aspect-level sentiment classification (ASC) is a fine-grained sentiment analysis task that involves detecting the sentiment polarity of a specific opinion target in a given sentence. Despite the popularity of deep learning methods, the limited availability of ASC training data has resulted in the suboptimal performance of neural network models. To address this issue, researchers have proposed transferring resource-rich document-level sentiment knowledge to ASC tasks using model transfer and shared parameters. However, these methods have neglected the impact of differences in the training data for different tasks on the shared layer parameters. Therefore, this paper proposes a transfer method that automatically adapts to task and domain differences—Auto-adaptive Model Transfer (AAMT). This method considers adding specific disturbance variables on shared layer parameters to subdivide task differences to further obtain more accurate and effective transfer knowledge. In addition, on this basis, we introduce an attention mechanism to combine aspect and sentence features to better capture the sentiment vocabulary associated with a given target. A series of experiments on three public datasets show that our proposed method can solve the problem of insufficient training data for ASC and achieve good results.

AELA-DLSTMs: Attention-Enabled and Location-Aware Double LSTMs for aspect-level sentiment classification

In this article, we present a novel Transformer-Based Aspect-Level Sentiment Classification with Intent (TASCI) model, designed to enhance sentiment analysis by integrating aspect-level sentiment classification with intent analysis. Traditional sentiment analysis methods often overlook the nuanced relationship between the intent behind a statement and the sentiment expressed toward specific aspects of an entity. TASCI addresses this gap by first extracting aspects using a self-attention mechanism and then employing a Transformer-based model to infer the speaker's intent from preceding sentences. This dual approach allows TASCI to contextualize sentiment analysis, providing a more accurate reflection of user opinions. We validate TASCI's performance on three benchmark datasets: Restaurant, Laptop, and Twitter, achieving state-of-the-art results with an accuracy of 89.10% and a macro-F1 score of 83.38% on the Restaurant dataset, 84.81% accuracy and 78.63% macro-F1 score on the Laptop dataset, and 79.08% accuracy and 77.27% macro-F1 score on the Twitter dataset. These results demonstrate that incorporating intent analysis significantly enhances the model's ability to capture complex sentiment expressions across different domains, thereby setting a new standard for aspect-level sentiment classification.

Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (<0.0001% of model weights). In case of BERT and other pre-trained encoder Transformers, the affected component is the scaling factors and biases in the LayerNorm. The outliers are high-magnitude normalization parameters that emerge early in pre-training and show up consistently in the same dimensional position throughout the model. We show that disabling them significantly degrades both the MLM loss and the downstream task performance. This effect is observed across several BERT-family models and other popular pre-trained Transformer architectures, including BART, XLNet and ELECTRA; we also show a similar effect in GPT-2.

In this paper, we aim to jointly extract aspects and aspect-specific sentiment knowledge from online reviews, where the sentiment knowledge refers to the aspect-specific opinion words along with their aspect-aware sentiment polarities. To this end, we propose a Joint Aspect/Sentiment model (JAS). JAS detects aspect-specific opinion words by integrating opinion word lexicon knowledge to explicitly separate opinion words from factual words. More importantly, JAS exploits sentiment prior and aspect-contextual sentence-level co-occurrences of opinion words in reviews to further identify aspect-aware sentiment polarities for the opinion words. We apply the learned aspect-specific sentiment knowledge to practical aspect-level sentiment analysis tasks. Experimental results show the effectiveness of JAS in learning aspect-specific sentiment knowledge and the practical value of this knowledge when applied to aspect-level sentiment classification.

In recent years, mining opinions from customer reviews has been widely explored. Aspect-level sentiment analysis is a fine-grained subtask, which aims to detect the sentiment polarity towards a particular target in a sentence. While most previous works focus on sentiment polarity classification, opinion words towards the target are also very important for that they provide details about target and contribute to judging polarity. To this end, we propose a hierarchical network for jointly modeling aspect-level sentiment classification and word-level opinion words extraction. Our joint model acquires superior performance in opinion words extraction and achieves comparable results in sentiment polarity classification on two datasets from SemEval 2014.

Aspect level sentiment classification aims to recognize the sentiment polarity of each aspect term in a sentence. However, most of the existing methods usually applied the attention mechanism over position-weighted memory and did not consider inter-aspect information. To address these issues, we propose a novel framework for aspect level sentiment classification, Deep Selective Memory Network (DSMN), which selects the context memory dynamically for better guiding the multi-hop attention mechanism and integrates inter-aspect information with deep memory network. By designing a selective attention mechanism based on the distance information between an aspect and its context, DSMN focuses on different parts of the context memory in different memory network layers to capture abundant aspect-aware context information. Besides, to make full use of the inter-aspect information, we also design effective inter-aspect modeling modules to generate both semantic and relation information of the nearby aspects for the desired aspect. We evaluate the advantages of our framework on three benchmark datasets, and experiment results show that our framework achieves state-of-the-art performance.

For aspect level sentiment classification, the attention mechanisms of memory network can capture the importance of each context word when inferring the sentiment polarity of an aspect. However, the existing attention mechanisms of memory network are either content-based or location-based while the part of speech (POS) of word is not used. Accordingly, this paper proposes a new attention mechanism called AM-PPOSC which uses the location, POS and content information. In addition, the common-used algorithms for learning word embedding typically only model the syntactic context of words but ignore the sentiment polarities of words. So, this paper proposes a method that based on TD-LSTM and SentiWordNet 3.0 to capture the sentiment polarities of words which represented as word sentiment vectors. Then, AM-PPOSC and word sentiment vectors are applied to memory network. Experiments on two datasets demonstrate that not only AM-PPOSC outperforms the original attention mechanisms of memory network when inferring the sentiment polarity of an aspect, but also word sentiment vectors can improve the performance of memory network. Furthermore, compared with feature-based SVM, LSTM and TD-LSTM, memory network with AM-PPOSC and word sentiment vectors achieves higher accuracies on two datasets.

Aspect-level sentiment classification aims to determine the sentiment polarity of the sentence towards the aspect. The key element of this task is to characterize the relationship between the aspect and the contexts. Some recent attention-based neural network methods regard the aspect as the attention calculation goal, so they can learn the association between aspect and contexts directly. However, the above attention model simply uses the word embedding to represent the aspect, it fails to make a further improvement on the performance of aspect sentiment classification. To solve this problem, this paper proposes a dependency subtree attention network (DSAN) model. The DSAN model firstly extracts the dependency subtree that contains the descriptive information of the aspect based on the dependency tree of the sentence, and then utilizes a bidirectional GRU network to generate an accurate aspect representation, and uses the dot-product attention function for the dependency subtree aspect representation, which finally yields the appropriate attention weights. The experimental results on SemEval 2014 Datasets demonstrate the effectiveness of the DSAN model.

Aspect-level sentiment classification aims to determine sentiment polarities of various aspects in reviews, where each review typically contains multiple aspects, that may correspond to different polarities. Aspect-level sentiment classification, unlike document-level sentiment classification, requires different context representations for different aspects. Existing methods normally use Long Short-Term Memory (LSTM) network to model aspects and contexts separately, and they combine attention mechanisms to extract features of a specific aspect in its context. Attention mechanisms are not used for sequence modeling, so aspects are not considered when generating context sequence representations. This study proposes a novel aspect-context interactive representation structure that only relies on an attention mechanism for generating sequence-to-sequence representations in both context and aspect. It is capable of extracting features related to the specific aspect during the process of its context sequence modeling, and generating a high quality aspect representation simultaneously. We have conducted comprehensive experiments to compare with thirteen existing methods. Our experimental results show that the proposed model is able to achieve significantly better performance on Restaurant dataset, as well as very competitive results on Laptop and Twitter datasets.

Aspect-level sentiment classification aims to predict sentiment polarities for different aspect terms within the same sentence or document. However, existing methods rely heavily on modeling the semantic relevance of an aspect term and its context words, and ignore the importance of syntax analysis to a certain extent. Consequently, this may cause the model to pay attention to the context word which is used to describe other aspect terms. In this paper, we propose a model which analyze sentences both syntactically and semantically. At the same time, we propose a simple and effective fusion mechanism to make the integration of aspect information and context information more adequately. We conduct extensive experiments on the SemEval 2014 benchmark datasets, and the results show that our model achieves a new state- of-the-art performance.

Background of study: The widespread adoption of Large Language Models (LLMs) raises concerns about biases that affect fairness and credibility. As LLMs affect areas such as recruitment and customer service, systematic quantitative analysis is essential to identify and mitigate these biases.Aims and scope of paper: This research investigates demographic bias in LLM quantitatively by analyzing sentiment polarity scores across different demographic categories. The goal is to provide a statistically confirmed analysis of sentiment bias and propose mitigation methods, focusing on GPT-4, LLaMA-2, Claude, and BLOOM.Methods: Quantitative analysis was performed on GPT-4, LLaMA-2, Claude, and BLOOM using sentiment and demographic data. Sentiment polarity assessments for gender and racial/ethnic groups were obtained with VADER and TextBlob. Demographic Disparity Score, ANOVA, and Cohen's Kappa assessed the significance and appropriateness of bias. Inter-rater reliability between automated tools and human annotators was also evaluated.Result: Sentiment bias was found in all models, varying by gender and race, particularly in GPT-4 and Claude. Sentiment scores were consistently higher for queries pertaining to females than those pertaining to males across all models, with GPT-4 and Claude showing the largest differences. Claude also showed racial sentiment alignment, favoring queries relating to white people over black people. ANOVA confirmed statistically significant sentiment variation by demographics across all models. High inter-rater reliability validated the sentiment analysis.Conclusion: This study shows demographic bias in GPT-4, LLaMA-2, Claude, and BLOOM, with different sentiment trends across demographic classifications. The models showed more positive sentiment for female questions and a trend towards certain racial groups. These findings indicate an embedded bias in the training data, which raises ethical concerns. Identifying and addressing these biases is critical to ensuring fairness and credibility in real-world LLM applications.

Pre-training and fine-tuning have become the de facto paradigm in many natural language processing (NLP) tasks. However, compared to other NLP tasks, neural machine translation (NMT) aims to generate target language sentences through the contextual representation from the source language counterparts. This characteristic means the optimization objective of NMT is far from that of the universal pre-trained models (PTMs), leading to the standard procedure of pre-training and fine-tuning does not work well in NMT. In this paper, we propose a novel framework to deep fuse the pre-trained representation into NMT, fully exploring the potential of PTMs in NMT. Specifically, we directly replace the randomly initialized Transformer encoder with a pre-trained encoder and propose a layer-wise coordination structure to coordinate PTM and NMT decoder learning. Then, we introduce a partitioned multi-task learning method to fine-tune the pre-trained parameter, reducing the gap between PTM and NMT by progressively learning the task-specific representation. Experimental results show that our approach achieves considerable improvements on WMT14 En2De, WMT14 En2Fr, and WMT16 Ro2En translation benchmarks and outperforms previous work in both autoregressive and non-autoregressive NMT models.

Aspect-Level Sentiment Classification aims to identify the sentiment polarity of each aspect in a sentence. Syntax-based graph neural networks have been used to model dependencies between opinion words and aspects with good results. However, the analysis of these works is highly dependent on the quality of the dependency graph and may achieve suboptimal results for comments with ambiguous syntax. We explore a novel Multi-View Gated Graph Convolutional Network (MGGCN) to address the above problems. We utilize a Gated Graph Convolutional Network (GateGCN) for a more reasonable interaction of syntactic dependencies and semantic information, where we refine our syntactic dependency graph by adding sentiment knowledge and aspect-aware information to the dependency tree. We use the Inter-aspect Graph Convolutional Network (InterGCN) to capture information about the sentiment dependencies between multiple aspects that appear in a sentence. Finally, by adaptively learning multi-view sentiment information through Simple Residual Multilayer Perceptron(SResMLP). Experimental results on four public datasets show that our proposed model outperforms state-of-the-art models.KeywordsAspect-level sentiment classificationMulti-view learningGated graph convolution network

Investors in Indonesia are increasing from year to year, especially mutual fund investors managed by investment managers. News is one of the factors considered by investment managers in making stock investment decisions. Very diverse news sources and different writing styles can make it difficult to retrieve information on each issuer in the news. In this research, the aspect-based sentiment analysis (ABSA) method is implemented to extract news specifically on each aspect (issuer) in the news and evaluate the issuer. The model used is a pre-trained Indonesian Bidirectional Encoder Representations from Transformers (BERT) model, IndoNLU, because the object of research is Indonesian-language stock news. The results show that with a combination of hyperparameters consisting of batch size 8, learning rate 0.00002, and epoch 8 for the IndoNLU model can produce an average evaluation metric value consisting of precision, recall, f1-score, and accuracy of 90%.