Fine-Tuning Word Embeddings for Aspect-Based Sentiment Analysis

Compared with traditional methods, word em-bedding is an efficient language representation that can learn syntax and semantics by using neural networks. As the result, more and more promising experiments in natural language processing (NLP) get the state-of-the-art results by introducing word embedding. In principle, embedding representation learning embeds words to a low-dimensional vector space, there-fore vectors support initialization of NLP tasks such as text classification, sentiment analysis, language understanding, etc. However, polysemy is very common in many languages, which causes word ambiguation, further influences the accuracy of the system. Additionally, language models based on distributed hypotheses mostly focused on word properties rather than morphology were our primary focus. This leads to unreasonable performance in different evaluations. At the same time, word embedding learning and measuring are two vital components of word representation. In this paper, we overviewed many language models including single sense and multiple sense word embedding, and many evaluated approaches including intrinsic and extrinsic evaluation. We found that there are obvious gaps between vectors and manual annotations in word similarity evaluation, and language models that achieved good performance in intrinsic evaluations could not produce similar results in extrinsic evaluations. To the best of our knowledge, there is no universal language model and embedding learning method for most NLP task, and each evaluations also hidden natural defects compared to human knowledge. More evaluated datasets are also investigated such as datasets used in intrinsic and extrinsic evaluations. We believe that an improved evaluation dataset and a more rational evaluation method would benefit from this overview.

Dengan semakin berkembangnya teknologi informasi, maka muncul istilah e-commerce dalam dunia bisnis. Pada e-commerce ada fitur review, pelanggan dapat memberikan review berupa teks, gambar, dan bintang. Review tersebut merupakan opini dari pelanggan terkait barang yang dibeli. Tetapi pada kebanyakan e-commerce tidak ada fitur kategori terkait review hal ini membuat calon pembeli kesusahan dalam menganalisa secara manual. Aspect-based sentiment analysis (ABSA) merupakan solusi dari permasalahan tersebut. ABSA memiliki tiga tugas salah satunya Aspect Category Detection yang memiliki fungsi untuk menggabungkan review pelanggan menjadi beberapa aspek dimana aspek-aspek tersebut sudah didefinisikan terlebih dahulu. Cukup banyak penelitian terkait Aspect Category Detection dengan mengunakan machine learning. Dari beberapa metode yang diuji, Convolutional Neural Network (CNN) merupakan metode terbaik. Selain itu penggunaan BERT sebagai word embedding menghasilkan output yang bagus baik dari pada word embedding konvensional. Penelitian ini menggunakan dataset dari e-commerce Bukalapak dengan 3114 review dan 6 aspek (Akurasi, Pengiriman, Kualitas, Harga, Pengemasan, dan Pelayanan). Berdasarkan ujicoba dengan menggunakan IndoBERT sebagai word embedding dan CNN untuk deteksi aspek, maka didapatkan akurasi sebesar 94,86%. Dengan demikian model tersebut dapat digunakan untuk deteksi aspek. Selain itu, metode CNN mendapatkan hasil yang lebih baik dari pada metode LSTM.

Aspect-based sentiment analysis is a vital issue in fine-grained sentiment evaluation, which intends to provide an automatic prediction of the sentiment polarity, given a particular aspect in its context. This paper presents an aspect-based sentiment analysis to find hate sentiment inside twitter data. Word embeddings have had prevalent utilisation in Natural Language Processing (NLP) applications because their vector representations have the ability to capture useful linguistic relationships and semantic properties between words with the help of deep neural networks. Word embeddings have often been used in machine learning models as feature input, which allows for the contextualisation of raw text data in machine learning techniques. The model has the ability to represent the relationship between the word embedding features and the aspects as feature representation within the suggested model. To assess the efficacy of the proposed method, extensive experiments were performed on the dataset of the researcher, as well as on widely utilised datasets. It was demonstrated by the experimental results that the proposed method was able to obtain impressive results among the three datasets.

Word embeddings have become the predominant representation scheme on a token-level for various clinical natural language processing (NLP) tasks. More recently, character-level neural language models, exploiting recurrent neural networks, have again received attention, because they achieved similar performance against various NLP benchmarks. We investigated to what extent character-based language models can be applied to the clinical domain and whether they are able to capture reasonable lexical semantics using this maximally fine-grained representation scheme. We trained a long short-term memory network on an excerpt from a table of de-identified 50-character long problem list entries in German, each of which assigned to an ICD-10 code. We modelled the task as a time series of one-hot encoded single character inputs. After the training phase we accessed the top 10 most similar character-induced word embeddings related to a clinical concept via a nearest neighbour search and evaluated the expected interconnected semantics. Results showed that traceable semantics were captured on a syntactic level above single characters, addressing the idiosyncratic nature of clinical language. The results support recent work on general language modelling that raised the question whether token-based representation schemes are still necessary for specific NLP tasks.

Aspect-based opinion mining is one among the thought-provoking research field which focuses on the extraction of vivacious aspects from opinionated texts and polarity value associated with these. The principal aim here is to identify user sentiments about specific features of a product or service rather than overall polarity. This fine-grained polarity identification about myriad aspects of an entity is highly beneficial for individuals or business organizations. Extricating these implicit or explicit aspects can be very challenging and this paper elaborates copious aspect extraction techniques, which is decisive for aspect-based sentiment analysis. This paper presents a novel idea of combining several approaches like Part of Speech tagging, dependency parsing, word embedding, and deep learning to enrich the aspect-based sentiment analysis specially designed for Twitter data. The results show that combining deep learning with traditional techniques can produce excellent results than lexicon-based methods.

BackgroundAutomated disease code classification using free-text medical information is important for public health surveillance. However, traditional natural language processing (NLP) pipelines are limited, so we propose a method combining word embedding with a convolutional neural network (CNN).ObjectiveOur objective was to compare the performance of traditional pipelines (NLP plus supervised machine learning models) with that of word embedding combined with a CNN in conducting a classification task identifying International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis codes in discharge notes.MethodsWe used 2 classification methods: (1) extracting from discharge notes some features (terms, n-gram phrases, and SNOMED CT categories) that we used to train a set of supervised machine learning models (support vector machine, random forests, and gradient boosting machine), and (2) building a feature matrix, by a pretrained word embedding model, that we used to train a CNN. We used these methods to identify the chapter-level ICD-10-CM diagnosis codes in a set of discharge notes. We conducted the evaluation using 103,390 discharge notes covering patients hospitalized from June 1, 2015 to January 31, 2017 in the Tri-Service General Hospital in Taipei, Taiwan. We used the receiver operating characteristic curve as an evaluation measure, and calculated the area under the curve (AUC) and F-measure as the global measure of effectiveness.ResultsIn 5-fold cross-validation tests, our method had a higher testing accuracy (mean AUC 0.9696; mean F-measure 0.9086) than traditional NLP-based approaches (mean AUC range 0.8183-0.9571; mean F-measure range 0.5050-0.8739). A real-world simulation that split the training sample and the testing sample by date verified this result (mean AUC 0.9645; mean F-measure 0.9003 using the proposed method). Further analysis showed that the convolutional layers of the CNN effectively identified a large number of keywords and automatically extracted enough concepts to predict the diagnosis codes.ConclusionsWord embedding combined with a CNN showed outstanding performance compared with traditional methods, needing very little data preprocessing. This shows that future studies will not be limited by incomplete dictionaries. A large amount of unstructured information from free-text medical writing will be extracted by automated approaches in the future, and we believe that the health care field is about to enter the age of big data.

The need to capture intra-word information in natural language processing (NLP) tasks has inspired research in learning various word representations at word, character, or morpheme levels, but little attention has been given to syllables from a syllabic alphabet. Motivated by the success of compositional models in morphological languages, we present a Convolutional-long short term memory (Conv-LSTM) model for constructing Swahili word representation vectors from syllables. The unified architecture addresses the word agglutination and polysemous nature of Swahili by extracting high-level syllable features using a convolutional neural network (CNN) and then composes quality word embeddings with a long short term memory (LSTM). The word embeddings are then validated using a syllable-aware language model ( 31.267 ) and a part-of-speech (POS) tagging task ( 98.78 ), both yielding very competitive results to the state-of-art models in their respective domains. We further validate the language model using Xhosa and Shona, which are syllabic-based languages. The novelty of the study is in its capability to construct quality word embeddings from syllables using a hybrid model that does not use max-over-pool common in CNN and then the exploitation of these embeddings in POS tagging. Therefore, the study plays a crucial role in the processing of agglutinative and syllabic-based languages by contributing quality word embeddings from syllable embeddings, a robust Conv–LSTM model that learns syllables for not only language modeling and POS tagging, but also for other downstream NLP tasks.

In the ever-expanding digital landscape, the abundance of user-generated content on consumer platforms such as Booking and TripAdvisor offers a rich source of information for both travellers and hoteliers. Sentiment analysis, a fundamental research task of natural language processing (NLP) is used for mining sentiments and opinions within this vast reservoir of text reviews. A more specific type of sentiment analysis, i.e., aspect-based sentiment analysis (ABSA), is used when processing customer reviews is required. In ABSA, we aim to capture aspect-level sentiments and intricate relationships between various aspects within reviews. This article proposes a novel approach to ABSA by introducing a novel technique of word sense disambiguation (WSD) and integrating it with the Transformer architecture bidirectional encoder representations from Transformers (BERT) and graph convolutional networks (GCNs). The proposed approach resolves the intriguing ambiguities of the words and represents the review data as a complex graph structure, facilitating the modeling of intricate relationships between different aspects. The combination of bidirectional long short-term memory (BiLSTM) and GCN proves effective in capturing inter-dependencies among various aspects, providing a nuanced understanding of customer sentiments. The experiments are conducted on the RABSA dataset (an enhanced and richer hotel review data collection), and results demonstrate that our approach outperforms previous baselines, showcasing the effectiveness of integrating WSD in ABSA. Furthermore, an ablation study confirms the significant contribution of the WSD module to the overall performance. Moreover, we explore different similarity measures and find that cosine similarity yields the best results when identifying the real sense of a word in a given sentence using WordNet. The findings of our work and future work related to our work create lots of interest for people in the tourism and hospitality industry. This research gives another boost to the concept of the potential of NLP techniques in sentiment analysis. It emphasizes that if we combine the potential of NLP techniques along with state-of-the-art machine learning frameworks, we can shape the future of this field.

Learning the inherent meaning of a word in Natural Language Processing (NLP) has motivated researchers to represent a word at various levels of abstraction, namely character-level, morpheme-level, and subword-level vector representations. Syllable-Aware Word Embedding (SAWE) can effectively handle agglutinative and fusion-based NLP tasks. However, research attempts on assessing the SAWE on such extrinsic NLP tasks has been scanty, especially for low-resource languages in the context of code-mixing with English. A model to learn SAWE to extract semantics at fine-grained subunits of a word is proposed in this article, and the representative ability of the embeddings is assessed through sentiment analysis of code-mixed Telugu-English review corpora. Multilingual societies and advancements in communication technologies have accounted for the prolific usage of mixed data, which renders the State-of-the-Art (SOTA) sentiment analysis models developed based on monolingual data ineffective. Social media users in the Indian subcontinent exhibit a tendency to mix English and their respective native language (using the phonetic form of English) in expressing their opinions or sentiments. A code-mixing scenario provides flexibility to borrow words from a foreign language, usage of shorthand notations, elongation of vowels, and usage of words without following syntactic/grammatical rules, which renders the sentiment analysis of code-mixed data challenging to perform. Deep neural architectures like Long Short-Term Memory and Gated Recurrent Unit networks have been shown to be effective in solving several NLP tasks, such as sequence labeling, named entity recognition, and machine translation. In this article, a framework to perform sentiment analysis on a code-mixed Telugu-English review corpus is implemented. Both word embedding and SAWE are input to a unified deep neural network that contains a two-level Bidirectional Long Short-Term Memory/Gated Recurrent Unit network with Softmax as the output layer. The proposed model leverages the advantages of both word embedding and SAWE, which enable the proposed model to outperform existing SOTA code-mixed sentiment analysis models on a Telugu-English code-mixed dataset of the International Institute of Information Technology–Hyderabad and a dataset curated by the authors. The improvement realized by the proposed model on these datasets is [3% increase in F1-score and 2% increase in accuracy] and [7% increase in F1-score and 5% in accuracy], respectively, in comparison with the best-performing SOTA model.

Conventional text embeddings, typically learned in the Euclidean space, may struggle to effectively capture word semantics based solely on the directional similarity between word vectors. In contrast, spherical text embeddings have demonstrated remarkable efficacy in various natural language processing (NLP) tasks recently. However, training spherical text generative models (STGMs) requires large representative datasets, which could potentially contain sensitive private information. To mitigate this concern, we propose a novel approach: the differential private spherical text generative model (DP-STGM), which facilitates learning text embeddings within the spherical space while ensuring privacy via efficient Riemannian optimization and the framework of differential privacy. To evaluate the efficacy of our privacy-preserving algorithm, we initially train an adversary using an external dataset without the application of differential privacy. Subsequently, we introduce two metrics to measure the model’s ability to protect privacy: (1) cosine similarity between recovered words from the adversary that generates the spherical text embeddings and those generated by DP-STGM, and (2) Top-n rank correlation. Our experimental findings demonstrate that DP-STGM outperforms baseline models, showcasing its superior performance. By leveraging the power of differential privacy in the Riemannian optimization process, our model achieves better preservation of sensitive information while simultaneously capturing the semantic nuances inherent in word embeddings. As a result, DP-STGM represents a robust and efficient solution for NLP tasks that require privacy protection without compromising on the quality of learned text embeddings. By offering a privacy-preserving alternative, DP-STGM broadens the range of applications for which STGMs can be safely employed, ensuring data privacy while harnessing the rich information contained in spherical text embeddings. Our work opens up new avenues for future research in privacy-aware NLP and advances the state-of-the-art in both privacy protection and semantic learning in this domain.

Aspect-Based Sentiment Analysis (ABSA) is one of the highly challenging tasks in natural language processing. It extracts fine-grained sentiment information in user-generated reviews, as it aims at predicting the polarities towards predefined aspect categories or relevant entities in free text. Previous deep learning approaches usually rely on large-scale pre-trained language models and the attention mechanism, which applies the complete computed attention weights and does not place any restriction on the attention assignment. We argue that the original attention mechanism is not the ideal configuration for ABSA, as for most of the time only a small portion of terms are strongly related to the sentiment polarity of an aspect or entity. In this paper, we propose a masked attention mechanism customized for ABSA, with two different approaches to generate the mask. The first method sets an attention weight threshold that is determined by the maximum of all weights, and keeps only attention scores above the threshold. The second selects the top words with the highest weights. Both remove the lower score parts that are assumed to be less relevant to the aspect of focus. By ignoring part of input that is claimed irrelevant, a large proportion of input noise is removed, keeping the downstream model more focused and reducing calculation cost. Experiments on the Multi-Aspect Multi-Sentiment (MAMS) and SemEval-2014 datasets show significant improvements over state-of-the-art pre-trained language models with full attention, which displays the value of the masked attention mechanism. Recent work shows that simple self-attention in Transformer quickly degenerates to a rank-1 matrix, and masked attention may be another cure for that trend.

Aspect-Based Sentiment Analysis (ABSA) is a Natural Language Processing (NLP) task that extracts referred aspects from text and assigns polarities to opinions about those aspects. Most research on ABSA focuses on English. Only a few ABSA works deal with the Portuguese language. In this work, we used BERTimbau to create a Question-Answer approach to ABSA in Portuguese. First, we post-trained this model with text from the same domain as our target corpus. Then, we constructed an auxiliary sentence from the aspect and converted ABSA to a sentence-pair classification task, such as question answering (QA) and natural language inference (NLI). Our experiments show that ABSA based on BERT for Portuguese achieved Balanced Accuracy (BACC) of 77% on a corpus of reviews about the accommodation sector using a post-trained model with QA approach.

Cross-lingual word embeddings are representations for vocabularies of two or more languages in one common continuous vector space and are widely used in various natural language processing tasks. A state-of-the-art way to generate cross-lingual word embeddings is to learn a linear mapping, with an assumption that the vector representations of similar words in different languages are related by a linear relationship. However, this assumption does not always hold true, especially for substantially different languages. We therefore propose to use kernel canonical correlation analysis to capture a non-linear relationship between word embeddings of two languages. By extensively evaluating the learned word embeddings on three tasks (word similarity, cross-lingual dictionary induction, and cross-lingual document classification) across five language pairs, we demonstrate that our proposed approach achieves essentially better performances than previous linear methods on all of the three tasks, especially for language pairs with substantial typological difference.

Convolutional neural networks (CNN) are multi-layer neural networks that are used to learn hierarchical data properties. In recent times, CNN has achieved remarkable advances in the architecture and computation of Natural Language Processing (NLP). The Word2vec technique is considered to introduce Word embeddings, which are used to improve the performance of a variety of Natural Language Processing (NLP) applications. It is a well-known technique for learning word embeddings, which are dense representations of words in a lower-dimensional vector space. Two prominent approaches are used for learning word embeddings, which are dense representations of words in a lower-dimensional vector space, are Continuous Bag-of-Words (CBOW) and Skip-Gram.

SummaryTaxonomic relations play an important role in various Natural Language Processing (NLP) tasks (eg, information extraction, question answering and knowledge inference). Existing approaches on embedding‐based taxonomic relation learning mainly rely on the word embeddings trained using co‐occurrence‐based similarity learning. However, the performance of these approaches is not quite satisfactory due to the lack of sufficient taxonomic semantic knowledge within word embeddings. To solve this problem, we propose an improved embedding‐based approach to learn taxonomic relations via incorporating relation descriptions into word embeddings. First, to capture additional taxonomic semantic knowledge, we train special word embeddings using not only co‐occurrence information of words but also relation descriptions (eg, taxonomic seed relations and their contextual triples). Then, using the trained word embeddings as features, we employ two learning models to identify and predict taxonomic relations, namely, offset‐based classification model and offset‐based similarity model. Experimental results on four real‐world domain datasets demonstrate that our proposed approach can capture additional taxonomic semantic knowledge and reduce dependence on the training dataset, outperforming the state‐of‐the‐art compared approaches on the taxonomic relation learning task.