Benchmarking BoW, TF-IDF and Bert-Based Sentiment Analysis in Low-Resource Minang Language

The rapid growth of digital communication in Indonesia has led to a distinct informal linguistic style that poses significant challenges for Natural Language Processing (NLP) systems trained on formal text. This discrepancy often degrades the performance of downstream tasks like machine translation and sentiment analysis. This study aims to provide the first systematic comparison of IndoNanoT5 (encoder-decoder) and IndoGPT (decoder-only) architectures for Indonesian informal-to-formal text style transfer. We conduct comprehensive experiments using the STIF-INDONESIA dataset through rigorous hyperparameter optimization, multiple evaluation metrics, and statistical significance testing. The results demonstrate clear superiority of the encoder-decoder architecture, with IndoNanoT5-base achieving a peak BLEU score of 55.99, significantly outperforming IndoGPT's highest score of 51.13 by 4.86 points—a statistically significant improvement (p<0.001) with large effect size (Cohen's d = 0.847). This establishes new performance benchmarks with 28.49 BLEU points improvement over previous methods, representing a 103.6% relative gain. Architectural analysis reveals that bidirectional context processing, explicit input-output separation, and cross-attention mechanisms provide critical advantages for handling Indonesian morphological complexity. Computational efficiency analysis shows important trade-offs between inference speed and output quality. This research advances Indonesian text normalization capabilities and provides empirical evidence for architectural selection in sequence-to-sequence tasks for morphologically rich, low-resource languages.

With the rapid expansion of the Internet and social media, there has been an explosion of text data, containing valuable emotional insights. Sentiment analysis, aimed at discerning and processing people's emotional inclinations, opinions, and perspectives, has emerged as a pivotal area within natural language processing (NLP). This research explores the application of Deep Learning-based BERT models, particularly DistilBERT, in sentiment analysis. It highlights the importance of sentiment analysis in understanding emotional insights from vast text data and introduces BERT models as revolutionary methodologies for enhancing accuracy and efficiency. The study conducts experiments using the SST2 dataset, showcasing the effectiveness of DistilBERT in sentiment classification tasks. Overall, it underscores the transformative potential of BERT models in revolutionizing sentiment analysis methodologies and driving advancements in natural language processing research.

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

Natural Language Processing (NLP) is now widely integrated into web and mobile applications, enabling natural interactions between humans and computers. Although there is a large body of NLP studies published in Information Systems (IS), a comprehensive review of how NLP research is conceptualised and realised in the context of IS has not been conducted. To assess the current state of NLP research in IS, we use a variety of techniques to analyse a literature corpus comprising 356 NLP research articles published in IS journals between 2004 and 2018. Our analysis indicates the need to move from semantics to pragmatics. More importantly, our findings unpack the challenges and assumptions underlying current research trends in NLP. We argue that overcoming these challenges will require a renewed disciplinary IS focus. By proposing a roadmap of NLP research in IS, we draw attention to three NLP research perspectives and present future directions that IS researchers are uniquely positioned to address.

The availability of huge annotated datasets and deep learning have led to astounding Natural Language Processing (NLP) progress. However, there is a plethora of low-resource languages, lack the labelled corpora to train the models in the supervision. This kind of digital divide does not allow fair NLP development across language groups. Self-supervised learning (SSL) is a new paradigm in recent years that leverages large amounts of untagged text to learn powerful representations with little or no manual annotation. This paper is a review of self-supervised deep learning models in low-resource NLP tasks. It begins with the definition of the principles of the SSL and the difference between this approach and supervised and unsupervised approaches. We describe why methods such as masked language modeling, contrastive learning, and autoregressive modeling underlie modern pre-trained transformers such as BERT, GPT, and mBERT. A particular interest is paid to multilingual and cross lingual SSL schemes which allow knowledge transfer between high-resource and low resource languages. Low-resource tasks, including machine translation, sentiment analysis, speech-to-text, and information retrieval are reviewed. As per benchmark studies, even in small labelled samples, the SSL models can achieve big gains in accuracy and generalization. Computation cost, representational bias and morphologically rich and under-documented language evaluation, are however, problematic. The researchers observe that the study suffers certain limitations, such as over-reliance on high-resource pretraining information, inequity between linguistic groups, and the difficulty of deploying large-scale SSL models in resource-constrained circumstances. The future directions include lightweight multilingual models, federation of learning in NLP and symbolic linguistic knowledge and combination with SSL. Self-supervised deep learning bridges an essential gap between the high-resource and low-resource languages, and would be a highly valuable step toward inclusive, global NLP innovation.

Urdu is still considered a low-resource language despite being ranked as the world’s 10th most spoken language with nearly 230 million speakers. The scarcity of benchmark datasets in low-resource languages has led researchers to utilize more ingenious techniques to curb the issue. One such option widely adopted is to use language translation services to replicate existing datasets from resource-rich languages such as English to low-resource languages, such as Urdu. For most natural language processing tasks, including polarity assessment, words translated via Google translator from one language to another often change the meaning. It results in a polarity shift causing the system’s performance degradation, particularly for sentiment classification and emotion detection tasks. This study evaluates the effect of translation on the sentiment classification task from a resource-rich language to a low-resource language. It identifies and enlists words causing polarity shift into five distinct categories. It further finds the correlation between the language with similar roots. Our study shows 2-3 percentage points performance degradation due to polarity shift as a result of translation from resource-rich languages to low-resource languages.

Depression detection in low-resource languages, such as Bangla, is an emerging challenge in mental health research due to the limited availability of labeled datasets and NLP resources. Bangla, the seventh most spoken language globally, is used by over 230 million people, yet it remains underrepresented in natural language processing (NLP) research compared to high-resource languages like English. In this study, we present a Bangla depression detection dataset collected from social media comprising 18,612 text samples, with 8556 labeled as depressive and the remaining as non-depressive. To effectively classify depressive content, we explore multiple deep learning architectures, including Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), Gated Recurrent Unit (GRU), and Bidirectional GRU (BiGRU). Additionally, we evaluate state-of-the-art transformer-based models such as BanglaBERT, BanglaBERT-Base, ELECTRA, RoBERTa, and BERT. Machine learning models are also employed as baselines for comparison. Our experimental results show that transformer-based models outperform both deep learning and machine learning approaches, with BanglaBERT achieving the highest accuracy of 89.23%, followed closely by BiLSTM with 88%. These findings highlight the effectiveness of transformer-based architectures in depression detection for Bangla text, contributing to improved mental health analysis in low-resource language settings. This study emphasizes the need for further NLP advancements in Bangla to develop AI-driven solutions for early depression detection and intervention.

This investigation investigates the influence of a variety of data augmentation techniques on sentiment analysis in low-resource languages, with a particular emphasis on Bulgarian, Croatian, Slovak, and Slovene. The following primary research topic is addressed: is it possible to improve sentiment analysis efficacy in low-resource languages through data augmentation? Our sub-questions look at how different augmentation methods affect performance, how effective WordNet-based augmentation is compared to other methods, and whether lemma-based augmentation techniques can be used, especially for Croatian sentiment tasks. The sentiment-labelled evaluations in the selected languages are included in our data sources, which were curated with additional annotations to standardise labels and mitigate ambiguities. Our findings show that techniques like replacing words with synonyms, masked language model (MLM)-based generation, and permuting and combining sentences can only make training datasets slightly bigger. However, they provide limited improvements in model accuracy for low-resource language sentiment classification. WordNet-based techniques, in particular, exhibit a marginally superior performance compared to other methods; however, they fail to substantially improve classification scores. From a practical perspective, this study emphasises that conventional augmentation techniques may require refinement to address the complex linguistic features that are inherent to low-resource languages, particularly in mixed-sentiment and context-rich instances. Theoretically, our results indicate that future research should concentrate on the development of augmentation strategies that introduce novel syntactic structures rather than solely relying on lexical variations, as current models may not effectively leverage synonymic or lemmatised data. These insights emphasise the nuanced requirements for meaningful data augmentation in low-resource linguistic settings and contribute to the advancement of sentiment analysis approaches.

Low-resource language (LRL) processing refers to the development of natural language processing (NLP) techniques and tools for languages with limited linguistic resources and data. These languages often lack well-annotated datasets and pre-training methods, making traditional approaches less effective. Sentiment analysis (SA), which involves identifying the emotional tone or sentiment expressed in text, poses unique challenges for LRLs due to the scarcity of labelled sentiment data and linguistic intricacies. NLP tasks like SA, powered by machine learning (ML) techniques, can generalize effectively when trained on suitable datasets. Recent advancements in computational power and parallelized graphical processing units have significantly increased the popularity of deep learning (DL) approaches built on artificial neural network (ANN) architectures. With this in mind, this manuscript describes the design of an LRL Processing technique that makes use of Improved Deep Learning with Hunter–Prey Optimization (LRLP-IDLHPO). The LRLP-IDLHPO technique enables the detection and classification of different kinds of sentiments present in LRL data. To accomplish this, the presented LRLP-IDLHPO technique initially pre-processes these data to improve their usability. Subsequently, the LRLP-IDLHPO approach applies the SentiBERT approach for word embedding purposes. For the sentiment classification process, the Element-Wise–Attention GRU network (EWAG-GRU) algorithm is used, which is an enhanced version of the recurrent neural network. The EWAG-GRU model is capable of processing temporal features and includes an attention strategy. Finally, the performance of the EWAG-GRU model can be boosted by adding the HPO algorithm for use in the hyperparameter tuning process. A widespread simulation analysis was performed to validate the superior results derived from using the LRLP-IDLHPO approach. The extensive results indicate the significant superiority of the performance of the LRLP-IDLHPO technique compared to the state-of-the-art approaches described in the literature.

Natural Language Processing (NLP) is a revolutionary discipline that resides at the crossroads of computer science and linguistics, with its primary emphasis on the interaction between computers and human language. With roots in artificial intelligence, NLP seeks to equip machines with the ability to comprehend, interpret, and respond to natural language, enabling more intuitive and meaningful communication between humans and computers. This multidisciplinary domain leverages advanced algorithms and models to tackle a range of linguistic challenges, from language translation and sentiment analysis to speech recognition. Recent breakthroughs, particularly in deep learning and neural networks, have propelled NLP to new heights, with applications spanning diverse sectors such as healthcare, finance, and education. As NLP continues to advance, its potential impact on enhancing human-machine interaction and information processing is increasingly evident, promising a future where technology seamlessly integrates with our natural modes of communication. The significance of Natural Language Processing (NLP) in research lies in its capacity to revolutionize human-computer interaction. NLP empowers machines to understand and generate human language, enabling advancements in sentiment analysis, language translation, and conversational AI. This transformative capability has profound implications across various industries, including healthcare, finance, and education. NLP research not only enhances the efficiency of information retrieval and processing but also holds the key to developing more intuitive and user-friendly technologies. As the frontier of NLP research expands, its potential to bridge the communication gap between humans and machines continues to shape the future of technology and information access. VIKOR functions as a multi-criteria decision-making technique that determines the best option by comparing it to the ideal option. The ranking procedure comprises determining distances concerning the optimal solution. VIKOR uses linear normalization to get the best possible outcomes. This technique was first presented by Opricovic in 1998 and was intended to optimize multi-attribute complex systems. Its main focus is on ranking lists that allow for flexibility in the strategy weight interval and incorporate compromise alternatives to obtain desired results. Alternative taken as Open AI GPT-4, Google BERT, Microsoft Azure Text Analytics, IBM Watson Natural Language Understanding, spaCy, NLTK (Natural Language Toolkit), Amazon Comprehend. Evaluation Parameter taken as Sentiment Analysis (F1 Score), NER Performance, Language Support, Processing Speed (Response Time) (ms), Translation Accuracy (BLEU Score), Data Privacy. The result it is seen that Amazon Comprehend is got the first rank where as is the IBM Watson Natural Language Understanding is having the lowest rank.

Automatic text summarization is an emerging field of research in Natural Language Processing. This work is a novel attempt to include a low-resource language to the domain of Automatic Text Summarization. We use supervised machine learning algorithms to perform single document extractive automatic text summarization on documents in a low-resource language, Konkani. In particular, we propose using language independent features to train supervised machine learning algorithms using a Konkani dataset, specifically devised for the experimentation using books on Konkani folktale literature. We approach the automatic text summarization task as a binary classification problem, and the algorithms, once trained, classify the sentences based on their relevance to generate a summary. Thereafter, the performance of popular linear and non-linear supervised machine learning algorithms is evaluated using K-fold cross-validation. The summary generated by the systems is compared with human-generated summaries to verify its effectiveness. The results show that the linear models exhibit better performance in comparison with the non-linear models; however, all the models could beat the baselines. The output produced by the proposed methodology generates promising summaries without the need for any language-specific domain knowledge.

With the advent of Neural Networks, Neural Machine Translation (NMT) has been a booming area of research in Natural Language Processing. Indic languages possess less significant parallel and monolingual corpus than English and other European languages. Even in Indic languages, many Indo-Aryan and Dravidian languages fall into low-resource languages, and techniques like Back Translation, Unsupervised Neural Machine Translation (UNMT), and Transfer Learning help achieve better translation accuracy. This paper uses the UNMT approach with a pre-trained Cross-Lingual Language Model (XLM) for the English to Kannada language pair. Our proposed methodology has achieved a 0.61 BLEU score in English to Kannada and a 0.32 BLEU score in Kannada to English translation.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Word sense disambiguation (WSD) is the process of determining the correct meaning of a word based on its context in a sentence, a task that remains one of the core challenges in natural language processing (NLP) despite the advancements made by LLMs. Although LLMs have improved WSD performance, challenges remain–especially in nuanced contexts, low-resource languages, and domain-specific terminology. These gaps can hinder the accuracy of LLMs in high-stakes applications and multilingual settings. In response to the need for WSD, our study introduces a novel approach to WSD data collection through gamified crowdsourcing, which, to our knowledge, has not been previously applied in this field. A messaging bot method has been used to engage a wide, diverse audience to generate high-quality WSD data. Using a multiplayer game format, native speakers provide examples for different senses of ambiguous words and rate others’ contributions. Together with the suggested enhancements, this platform attracted a diverse range of participants–spanning ages, backgrounds, and genders–20 times more than a similar crowdsourcing method applied to a different NLP task, and sustained their engagement over an extended period. Unlike conventional academic crowdsourcing pools, this approach focuses on drawing individuals from varied backgrounds beyond academia or AI-focused communities. It not only gathered data but also encouraged participants to evaluate each other’s contributions, creating a rich and reliable dataset. Our findings suggest that gamified crowdsourcing can be a powerful tool for creating WSD corpora. Our approach not only supports future WSD research but also contributes valuable training data for developing more precise and resilient language models.

Machine Translation (MT) systems are now being improved with the use of an ongoing methodology known as Neural Machine Translation (NMT). Natural language processing (NLP) researchers have shown that NMT systems are unable to deal with out-of-vocabulary (OOV) words and multi-word expressions (MWEs) in the text. OOV terms are those that are not currently included in the vocabulary that is used by the NMT system. MWEs are phrases that consist of a minimum of two terms but are treated as a single unit. MWEs have great importance in NLP, linguistic theory, and MT systems. In this article, OOV words and MWEs are handled for the Punjabi to English NMT system. A parallel corpus for Punjabi to English containing MWEs was developed and used to train the different models of NMT. Punjabi is a low-resource language as it lacks the availability of a large parallel corpus for building various NLP tools, and this is an attempt to improve the accuracy of Punjabi in the English NMT system by using named entities and MWEs in the corpus. The developed NMT models were assessed using human evaluation through adequacy, fluency and overall rating as well as automated assessment tools such as the bilingual evaluation study (BLEU) and translation error rate (TER) score. Results show that using word embedding (WE) and MWEs corpus increased the accuracy of translation for the Punjabi to English language pair. The best BLEU score obtained was 15.45 for the small test set, 43.32 for the medium test set, and 34.5 for the large test set, respectively. The best TER rate score obtained was 57.34% for the small test set, 37.29% for the medium test set, and 53.79% for the large test set, repectively.