Does learning from language family help? A case study on a low-resource question-answering task

The availability of huge annotated datasets and deep learning have led to astounding Natural Language Processing (NLP) progress. However, there is a plethora of low-resource languages, lack the labelled corpora to train the models in the supervision. This kind of digital divide does not allow fair NLP development across language groups. Self-supervised learning (SSL) is a new paradigm in recent years that leverages large amounts of untagged text to learn powerful representations with little or no manual annotation. This paper is a review of self-supervised deep learning models in low-resource NLP tasks. It begins with the definition of the principles of the SSL and the difference between this approach and supervised and unsupervised approaches. We describe why methods such as masked language modeling, contrastive learning, and autoregressive modeling underlie modern pre-trained transformers such as BERT, GPT, and mBERT. A particular interest is paid to multilingual and cross lingual SSL schemes which allow knowledge transfer between high-resource and low resource languages. Low-resource tasks, including machine translation, sentiment analysis, speech-to-text, and information retrieval are reviewed. As per benchmark studies, even in small labelled samples, the SSL models can achieve big gains in accuracy and generalization. Computation cost, representational bias and morphologically rich and under-documented language evaluation, are however, problematic. The researchers observe that the study suffers certain limitations, such as over-reliance on high-resource pretraining information, inequity between linguistic groups, and the difficulty of deploying large-scale SSL models in resource-constrained circumstances. The future directions include lightweight multilingual models, federation of learning in NLP and symbolic linguistic knowledge and combination with SSL. Self-supervised deep learning bridges an essential gap between the high-resource and low-resource languages, and would be a highly valuable step toward inclusive, global NLP innovation.

This paper investigates how multilingual language models benefit low-resource languages through our submission to the WMT 2025 Low-Resource Indic Language Translation shared task.We explore whether languages from related families can effectively support translation for low-resource languages that were absent or underrepresented during model training.Using a quantized multilingual pretrained foundation model, we examine zero-shot translation capabilities and cross-lingual transfer effects across three language families: Tibeto-Burman, Indo-Aryan, and Austroasiatic.Our findings demonstrate that multilingual models failed to leverage linguistic similarities, particularly evidenced within the Tibeto-Burman family.The study provides insights into the practical feasibility of zero-shot translation for low-resource language settings and the role of language family relationships in multilingual model performance.

Natural language processing (NLP) has significantly advanced our ability to model and interact with human language through technology. However, these advancements have disproportionately benefited high-resource languages with abundant data for training complex models. Low-resource languages, often spoken by smaller or marginalized communities, need help realizing the full potential of NLP applications. The primary challenges in developing NLP applications for low-resource languages stem from the need for large, well-annotated datasets, standardized tools, and linguistic resources. This scarcity of resources hinders the performance of data-driven approaches that have excelled in high-resource settings. Further, low-resource languages frequently exhibit complex grammatical structures, diverse vocabularies, and unique social contexts, which pose additional challenges for standard NLP techniques. Innovative strategies are emerging to address these challenges. Researchers are actively collecting and curating datasets, even utilizing community engagement platforms to expand data resources. Transfer learning, where models pre-trained on high-resource languages are adapted to low-resource settings, has shown significant promise. Multilingual models like Multilingual Bidirectional Encoder Representations from Transformers (mBERT) and Cross Lingual Models (XLM-R), trained on vast quantities of multilingual data, offer a powerful avenue for cross-lingual knowledge transfer. Additionally, researchers are exploring integrating multimodal approaches, combining textual data with images, audio, or video, to enhance NLP performance in low-resource language scenarios. This survey covers applications like part-of-speech tagging, morphological analysis, sentiment analysis, hate speech detection, dependency parsing, language identification, discourse annotation guidelines, question answering, machine translation, information retrieval, and predictive authoring for augmentative and alternative communication systems. The review also highlights machine learning approaches, deep learning approaches, Transformers, and cross-lingual transfer learning as practical techniques. Developing practical NLP applications for low-resource languages is crucial for preserving linguistic diversity, fostering inclusion within the digital world, and expanding our understanding of human language. While challenges remain, the strategies outlined in this survey demonstrate the ongoing progress and highlight the potential for NLP to empower communities that speak low-resource languages and contribute to a more equitable landscape within language technology.

The recent advances of Natural Language Processing (NLP) have significantly benefited people across the globe who speak and write some specific languages, commonly termed high-resource languages (HRLs). These are languages with abundant digital resources. One of the most significant areas where NLP has recently made its mark is environment preservation. However, a significant digital divide is observed for low-resource languages (LRLs), which are not rich in terms of digital resources. Even though many LRLs, like Assamese, are very important for the cultural and linguistic identity of many indigenous communities, they remain digitally underrepresented due to a lack of annotated corpora and computational tools. This gap is even more crucial in the environmental domain, where knowledge is very limited even for high-resource languages. Multilingual information is very important for NLP applications related to climate change, disaster management and other issues related to environmental science. This paper reports a study of the challenges and strategies to address the gap in NLP, both in the context of the global and Indian landscape. The paper also highlights the key problems while addressing issues related to environmental science due to the lack of digital resources. Some of the significant factors include the absence of large corpora, labelled datasets and socio-economic factors. The paper proposes to emphasize data collection and digitization, among many other measures, to address this gap. The integration of multilingual pretrained models and transfer learning approaches also provides a pathway for enhancing performance with limited resources. The paper also presents a detailed analysis of resources available for global and Indian languages in addition to proposing a set of strategic actions, including government policies, and others, for inclusive development in NLP and sustainable environment. Bridging the divide and enabling linguistic equity will ensure participation of all sections of the society in the advancement of inclusive technological growth and solutions to the emerging problems.

Urdu is still considered a low-resource language despite being ranked as the world’s 10th most spoken language with nearly 230 million speakers. The scarcity of benchmark datasets in low-resource languages has led researchers to utilize more ingenious techniques to curb the issue. One such option widely adopted is to use language translation services to replicate existing datasets from resource-rich languages such as English to low-resource languages, such as Urdu. For most natural language processing tasks, including polarity assessment, words translated via Google translator from one language to another often change the meaning. It results in a polarity shift causing the system’s performance degradation, particularly for sentiment classification and emotion detection tasks. This study evaluates the effect of translation on the sentiment classification task from a resource-rich language to a low-resource language. It identifies and enlists words causing polarity shift into five distinct categories. It further finds the correlation between the language with similar roots. Our study shows 2-3 percentage points performance degradation due to polarity shift as a result of translation from resource-rich languages to low-resource languages.

POS tagging is a basic Natural Language Processing (NLP) task that tags the words in an input text according to its grammatical values. Although POS Tagging is a fundamental application for very resourced languages, such as Limbu, is still unknown due to only few tagged datasets and linguistic resources. This research project uses deep learning techniques, transfer learning, and the BiLSTM-CRF model to develop an accurate POS-tagging system for the Limbu language. Using annotated and unannotated language data, we progress in achieving a small yet informative dataset of Limbu text. Skilled multilingual tutoring was modified to enhance success on low-resource language tests. The model as propose attains 90% accuracy, which is very much better than traditional rule-based and machine learning methods for Limbu POS tagging. The results indicate that deep learning methods can address linguistic issues facing low-resource languages even with limited data. In turn, this study provides a cornerstone for follow up NLP-based applications of Limbu and similar low-resource languages, demonstrating how deep learning can fill the gap where data is scarce.

Electronic health records (EHRs) contain patient-related information formed by structured and unstructured data, a valuable data source for Natural Language Processing (NLP) in the healthcare domain. The contextual word embeddings and Transformer-based models have proved their potential, reaching state-of-the-art for various NLP tasks. Although the performance for downstream NLP tasks with free-texts written in English has recently improved, less resource is available considering clinical texts and low-resource languages such as Portuguese. Our objective is to develop a Generative Pre-trained Transformer 2 (GPT-2) language model for Portuguese to support clinical and biomedical NLP tasks. We fine-tuned a generic Portuguese GPT-2 model to corpora of biomedical texts written in Portuguese, using transfer learning. We experimented on a public dataset, manually annotated for detecting patient fall, i.e., a classification task. Our in-domain GPT-2 model outperformed the generic Portuguese GPT-2 model by 3.43 in F1-score (weighted). Our preliminary results show that transfer learning with domain literature can benefit Portuguese biomedical NLP tasks, aligned with other languages' results.

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT^a, a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.

To obtain a reliable and accurate automatic speech recognition (ASR) machine learning model, it is necessary to have sufficient audio data transcribed, for training. Many languages in the world, especially the agglutinative languages of the Turkic family, suffer from a lack of this type of data. Many studies have been conducted in order to obtain better models for low-resource languages, using different approaches. The most popular approaches include multilingual training and transfer learning. In this study, we combined five agglutinative languages from the Turkic family—Kazakh, Bashkir, Kyrgyz, Sakha, and Tatar,—in order to provide multilingual training using connectionist temporal classification and an attention mechanism including a language model, because these languages have cognate words, sentence formation rules, and alphabet (Cyrillic). Data from the open-source database Common voice was used for the study, to make the experiments reproducible. The results of the experiments showed that multilingual training could improve ASR performances for all languages included in the experiment, except Bashkir language. A dramatic result was achieved for the Kyrgyz language: word error rate decreased to nearly one-fifth and character error rate decreased to one-fourth, which proves that this approach can be helpful for critically low-resource languages.

Sentiment analysis in regional languages remains a critical challenge in low-resource natural language processing (NLP). The Minang language, spoken by over eight million people in Indonesia, is underrepresented in NLP research. This study evaluates the performance of IndoBERT—a pre-trained transformer model for Bahasa Indonesia—on sentiment classification tasks in the Minang language. A dataset of 5,000 tweets containing parallel Indonesian and Minang texts is used for fine-tuning IndoBERT and benchmarking it against traditional methods such as Bag-of-Words and TF-IDF, as well as the multilingual BERT (mBERT) model. Experimental results show that IndoBERT achieves the highest overall accuracy (0.870) and F1-score (0.840), outperforming baseline models across multiple evaluation metrics. The findings demonstrate IndoBERT effectiveness for sentiment analysis in morphologically rich, low-resource languages and highlight the potential of transfer learning from a national to a regional language. This work contributes to advancing regional language NLP and establishes a benchmark for future research in Minang-language sentiment modelling.

Cross-lingual transfer, where a high-resource transfer language is used to improve the accuracy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low-resource languages. However, given a particular task language, it is not clear which language to transfer from, and the standard strategy is to select languages based on ad hoc criteria, usually the intuition of the experimenter. Since a large number of features contribute to the success of cross-lingual transfer (including phylogenetic similarity, typological properties, lexical overlap, or size of available data), even the most enlightened experimenter rarely considers all these factors for the particular task at hand. In this paper, we consider this task of automatically selecting optimal transfer languages as a ranking problem, and build models that consider the aforementioned features to perform this prediction. In experiments on representative NLP tasks, we demonstrate that our model predicts good transfer languages much better than ad hoc baselines considering single features in isolation, and glean insights on what features are most informative for each different NLP tasks, which may inform future ad hoc selection even without use of our method. Code, data, and pre-trained models are available at https://github.com/neulab/langrank

Semantic representation is the task of conveying the meaning of a natural language utterance by converting it to a logical form that can be processed and understood by machines. It is utilised by many applications in natural language processing (NLP), particularly in tasks relevant to natural language understanding (NLU). Due to the widespread use of semantic parsing in NLP, many semantic representation schemes with different forms have been proposed; Universal Conceptual Cognitive Annotation (UCCA) is one of them. UCCA is a cross-lingual semantic annotation framework that allows easy annotation without requiring substantial linguistic knowledge. UCCA-annotated datasets have been released so far for English, French, German, Russian, and Hebrew. In this paper, we present a UCCA-annotated Turkish dataset of 400 sentences that are obtained from the METU-Sabanci Turkish Treebank. We provide the UCCA annotation specifications defined for the Turkish language so that it can be extended further. We followed a semi-automatic annotation approach, where an external semantic parser is utilised for the initial annotation of the dataset, which is manually revised by two annotators. We used the same semantic parser model to evaluate the dataset with zero-shot and few-shot learning, demonstrating that even a small sample set from the target language in the training data has a notable impact on the performance of the parser (15.6% and 2.5% gain over zero-shot for labelled and unlabelled results, respectively).

Semantic Role Labeling (SRL) is one of the most challenging tasks in Natural Language Processing (NLP). SRL is a task that consists of predicate identification, argument identification and argument classification. In this article we present novel approach for argument classification that is based on deep neural network architecture. Traditional discrete based SRL relies heavily on feature engineering that uses syntactic structures in contrast to deep learning approaches that encode whole sentences without taking into account syntactic features. We present an approach that uses combination of syntactic features and external word representations from FastText. The advantages of using FastText embeddings is the generation of better vector representations for rare words and FastText gives better results for words that are not within the dictionary. These attributes for vector representations give good results for morphologically rich languages. Most of the SRL approaches today are trained on resource rich languages. In this article we present novel neural architecture for SRL that is suitable for resource poor morphology rich languages. Experiments on hr500k corpus shows that our syntax-aware approach shows competitive results for argument classification. We present architecture for argument classification that is based on Bidirectional Long-Short Term Memory (Bi-LSTM) and Conditional Random Field (CRF) decoding for finding optimal sequence. Our approach showed results that are very close to benchmark results with F1 score of 72%.

ABSTRACTBreast cancer (BC) detection based on mammogram images is still an open issue, particularly when there is little annotated data. Combining few‐shot learning (FSL) with transfer learning (TL) has been identified as a potential solution to overcome this problem due to its ability to learn from a few examples while producing robust features for classification. The objective of this study is to use and analyze FSL integrated with TL to enhance the classification accuracy and generalization ability in a limited dataset. The proposed approach integrates the FSL models (prototypical networks, matching networks, and relation networks) with the TL procedures. The models are trained using a small set of samples with annotation and can be assessed using various performance metrics. The models were trained and compared to the TL and the state‐of‐the‐art methods regarding accuracy, precision, recall, F1‐score, and area under the ROC curve (AUC). The models proved to be effective when integrated, and the relation networks model was the most accurate, with an accuracy of 95.6% and an AUC of 0.970. The models provided higher accuracy, recall, and F1‐scores, especially in the case of discerning between normal, benign, and malignant cases, as compared to TL traditional techniques and the various recent state‐of‐the‐art techniques. This integrated approach gives high efficiency, accuracy, and scalability to the whole BC detection process, and it has potential for further medical imaging domains. Future research will explore hyperparameter tuning and incorporating electronic health record systems to enhance diagnostic precision and individualized care.

Deep learning has been rapidly developed and obtained great achievements with a dataintensive condition. However, sufficient datasets are not always available in practical application. In the absence of data, humans can still perform well in studying and recognizing new items while it becomes a hard task for the computer to learn and generate from a small dataset. Thus, researchers are increasingly interested in few-shot learning. The purpose of few-shot learning is to allow computers to carry out unknown tasks with a few examples. Recently, effective few-shot models have frequently been designed using transfer learning approaches, with the metric method being an important branch in transfer learning. This article reviews the metric methodologies for few-short learning, analyzing the development of the metric based few-shot learning in the following three categories: traditional metric methods, relation network based metric methods and graph based metric methods. Then it compares the effectiveness of those models on a representative dataset and illustrates the feature of each category. Finally, it discusses the potential future research fields.