A Joint Back-Translation and Transfer Learning Method for Low-Resource Neural Machine Translation

This paper proposes an English-Chinese machine translation research method based on transfer learning. First, it expounds the theory of neural machine translation and transfer learning and related technologies. Neural machine translation is discussed, the advantages and disadvantages of various models are introduced, and the transformer neural machine translation model framework is selected. For low-resource Chinese-English parallel corpus and Tibetan-Chinese parallel corpus, 30 million Chinese-English parallel corpora, 100,000 Chinese-English low-resource parallel corpora, and 100,000 Tibetan-Chinese parallel corpora were used to pretrain the transformer machine translation architecture. The decoders are all composed of 6 identical hidden layers, the initialization of the model parameters is done by the transformer uniform distribution, and the model training uses Adam as the optimizer. In the model transfer part, the parameters with the better effect of the pretrained model are transferred to the low-resource Chinese-English and Tibetan-Chinese machine translation model training, so as to achieve the purpose of knowledge transfer. The results show that the model transfer learning of low-resource Chinese-English parallel corpus improves the translation system's translation by 3.97 BLEU values compared with the translation system without transfer learning at 0.34 BLEU values. Model transfer learning on low-resource Tibetan-Chinese parallel corpus increases the BLEU value by 2.64 BLEU compared to the translation system without transfer learning. The neural machine translation system that uses BPE technology for preprocessing plus model transfer learning is compared to the translation system that only performs transfer learning and shows an improved 0.26 BLEU value.It is verified that the transfer learning method proposed in this paper has a certain improvement in the effect of low-resource Chinese-English and Tibetan-Chinese neural machine translation models.

Most dominant neural machine translation (NMT) models are restricted to make predictions only according to the local context of preceding words in a left-to-right manner. Although many previous studies try to incorporate global information into NMT models, there still exist limitations on how to effectively exploit bidirectional global context. In this paper, we propose a Confidence Based Bidirectional Global Context Aware (CBBGCA) training framework for NMT, where the NMT model is jointly trained with an auxiliary conditional masked language model (CMLM). The training consists of two stages: (1) multi-task joint training; (2) confidence based knowledge distillation. At the first stage, by sharing encoder parameters, the NMT model is additionally supervised by the signal from the CMLM decoder that contains bidirectional global contexts. Moreover, at the second stage, using the CMLM as teacher, we further pertinently incorporate bidirectional global context to the NMT model on its unconfidently-predicted target words via knowledge distillation. Experimental results show that our proposed CBBGCA training framework significantly improves the NMT model by +1.02, +1.30 and +0.57 BLEU scores on three large-scale translation datasets, namely WMT'14 English-to-German, WMT'19 Chinese-to-English and WMT'14 English-to-French, respectively.

The dominant neural machine translation (NMT) models that based on the encoder-decoder architecture have recently achieved the state-of-the-art performance. Traditionally, the NMT models only depend on the representations learned during training for mapping a source sentence into the target domain. However, the learned representations often suffer from implicit and inadequately informed properties. In this paper, we propose a novel bilingual topic enhanced NMT (BLTNMT) model to improve translation performance by incorporating bilingual topic knowledge into NMT. Specifically, the bilingual topic knowledge is included into the hidden states of both encoder and decoder, as well as the attention mechanism. With this new setting, the proposed BLT-NMT has access to the background knowledge implied in bilingual topics which is beyond the sequential context, and enables the attention mechanism to attend to topic-level attentions for generating accurate target words during translation. Experimental results show that the proposed model consistently outperforms the traditional RNNsearch and the previous topic-informed NMT on Chinese-English and EnglishGerman translation tasks. We also introduce the bilingual topic knowledge into the newly emerged Transformer base model on English-German translation and achieve a notable improvement.

Neural Machine Translation (NMT) has recently achieved the state-of-the-art in many machine translation tasks, but one of the challenges that NMT faces is the lack of parallel corpora, especially for low-resource language pairs. And the result is that the performance of NMT is much less effective for low-resource languages. To address this specific problem, in this paper, we describe a novel NMT model that is based on encoder-decoder architecture and relies on character-level inputs. Our proposed model employs Convolutional Neural Networks (CNN) and highway networks over character inputs, whose outputs are given to an encoder-decoder neural machine translation network. Besides, we also present two other approaches to improve the performance of the low-resource NMT system much further. First, we use language modeling implemented by denoising autoencoding to pre-train and initialize the full model. Second, we share the weights of the front few layers of two encoders between two languages to strengthen the encoding ability of the model. We demonstrate our model on two low-resource language pairs. On the IWSLT2015 English-Vietnamese translation task, our proposed model obtains improvements up to 2.5 BLEU points compared to the baseline. We also outperform the baseline approach more than 3 BLEU points on the CWMT2018 Chinese-Mongolian translation task.

Recent works have shown that Neural Machine Translation (NMT) models achieve impressive performance, however, questions about understanding the behavior of these models remain unanswered. We investigate the unexpected volatility of NMT models where the input is semantically and syntactically correct. We discover that with trivial modifications of source sentences, we can identify cases where unexpected changes happen in the translation and in the worst case lead to mistranslations. This volatile behavior of translating extremely similar sentences in surprisingly different ways highlights the underlying generalization problem of current NMT models. We find that both RNN and Transformer models display volatile behavior in 26% and 19% of sentence variations, respectively.

Transfer Learning Based Neural Machine Translation of English-Khasi on Low-Resource Settings

Success of neural networks in natural language processing has paved the way for neural machine translation (NMT), which rapidly became the mainstream approach in machine translation. Significant improvement in translation performance has been achieved with breakthroughs such as encoder-decoder networks, attention mechanism, and Transformer architecture. However, the necessity of large amounts of parallel data for training an NMT system and rare words in translation corpora are issues yet to be overcome. In this article, we approach NMT of the low-resource Turkish-English language pair. We employ state-of-the-art NMT architectures and data augmentation methods that exploit monolingual corpora. We point out the importance of input representation for the morphologically rich Turkish language and make a comprehensive analysis of linguistically and non-linguistically motivated input segmentation approaches. We prove the effectiveness of morphologically motivated input segmentation for the Turkish language. Moreover, we show the superiority of the Transformer architecture over attentional encoder-decoder models for the Turkish-English language pair. Among the employed data augmentation approaches, we observe back-translation to be the most effective and confirm the benefit of increasing the amount of parallel data on translation quality. This research demonstrates a comprehensive analysis on NMT architectures with different hyperparameters, data augmentation methods, and input representation techniques, and proposes ways of tackling the low-resource setting of Turkish-English NMT.

Machine Translation (MT) has recently been strengthened by the emergence of deep neural network techniques, which have brought the dream of language translation in community-based practices and clinical practices closer to reality. This chapter introduces the clinical practice of MT, especially in low resource languages. The following are some of the highlights of this chapter. How MT played a crucial role in good social reasons is focused on in this chapter. The chapter starts with the history of translation technologies that have played a vital role in various crisis and relief scenarios as the Haitian earthquake in 2010 and Translators without Borders (TWB), respectively. The Standby Task Force (SBTF) deployed the technology to tackle misinformation during Coronavirus Pandemic 2019. Next, the chapter explores the importance of the MT system, where it describes MT literacy as required across society to impact the healthcare of human beings. It describes the challenges of translation for clinical texts. Some of those are highlighted as (a) Domain: characteristics of clinical text differentiated from texts with other domains, (b) code mixed low-resource languages: noisy clinical data and lower quantity of data in the digital form available for computation, (c) lack of clinical domain corpora, and (d) very rich morphological structure: e.g., procedures (e.g., hepatico-cholangio-jejuno-stomy) and chemicals (e.g., Hydro-xy-nitro-di-hydro-thy-mine). Strategies or approaches to building MT on the clinical text can be categorized as: (1) Rule-based MT (early days), (2) Statistical MT (SMT, one of the successful methods in MT history), and (3) Neural MT (NMT, recent advancements of MT). Both SMT and NMT that work on same grounds by learning from text and are the primary focus of this chapter. L earning transparency for the patient's use of clinical data is an essential component. Much of today's healthcare data is unstructured text, such as clinical text, doctor's notes, nursing notes, lab reports, test reports, and more. A clinical practitioner is faced with a language barrier for communicating with a patient to interpret patient histories, examine a clinical diagnosis, or restate the recommended treatment method and follow-up to aid comprehension. The only way a medical practitioner encourages patients to ask questions or respond to queries is by directing them to input text with the help of MT. Finally, the chapter discusses contemporary techniques with Neural Machine Translation (NMT), namely multilingual, transfer learning on clinical data.

Machine translation has significantly advanced due to the development of transformer architecture, which is utilised by many modern deep-learning models.However, low-resource languages, such as Lithuanian, still face challenges stemming from the limited availability of training data and resource constraints.This study examines the translation capabilities of Neural Machine Translation (NMT) models and Large Language Models (LLMs), comparing their performance in low-resource translation tasks.Furthermore, it assesses the impact of parameter scaling and fine-tuning on their effectiveness in enhancing model performance.The evaluation showed that while LLMs demonstrated proficiency in low-resource translation, their results were lower compared to NMT models, which remained consistent across smaller variants.However, as model size increased, the lead was not as prominent, correlating with automatic and human evaluations.The effort to enhance translation accuracy through fine-tuning proved to be an effective strategy, demonstrating improvements in vocabulary expansion and structural coherence in both architectures.These findings highlight the importance of diverse datasets, comprehensive model design, and finetuning techniques in addressing the challenges of low-resourced language translation.This project, one of the first studies to focus on the low-resourced Lithuanian language, aims to contribute to the broader discourse and ongoing efforts to enhance accessibility and inclusivity in Natural Language Processing.

Machine translation (MT) for low-resource languages continues to face significant challenges because of limited digital resources and parallel corpora, despite remarkable developments in neural machine translation (NMT). Addressing these challenges requires a thorough review of existing research to identify effective strategies and methods. To achieve this, a systematic literature review (SLR) is conducted following PRISMA guidelines and systematically analysing studies published in various academic databases in the last five years (between 2020 and 2024). A total of 69 relevant articles were examined to evaluate the performance of MT, explore persistent challenges and assess the effectiveness of proposed or used solutions. The analysis shows that while NMT has emerged as the predominant approach, its effectiveness is often reduced by the scarcity of training data and the structural complexity of low-resource languages. Strategies such as active learning, data augmentation, multilingual model and transfer learning are identified as critical for improving translation performance. Additionally, emerging research trends, including data pre-processing, optimization of decoder and rule-based approach demonstrate promising directions for addressing existing limitations. In terms of evaluation, most of the studies used Character n-gram F-score (ChrF), Translation Edit Rate (TER), Metric for Evaluation of Translation with Explicit Ordering (METEOR), Word Error Rate (WER) and Bilingual Evaluation underscore (BLEU) as techniques’ validation metrics. This review provides a detailed evaluation of the current state of MT for low-resource languages and emphasizes the need for further research into underrepresented languages and the development of comprehensive datasets.

To improve the function of machine translation to adapt to global language translation, the work takes deep neural network (DNN) as the basic theory, carries out transfer learning and neural network translation modeling, and optimizes the word alignment function in machine translation performance. First, the work implements a deep learning translation network model for English translation. On this basis, the neural machine translation model is designed under transfer learning. The random shielding method is introduced to implement the language training model, and the machine translation is slightly adjusted as the goal of transfer learning, thereby improving the semantic understanding ability in translation performance. Meanwhile, the work design introduces the method of word alignment optimization and optimizes the performance of word alignment in the transformer system by using word corpus. The experimental results show that the proposed method reduces the average alignment error rate by 8.1%, 24.4%, and 22.1% in EnRo (English-Roman), EnGe (English-German), and EnFr (English-French), respectively, compared with the previous algorithms. Compared with the designed optimization method, the word alignment error rate is lower than that of traditional methods. The modeling and optimization method is feasible, which can effectively solve the problems of insufficient information utilization, large parameter scale, and difficult storage in the process of machine translation. Additionally, it provides a feasible idea and direction for the optimization and improvement in neural machine translation (NMT) system.

In this paper we describe our team's (NICT-5) Neural Machine Translation (NMT) models whose translations were submitted to shared tasks of the 7 th Workshop on Asian Translation.We participated in the Indic language multilingual sub-task as well as the NICT-SAP multilingual multi-domain sub-task.We focused on naive many-to-many NMT models which gave reasonable translation quality despite their simplicity.Our observations are twofold: (a.) Many-to-many models suffer from a lack of consistency where the translation quality for some language pairs is very good but for some others it is terrible when compared against one-to-many and many-toone baselines.(b.) Oversampling smaller corpora does not necessarily give the best translation quality for the language pair associated with that pair.

The classification of brain tumor is a very important section of medical diagnosis because correct and early diagnosis can greatly enhance the outcome of the patient. To aid in automated and dependable diagnostic methods, this paper examines the efficacy of already trained deep learning model to classify brain magnetic resonance imaging (MRI) scans into four categories, namely, pituitary tumor, meningioma, glioma, and no tumor. This was done using a publicly available set of 7,023 brain MRI images. DenseNet121, ResNet50, Xception and MobileNet four advanced convolutional neural network (CNN) models were also fine-tuned with transfer learning, as well as image preprocessing and data augmentation methods. Transfer learning was used to facilitate effective model adaptation with a low computation complexity and improved classification. According to the experimental findings, DenseNet121 performed better as it obtained the highest classification accuracy of 98.47% and the highest F1 score of 98.47%. The assessed models presented a high level of generalization and steady performance under a variety of evaluation criteria, which also indicates their possible application in the clinical decision support systems. These encouraging findings, however, need additional performance to improve recall on individual tumor classes, as well as the interpretability of deep-learning predictions in clinical settings. On the whole, the results highlight the great potential of the deep learning and transfer learning methods in medical image analysis, which is an opportunity to establish a more reliable, scalable and efficient solution to diagnosis. The next phase of work will involve diversifying the datasets, enhancing the model explainability, and assessing the performance of AI-oriented healthcare systems in practice to make them more widely adopted.

Generative adversarial networks (GANs) has achieved great success in the field of image processing, Adversarial Neural Machine Translation(NMT) is the application of GANs to machine translation. Unlike previous work training NMT model through maximizing the likelihood of the human translation, Adversarial NMT minimizes the distinction between human translation and the translation generated by a NMT model. Even though Adversarial NMT has achieved impressive results, while using little in the way of prior knowledge. In this paper, we integrated bilingual dictionaries to Adversarial NMT by leveraging a character model. Extensive experiment shows that our proposed methods can achieve remarkable improvement on the translation quality of Adversarial NMT, and obtain better result than several strong baselines.

While end-to-end neural machine translation (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low-resource language pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on the assumption, our method is able to train a source-to-target NMT model ("student") without parallel corpora available guided by an existing pivot-to-target NMT model ("teacher") on a source-pivot parallel corpus. Experimental results show that the proposed method significantly improves over a baseline pivot-based model by +3.0 BLEU points across various language pairs.