Robust Unsupervised Neural Machine Translation with Adversarial Denoising Training

Unsupervised Neural Machines Translation is the most ideal method to apply to Cantonese and Chinese translation because parallel data is scarce in this language pair. In this paper, we proposed a method that combined a modified cross-lingual language model and performed layer to layer attention on unsupervised neural machine translation. In our experiments, we observed that our proposed method does improve the Cantonese to Chinese and Chinese to Cantonese translation by 1.088 and 0.394 BLEU scores. We finally developed a web service based on our ideal approach to provide Cantonese to Chinese Translation and vice versa.

We describe an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text. We use multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. We validate our technique by extracting parallel sentence pairs on the BUCC 2017 bitext mining task and observe up to a 24.5 point increase (absolute) in F1 scores over previous unsupervised methods. We then improve an XLM-based unsupervised neural MT system pre-trained on Wikipedia by supplementing it with pseudo-parallel text mined from the same corpus, boosting unsupervised translation performance by up to 3.5 BLEU on the WMT'14 French-English and WMT'16 German-English tasks and outperforming the previous state-of-the-art. Finally, we enrich the IWSLT'15 English-Vietnamese corpus with pseudo-parallel Wikipedia sentence pairs, yielding a 1.2 BLEU improvement on the low-resource MT task. We demonstrate that unsupervised bitext mining is an effective way of augmenting MT datasets and complements existing techniques like initializing with pre-trained contextual embeddings.

In this paper, we deal with two problems in Japanese-English machine translation of news articles. The first problem is the quality of parallel corpora. Neural machine translation (NMT) systems suffer degraded performance when trained with noisy data. Because there is no clean Japanese-English parallel data for news articles, we build a novel parallel news corpus consisting of Japanese news articles translated into English in a content-equivalent manner. This is the first content-equivalent Japanese-English news corpus translated specifically for training NMT systems. The second problem involves the domain-adaptation technique. NMT systems suffer degraded performance when trained with mixed data having different features, such as noisy data and clean data. Though the existing methods try to overcome this problem by using tags for distinguishing the differences between corpora, it is not sufficient. We thus extend a domain-adaptation method using multi-tags to train an NMT model effectively with the clean corpus and existing parallel news corpora with some types of noise. Experimental results show that our corpus increases the translation quality, and that our domain-adaptation method is more effective for learning with the multiple types of corpora than existing domain-adaptation methods are.

Neural machine translation (NMT) systems trained on clean data usually suffer from performance degradation when translating noisy inputs. Existing works attempt to improve the robustness of NMT normally via data augmentation, where synthetic noisy data are mixed with original clean data, either for training NMT with the standard NMT loss alone, or for tuning auxiliary tasks in a multi-task learning manner. Typical auxiliary tasks include detecting and correcting noises, exploiting noisy outputs for contrastive learning etc. The aforementioned two auxiliary tasks are generally designed independently, and the modules for detecting and correcting noises are heavyweight. In this article, we propose a new framework, DetTransNet ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Det</b> ector- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Trans</b> lator <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Net</b> work), aiming to detect positions of noises in the input and translate the input simultaneously. The newly introduced noise detector module is essentially a lightweight binary classifier built upon the final layer of the encoder of the original Transformer model for the translation task, which is to identify at which position of the input has potential noise. The module has a very few parameters. In order to help the model capture the relationship between clean instances and their noisy counterparts, an extra loss is further introduced to enhance the interaction between clean and noisy data. In this way, we combine noise detection and contrastive learning together. As the model is able to identify and locate noises, a heuristic method is proposed to correct detected noises, in order to achieve better translations. Experiments show that DetTransNet is robust to four types of noises (deletion, insertion, swapping, keyboard), and obtain a substantial improvement of up to 1.6 BLEU points across different datasets.

With the advent of Neural Networks, Neural Machine Translation (NMT) has been a booming area of research in Natural Language Processing. Indic languages possess less significant parallel and monolingual corpus than English and other European languages. Even in Indic languages, many Indo-Aryan and Dravidian languages fall into low-resource languages, and techniques like Back Translation, Unsupervised Neural Machine Translation (UNMT), and Transfer Learning help achieve better translation accuracy. This paper uses the UNMT approach with a pre-trained Cross-Lingual Language Model (XLM) for the English to Kannada language pair. Our proposed methodology has achieved a 0.61 BLEU score in English to Kannada and a 0.32 BLEU score in Kannada to English translation.

While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric, MacroF1, and study its applicability to MT evaluation. We find that MacroF1 is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that MacroF1 can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the methods' outputs.

Neural Machine Translation (NMT) is one of the approaches of Machine Translation (MT) that have marked its significant progress in recent years for Indic languages. Even though a lot of contributions have been done for many Indic language pairs, there is no still such model which can be considered as a benchmark due to scarcity in rich parallel corpus, syntactic, semantic and morphological divergence across languages, difference in sentence ordering and so on. The lack of a rich parallel corpus has created a great hindrance in generating efficient NMT systems. Unsupervised learning based NMT (UNMT) was introduced to cater this concern as it is based on monolingual corpus. In this paper, we propose pivot-based UNMT model for English to Kannada translation with Telugu as pivot language using monolingual corpus of 1 lakh sentences for each language. To further enhance the translation quality, mBART pre-trained model is used. Since partial corpus is taken a conventional UNMT model is built for the comparison of proposed models with state-of-art model. With the conventional UNMT model and proposed architecture, BLEU score of 0.2 and 0.5 is achieved respectively indicating that pivot-based approach with Telugu as pivot enhances the translation accuracy.

Machine translation aims to minimize the language barrier between people of different linguistic backgrounds. Machine translation is an automatic translation technique between pairs of different languages. Machine translation using neural network or neural machine translation came into the picture due to several limitation associated with its predecessors which are rule based and statistical based models. For large data sets with a rich range of vocabulary, the neural network machine translation system provides fair translation accuracy. We have observed that there remain very few machine translators dedicated to Indian languages, especially those spoken in the North-East area. So our paper mainly focuses on implementing a neural machine translation system for the English-Assamese language. In this paper we used five different neural machine translation models for English-Assamese language pair using LSTMs and GRUs, along with an attention layer. We have made a comparison analysis based on the performance results of these models. We used BLEU Score to calculate the accuracy of these five models, thus achieving a BLEU score of 34.168%, which was the highest among the five models.

In this paper, we alleviate the local optimality of back-translation by learning a policy (takes the form of an encoder-decoder and is defined by its parameters) with future rewarding under the reinforcement learning framework, which aims to optimize the global word predictions for unsupervised neural machine translation. To this end, we design a novel reward function to characterize high-quality translations from two aspects: n-gram matching and semantic adequacy. The n-gram matching is defined as an alternative for the discrete BLEU metric, and the semantic adequacy is used to measure the adequacy of conveying the meaning of the source sentence to the target. During training, our model strives for earning higher rewards by learning to produce grammatically more accurate and semantically more adequate translations. Besides, a variational inference network (VIN) is proposed to constrain the corresponding sentences in two languages have the same or similar latent semantic code. On the widely used WMT’14 English-French, WMT’16 English-German and NIST Chinese-to-English benchmarks, our models respectively obtain 27.59/27.15, 19.65/23.42 and 22.40 BLEU points without using any labeled data, demonstrating consistent improvements over previous unsupervised NMT models.

Recently, the pretraining of models has been successfully applied to unsupervised and semi-supervised neural machine translation. A cross-lingual language model uses a pretrained masked language model to initialize the encoder and decoder of the translation model, which greatly improves the translation quality. However, because of a mismatch in the number of layers, the pretrained model can only initialize part of the decoder’s parameters. In this paper, we use a layer-wise coordination transformer and a consistent pretraining translation transformer instead of a vanilla transformer as the translation model. The former has only an encoder, and the latter has an encoder and a decoder, but the encoder and decoder have exactly the same parameters. Both models can guarantee that all parameters in the translation model can be initialized by the pretrained model. Experiments on the Chinese–English and English–German datasets show that compared with the vanilla transformer baseline, our models achieve better performance with fewer parameters when the parallel corpus is small.

Machine Translation (MT) has recently been strengthened by the emergence of deep neural network techniques, which have brought the dream of language translation in community-based practices and clinical practices closer to reality. This chapter introduces the clinical practice of MT, especially in low resource languages. The following are some of the highlights of this chapter. How MT played a crucial role in good social reasons is focused on in this chapter. The chapter starts with the history of translation technologies that have played a vital role in various crisis and relief scenarios as the Haitian earthquake in 2010 and Translators without Borders (TWB), respectively. The Standby Task Force (SBTF) deployed the technology to tackle misinformation during Coronavirus Pandemic 2019. Next, the chapter explores the importance of the MT system, where it describes MT literacy as required across society to impact the healthcare of human beings. It describes the challenges of translation for clinical texts. Some of those are highlighted as (a) Domain: characteristics of clinical text differentiated from texts with other domains, (b) code mixed low-resource languages: noisy clinical data and lower quantity of data in the digital form available for computation, (c) lack of clinical domain corpora, and (d) very rich morphological structure: e.g., procedures (e.g., hepatico-cholangio-jejuno-stomy) and chemicals (e.g., Hydro-xy-nitro-di-hydro-thy-mine). Strategies or approaches to building MT on the clinical text can be categorized as: (1) Rule-based MT (early days), (2) Statistical MT (SMT, one of the successful methods in MT history), and (3) Neural MT (NMT, recent advancements of MT). Both SMT and NMT that work on same grounds by learning from text and are the primary focus of this chapter. L earning transparency for the patient's use of clinical data is an essential component. Much of today's healthcare data is unstructured text, such as clinical text, doctor's notes, nursing notes, lab reports, test reports, and more. A clinical practitioner is faced with a language barrier for communicating with a patient to interpret patient histories, examine a clinical diagnosis, or restate the recommended treatment method and follow-up to aid comprehension. The only way a medical practitioner encourages patients to ask questions or respond to queries is by directing them to input text with the help of MT. Finally, the chapter discusses contemporary techniques with Neural Machine Translation (NMT), namely multilingual, transfer learning on clinical data.

Neural Machine Translation (NMT) has recently achieved the state-of-the-art in many machine translation tasks, but one of the challenges that NMT faces is the lack of parallel corpora, especially for low-resource language pairs. And the result is that the performance of NMT is much less effective for low-resource languages. To address this specific problem, in this paper, we describe a novel NMT model that is based on encoder-decoder architecture and relies on character-level inputs. Our proposed model employs Convolutional Neural Networks (CNN) and highway networks over character inputs, whose outputs are given to an encoder-decoder neural machine translation network. Besides, we also present two other approaches to improve the performance of the low-resource NMT system much further. First, we use language modeling implemented by denoising autoencoding to pre-train and initialize the full model. Second, we share the weights of the front few layers of two encoders between two languages to strengthen the encoding ability of the model. We demonstrate our model on two low-resource language pairs. On the IWSLT2015 English-Vietnamese translation task, our proposed model obtains improvements up to 2.5 BLEU points compared to the baseline. We also outperform the baseline approach more than 3 BLEU points on the CWMT2018 Chinese-Mongolian translation task.

The question of context in neural machine translation often focuses on topics related to document-level translation or intersentential context. However, there is a wide range of other aspects that can be considered under the umbrella of context. In this work, we survey ways that researchers have incorporated context into neural machine translation systems and the evaluation thereof. This includes building translation systems that operate at the paragraph level or the document level or ones that translate at the sentence level but incorporate information from other sentences. We also consider how issues like terminology consistency, anaphora, and world knowledge or external information can be considered as types of context relevant to the task of machine translation and its evaluation. Closely tied to these topics is the question of how to best evaluate machine translation output in a way that is sensitive to the contexts in which it appears. To this end, we discuss work on incorporating context into both human and automatic evaluations of machine translation quality. Furthermore, we also discuss recent experiments in the field as they relate to the use of large language models in translation and evaluation. We conclude with a view of the future of machine translation, where we expect to see issues of context continue to come to the forefront.

One of the important applications for which natural language processing (NLP) is used is the machine translation (MT) system, which automatically converts one natural language to another. It has witnessed various paradigm shifts since its inception. Statistical machine translation (SMT) has dominated MT research for decades. In the recent past, researchers have focused on developing MT systems based on artificial neural networks (ANN). In this paper, first, some important deep learning models that are mostly exploited in Neural Machine Translation (NMT) design are discussed. A systematic comparison was done between the performances of SMT and NMT concerning the English-to-Bangla and English-to-Hindi translation tasks. Most of the Indian scripts are morphologically rich, and the availability of a sufficient corpus is rare. We have presented and analyzed our work and a survey was conducted on other low-resource languages, and finally some useful conclusions have been drawn.

Intuitively, machine translation (MT) between closely related languages, such as Swedish and Danish, is easier than MT between more distant pairs, such as Finnish and Danish.Yet, the notions of 'closely related' languages and 'easier' translation have so far remained underspecified.Moreover, in the context of neural MT, this assumption was almost exclusively evaluated in scenarios where English was either the source or target language, leaving a broader cross-lingual view unexplored.In this work, we present a controlled study of language similarity and neural MT difficulty for 56 European translation directions.We test a range of language similarity metrics, some of which are reasonable predictors of MT difficulty.On a text-level, we reassess previously introduced indicators of MT difficulty, and find that they are not well-suited to our domain, or neural MT more generally.Ultimately, we hope that this work inspires further cross-lingual investigations of neural MT difficulty.