Semi-supervised Learning for Neural Machine Translation

In this paper, we deal with two problems in Japanese-English machine translation of news articles. The first problem is the quality of parallel corpora. Neural machine translation (NMT) systems suffer degraded performance when trained with noisy data. Because there is no clean Japanese-English parallel data for news articles, we build a novel parallel news corpus consisting of Japanese news articles translated into English in a content-equivalent manner. This is the first content-equivalent Japanese-English news corpus translated specifically for training NMT systems. The second problem involves the domain-adaptation technique. NMT systems suffer degraded performance when trained with mixed data having different features, such as noisy data and clean data. Though the existing methods try to overcome this problem by using tags for distinguishing the differences between corpora, it is not sufficient. We thus extend a domain-adaptation method using multi-tags to train an NMT model effectively with the clean corpus and existing parallel news corpora with some types of noise. Experimental results show that our corpus increases the translation quality, and that our domain-adaptation method is more effective for learning with the multiple types of corpora than existing domain-adaptation methods are.

Most neural machine translation (NMT) systems rely on parallel data, comprising text in the source language and its corresponding translation in the target language. While it’s acknowledged that context enhances NMT models, this work proposes a novel approach by incorporating external context, specifically explanations of source text meanings, akin to how human translators leverage context for comprehension. The suggested methodology innovatively addresses the challenge of incorporating lengthy contextual information into NMT systems. By employing state-of-the-art transformer-based models, external context is integrated, thereby enriching the translation process. A key aspect of the approach lies in the utilization of diverse text summarization techniques, strategically employed to efficiently distill extensive contextual details into the NMT framework. This novel solution not only overcomes the obstacle posed by lengthy context but also enhances the translation quality, marking a advancement in the field of NMT. Furthermore, the data-centric approach ensures robustness and effectiveness, yielding improvements in translation quality, as evidenced by a considerable boost in BLEU score points ranging from 0.46 to 1.87 over baseline models. Additionally, we make our dataset publicly available, facilitating further research in this domain.

In this paper, we focus on developing a Neural Machine Translation (NMT) system on English-to-Traditional-Chinese translation for financial prospectuses of companies which seek listing on the Hong Kong Stock Exchange. To the best of our knowledge, this is the first work on NMT for this specific domain. We propose a domain-specific NMT system by introducing a domain flag to indicate the target-side domain. By training the NMT model on the data from both the IPO corpus and the general domain corpus, we can expand the vocabulary while capturing the common writing styles and sentence structures. Our experimental results show that the proposed NMT system can achieve a significant improvement on translating the IPO documents. More significantly, through a blind assessment by a translator expert, our system outperforms two mainstream commercial tools, the Google translator and SDL Trado for some IPO documents.

This paper proposes an English-Chinese machine translation research method based on transfer learning. First, it expounds the theory of neural machine translation and transfer learning and related technologies. Neural machine translation is discussed, the advantages and disadvantages of various models are introduced, and the transformer neural machine translation model framework is selected. For low-resource Chinese-English parallel corpus and Tibetan-Chinese parallel corpus, 30 million Chinese-English parallel corpora, 100,000 Chinese-English low-resource parallel corpora, and 100,000 Tibetan-Chinese parallel corpora were used to pretrain the transformer machine translation architecture. The decoders are all composed of 6 identical hidden layers, the initialization of the model parameters is done by the transformer uniform distribution, and the model training uses Adam as the optimizer. In the model transfer part, the parameters with the better effect of the pretrained model are transferred to the low-resource Chinese-English and Tibetan-Chinese machine translation model training, so as to achieve the purpose of knowledge transfer. The results show that the model transfer learning of low-resource Chinese-English parallel corpus improves the translation system's translation by 3.97 BLEU values compared with the translation system without transfer learning at 0.34 BLEU values. Model transfer learning on low-resource Tibetan-Chinese parallel corpus increases the BLEU value by 2.64 BLEU compared to the translation system without transfer learning. The neural machine translation system that uses BPE technology for preprocessing plus model transfer learning is compared to the translation system that only performs transfer learning and shows an improved 0.26 BLEU value.It is verified that the transfer learning method proposed in this paper has a certain improvement in the effect of low-resource Chinese-English and Tibetan-Chinese neural machine translation models.

Machine translation aims to minimize the language barrier between people of different linguistic backgrounds. Machine translation is an automatic translation technique between pairs of different languages. Machine translation using neural network or neural machine translation came into the picture due to several limitation associated with its predecessors which are rule based and statistical based models. For large data sets with a rich range of vocabulary, the neural network machine translation system provides fair translation accuracy. We have observed that there remain very few machine translators dedicated to Indian languages, especially those spoken in the North-East area. So our paper mainly focuses on implementing a neural machine translation system for the English-Assamese language. In this paper we used five different neural machine translation models for English-Assamese language pair using LSTMs and GRUs, along with an attention layer. We have made a comparison analysis based on the performance results of these models. We used BLEU Score to calculate the accuracy of these five models, thus achieving a BLEU score of 34.168%, which was the highest among the five models.

State-of-the-art (SOTA) neural machine translation (NMT) systems translate texts at sentence level, ignoring context: intra-textual information, like the previous sentence, and extra-textual information, like the gender of the speaker. As a result, some sentences are translated incorrectly. Personalised NMT (PersNMT) and document-level NMT (DocNMT) incorporate this information into the translation process. Both fields are relatively new and previous work within them is limited. Moreover, there are no readily available robust evaluation metrics for them, which makes it difficult to develop better systems, as well as track global progress and compare different methods. This thesis proposal focuses on PersNMT and DocNMT for the domain of dialogue extracted from TV subtitles in five languages: English, Brazilian Portuguese, German, French and Polish. Three main challenges are addressed: (1) incorporating extra-textual information directly into NMT systems; (2) improving the machine translation of cohesion devices; (3) reliable evaluation for PersNMT and DocNMT.

Neural Machine Translation (NMT) has attracted growing interest in recent years for its promising performance compared to traditional approaches such as Statistical Machine Translation. However, its application to languages having different structures, like the (English, Arabic) pair that interests us in this work, degrades its performance. Indeed, the limited vocabulary size required by the NMT models decreases the vocabulary coverage rate of the Arabic language, well known by its morphological richness. Likewise, long sentences present an additional challenge to NMT systems because they perform less well for longer sentences than for the shorter ones. In this paper, we provide a series of experiments to mitigate the effects of these constraints. To address the problem of out-of-vocabulary words, we integrated into factored NMT models morphosyntactic features as an output factor, namely stem, lemma, POS, root, and pattern. We have also developed two techniques for segmenting long sentences into smaller sub-sentences. The first uses a list of lexical markers that we have collected as segmentation points, and the second integrates into the NMT model the parallel phrases extracted by an SMT system. The experiments carried out on the English–Arabic pair show that the proposed approaches considerably improve the translation quality compared to the basic NMT system.

Machine Translation (MT) systems are now being improved with the use of an ongoing methodology known as Neural Machine Translation (NMT). Natural language processing (NLP) researchers have shown that NMT systems are unable to deal with out-of-vocabulary (OOV) words and multi-word expressions (MWEs) in the text. OOV terms are those that are not currently included in the vocabulary that is used by the NMT system. MWEs are phrases that consist of a minimum of two terms but are treated as a single unit. MWEs have great importance in NLP, linguistic theory, and MT systems. In this article, OOV words and MWEs are handled for the Punjabi to English NMT system. A parallel corpus for Punjabi to English containing MWEs was developed and used to train the different models of NMT. Punjabi is a low-resource language as it lacks the availability of a large parallel corpus for building various NLP tools, and this is an attempt to improve the accuracy of Punjabi in the English NMT system by using named entities and MWEs in the corpus. The developed NMT models were assessed using human evaluation through adequacy, fluency and overall rating as well as automated assessment tools such as the bilingual evaluation study (BLEU) and translation error rate (TER) score. Results show that using word embedding (WE) and MWEs corpus increased the accuracy of translation for the Punjabi to English language pair. The best BLEU score obtained was 15.45 for the small test set, 43.32 for the medium test set, and 34.5 for the large test set, respectively. The best TER rate score obtained was 57.34% for the small test set, 37.29% for the medium test set, and 53.79% for the large test set, repectively.

Neural machine translation (NMT) systems are usually trained on clean parallel data. They can perform very well for translating clean in-domain texts. However, as demonstrated by previous work, the translation quality significantly worsens when translating noisy texts, such as user-generated texts (UGT) from online social media. Given the lack of parallel data of UGT that can be used to train or adapt NMT systems, we synthesize parallel data of UGT, exploiting monolingual data of UGT through cross-lingual language model pre-training and zero-shot NMT systems. This paper presents two different but complementary approaches: one alters given clean parallel data into UGT-like parallel data whereas the other generates translations from monolingual data of UGT. On the MTNT translation tasks, we show that our synthesized parallel data can lead to better NMT systems for UGT while making them more robust in translating texts from various domains and styles.

In Brazil, the governmental body responsible for overseeing and coordinating post-graduate programs, CAPES, keeps records of all theses and dissertations presented in the country. Information regarding such documents can be accessed online in the Theses and Dissertations Catalog (TDC), which contains abstracts in Portuguese and English, and additional metadata. Thus, this database can be a potential source of parallel corpora for the Portuguese and English languages. In this article, we present the development of a parallel corpus from TDC, which is made available by CAPES under the open data initiative. Approximately 240,000 documents were collected and aligned using the Hunalign tool. We demonstrate the capability of our developed corpus by training Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) models for both language directions, followed by a comparison with Google Translate (GT). Both translation models presented better BLEU scores than GT, with NMT system being the most accurate one. Sentence alignment was also manually evaluated, presenting an average of 82.30% correctly aligned sentences. Our parallel corpus is freely available in TMX format, with complementary information regarding document metadata

Neural Machine Translation (NMT), like many other deep learning domains, typically suffers from over-parameterization, resulting in large storage sizes. This paper examines three simple magnitude-based pruning schemes to compress NMT models, namely class-blind, class-uniform, and class-distribution, which differ in terms of how pruning thresholds are computed for the different classes of weights in the NMT architecture. We demonstrate the efficacy of weight pruning as a compression technique for a state-of-the-art NMT system. We show that an NMT model with over 200 million parameters can be pruned by 40% with very little performance loss as measured on the WMT'14 English-German translation task. This sheds light on the distribution of redundancy in the NMT architecture. Our main result is that with retraining, we can recover and even surpass the original performance with an 80%-pruned model.

Introduction to the special issue on deep learning approaches for machine translation

Neural machine translation, which has an encoder-decoder framework, is considered to be a feasible way for future machine translation. Nevertheless, with the fusion of multiple languages and the continuous emergence of new words, most current neural machine translation systems based on von Neumann’s architecture have seen a substantial increase in the number of devices for the decoder, resulting in high-energy consumption rate. Here, a multilevel photosensitive blending semiconductor optoelectronic synaptic transistor (MOST) with two different trapping mechanisms is firstly demonstrated, which exhibits 8 stable and well distinguishable states and synaptic behaviors such as excitatory postsynaptic current, short-term memory, and long-term memory are successfully mimicked under illumination in the wavelength range of 480–800 nm. More importantly, an optical decoder model based on MOST is successfully fabricated, which is the first application of neuromorphic device in the field of neural machine translation, significantly simplifying the structure of traditional neural machine translation system. Moreover, as a multi-level synaptic device, MOST can further reduce the number of components and simplify the structure of the codec model under light illumination. This work first applies the neuromorphic device to neural machine translation, and proposes a multi-level synaptic transistor as the based cell of decoding module, which would lay the foundation for breaking the bottleneck of machine translation.

Neural machine translation (NMT) is often described as ‘data hungry’ as it typically requires large amounts of parallel data in order to build a good-quality machine translation (MT) system. However, most of the world's language-pairs are low-resource or extremely low-resource. This situation becomes even worse if a specialised domain is taken into consideration for translation. In this paper, we present a novel data augmentation method which makes use of bilingual word embeddings (BWEs) learned from monolingual corpora and bidirectional encoder representations from transformer (BERT) language models (LMs). We augment a parallel training corpus by introducing new words (i.e. out-of-vocabulary (OOV) items) and increasing the presence of rare words on both sides of the original parallel training corpus. Our experiments on the simulated low-resource German–English and French–English translation tasks show that the proposed data augmentation strategy can significantly improve state-of-the-art NMT systems and outperform the state-of-the-art data augmentation approach for low-resource NMT.

This paper introduces current-day automated metrics for neural machine translation evaluation and discusses two ways of result evaluation, which include evaluation using specific automated metrics as well as evaluation by an expert translator. Neural machine translation systems were tested for analysis and evaluation purposes. Google Translate and DeepL Translate systems, which apply a neural network approach, are presented as candidates. The following metrics were considered for evaluation: METEOR as a traditional reference-based metric, COMET as a neural reference-based metric, and COMET-kiwi as a neural reference-free metric. It can be noted that even if these metrics are automated, there is a huge human impact in this automation. Even models with a neural network approach are trained on data provided by humans, as today it is impossible to get rid of reference translations or quality estimations made by experts. Metrics allow to understand and better investigate machine translation, its features and limitations, as well as to determine the direction of its development. As a part of the analysis, a piece of source text was selected for translation, translated into the target languages using selected neural machine translation systems to get candidate translations, and then a reference translation was specified for each of them. The results of the metrics evaluation were helpful for understanding how close machine translation is to human translation, and also helped to learn more about the current stage of machine translation systems development. Expert evaluation helped to understand how well such systems are doing in terms of translation performance.