Towards Multimodal Simultaneous Neural Machine Translation

Multi-modal neural machine translation (MNMT) aims to translate sentences from the source language into the target language with the aid of corresponding images. Unfortunately, there is a considerable modality gap between the semantic-related images and texts in terms of data form and semantic expression. How to fully incorporate visual information into texts to enhance the performance of machine translation is one of the critical issues for MNMT. However, the initial visual and textual features are generally extracted with their modality-specific models; Consequently, there is a considerable representation gap between images and texts. Most previous MNMT works prefer only to adopt the feature-level fusion strategies to learn multi-modal representation, while the modality representation gap is often ignored. To this end, this paper proposes a progressive multi-modal Transformer (ProMul-Trans) with Modality Difference-Aware (MDA) to address the visual-to-textual fusion problem raised in MNMT. We first employ MDA to capture the modality-consistency information by taking visual and textual representations as inputs in each Transformer layer. Then a layer-level progressive fusion (Layer-ProFusion) strategy is adopted to progressively align visual and textual representations layer-by-layer to enhance machine translation performance. Experiment results on the Multi30k dataset are conducted, and the results show that the proposed approach outperforms the compared state-of-the-art (SOTA) methods on English <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rightarrow$</tex-math></inline-formula> German (En <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rightarrow$</tex-math></inline-formula> De), English <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rightarrow$</tex-math></inline-formula> French (En <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rightarrow$</tex-math></inline-formula> Fr) and English <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rightarrow$</tex-math></inline-formula> Czech (En <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rightarrow$</tex-math></inline-formula> Cs) tasks. We release the code at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JunjieYe-MMT/HierProMul-Trans</uri> .

Multi-modal simultaneous machine translation fusion of image information

Current multimodal neural machine translation (MNMT) approaches primarily focus on ensuring consistency between visual annotations and the source language, often overlooking the broader aspect of multimodal coherence, including target–visual and bilingual–visual alignment. In this paper, we propose a novel approach that effectively leverages target–visual consistency (TVC) and bilingual–visual consistency (BiVC) to improve MNMT performance. Our method leverages visual annotations depicting concepts across bilingual parallel sentences to enhance multimodal coherence in translation. We exploit target–visual harmony by extracting contextual cues from visual annotations during auto-regressive decoding, incorporating vital future context to improve target sentence representation. Additionally, we introduce a consistency loss promoting semantic congruence between bilingual sentence pairs and their visual annotations, fostering a tighter integration of textual and visual modalities. Extensive experiments on diverse multimodal translation datasets empirically demonstrate our approach’s effectiveness. This visually aware, data-driven framework opens exciting opportunities for intelligent learning, adaptive control, and robust distributed optimization of multi-agent systems in uncertain, complex environments. By seamlessly fusing multimodal data and machine learning, our method paves the way for novel control paradigms capable of effectively handling the dynamics and constraints of real-world multi-agent applications.

This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding regions of an image. The proposed visual attention mechanism captures the relationship between a word and an image region more precisely than a conventional visual attention mechanism trained through MNMT in an unsupervised manner. Our experiments on English-German and German-English translation tasks using the Multi30k dataset and on English-Japanese and Japanese-English translation tasks using the Flickr30k Entities JP dataset show that a Transformer-based MNMT model can be improved by incorporating our proposed supervised visual attention mechanism and that further improvements can be achieved by combining it with a supervised cross-lingual attention mechanism (up to +1.61 BLEU, +1.7 METEOR).

Simultaneous translation is a task in which translation begins before the speaker has finished speaking, so it is important to decide when to start the translation process. However, deciding whether to read more input words or start to translate is difficult for language pairs with different word orders such as English and Japanese. Motivated by the concept of pre-reordering, we propose a couple of simple decision rules using the label of the next constituent predicted by incremental constituent label prediction. In experiments on English-to-Japanese simultaneous translation, the proposed method outperformed baselines in the quality-latency trade-off.

Multimodal machine translation (MMT) aims to improve neural machine translation (NMT) with additional visual information, but most existing MMT methods require paired input of source sentence and image, which makes them suffer from shortage of sentence-image pairs. In this paper, we propose a phrase-level retrieval-based method for MMT to get visual information for the source input from existing sentence-image data sets so that MMT can break the limitation of paired sentence-image input. Our method performs retrieval at the phrase level and hence learns visual information from pairs of source phrase and grounded region, which can mitigate data sparsity. Furthermore, our method employs the conditional variational auto-encoder to learn visual representations which can filter redundant visual information and only retain visual information related to the phrase. Experiments show that the proposed method significantly outperforms strong baselines on multiple MMT datasets, especially when the textual context is limited.

This paper introduces and summarises the findings of a new shared task at the intersection of Natural Language Processing and Computer Vision: the generation of image descriptions in a target language, given an image and/or one or more descriptions in a different (source) language. This challenge was organised along with the Conference on Machine Translation (WMT16), and called for system submissions for two task variants: (i) a translation task, in which a source language image description needs to be translated to a target language, (optionally) with additional cues from the corresponding image, and (ii) a description generation task, in which a target language description needs to be generated for an image, (optionally) with additional cues from source language descriptions of the same image. In this first edition of the shared task, 16 systems were submitted for the translation task and seven for the image description task, from a total of 10 teams.

We introduce a Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation. Our decoder learns to attend to source-language words and parts of an image independently by means of two separate attention mechanisms as it generates words in the target language. We find that our model can efficiently exploit not just back-translated in-domain multi-modal data but also large general-domain text-only MT corpora. We also report state-of-the-art results on the Multi30k data set.

The present study proposes a source speech register analysis model that incorporates textual and situational components and parameters of register equivalence in the target language to identify the relations between the source language register and student interpreter’s performance, textual register mismatch between the source and target languages, and the relevance between register equivalence parameters in the target language and interpreting quality. The findings are as follows: first, source language register does have influence on interpreter’s performance, and relations exist between the two. In general, higher source language register brings better interpreting quality. Three assessment criteria and the overall quality differ across different levels of registers in the source language. Second, register mismatch between the source and target languages happens since interpreters are inclined to lower the register of interpreting output from the high-register source speech and increase the register level in the target language in the case of the low-register speech. Third, intonation and voice have the highest correlations with the interpreters’ performance. Fluency ranks the second, and lexical-semantic choices the third.

The study focuses on teaching the translation of audiovisual memes: comic image/video and text combinations that tend to go viral on the Internet. The key features of a meme are a comic effect resulting from surprise, disrupted expectations, the clash between the text and the video; “viral” expansion across the Internet and other social media. Mastering the skills required to translate such audiovisual content is essential since memes are highly popular on the Internet with all age groups, there are numerous allusions references to the popular memes in the mass media, fiction, movies, and TV series. Teaching audiovisual meme translation shall consist of analyzing the meme properties, building meme comprehension and metacognitive analysis skills, and mastering specific approaches and techniques for meme representation in the source and target languages. An emphasis should be placed on developing the ability to adequately reconstruct meme ecosystems in the source language and find their analogs in the target language as well as the skills required to identify archetypical narratives in the source language and build them in the target language. It is shown that memes have some similarities to set expressions and proverbs. The meme translation curriculum shall build the ability to find and translate set expressions and proverbs in audiovisual works. It is also shown that translation teachers should have specialized training in teaching the translation of memes as a special type of audiovisual content. It can be implemented in the format of short seminars/webinars, or within the teacher further education framework.

Problem statement: Any Arabic to English Machine Translation (MT) system should be capable of dealing with word order which Arabic exhibits. This poses a significant challenge to MT due to the vast number of ways to express the same sentence in Arabic. The ordering features are very important and should be carefully applied to ensure the generation of sentence in the target language. Because they apply to the target language, it shoul d fulfill the specific requirement of this language . Mistakes in the MT output can be either the result of analysis problems at the source language level, or due to generation problem at target language level. Word order rules are crucial for the generation of sentences in the target language. They also serve a s rules for the ordering of sentence constituents. These rules draw their information from the syntact ic knowledge. The word order problem becomes more obvious when making machine translation between languages that have rich morphological variations. Approach: The main objective of this research is to develop a machine translation that translates Arabic noun phrases into English by usin g transfer-based approach. A system called Npae- Rbmt has been developed in this research. Transfer- based machine translation is one instance of rule- based machine-translation approaches and is current ly one of the most widely used methods of machine translation. The idea of transfer-based mac hine translation it is necessary to have an intermediate representation that captures the mean ing of the original sentence in order to generate the correct translation. Using advantages of transf er-based machine translation such as analysis step, the Transfer-based becomes simpler as linguistic an alysis goes deeper-as the representation of analysi s step becomes more abstract. In fact, a major goal o f MT research is to define a level of analysis whic h is so deep in which transfer-based machine translat ion is able to do. Results: The method was tested on 88 thesis titles and journals from the computer science domain. The accuracy of the result was 94.6%. These results proved the viability of this a pproach for distant languages. Conclusion: Based on the achieved results, we have managed to perform th e syntactic reordering within an Arabic noun phrases to English translation task by using transf er-based machine translation and also achieved reasonable improvements in translation quality over related approach.

This paper discusses English-poetry translation, Heusca by John Conford into Indonesian translated by Chairil Anwar. The purpose of this study is to determine how the translator considers aesthetic and expressive values from the source language and translate them into target language. Aesthetic values are the means of delivery thought through the use of poetic language such as diction or word choice, metaphor, imageri, and others figurative language. Meanwhile, the expressive values are the means of delivery thoughts through the author's mind or emotions such as structure, rhyme, and pronunciation. The method used in this study is a qualitative descriptive analysis of the meaning. The results of analysis show that in terms of meaning, aesthetic values, and structures, Heusca is the result of a semantic translation and faithful to source language. Translator managed to retain the meaning of source language. In other word, the translation can meet the requirements of fidelity. In terms of expressive values in particular aspects of meter and rhyme are not so faithful to source language because the source language and the target language have some differences. Basically, this is acceptable that the Indonesian language as target language has a very big difference in terms of pronunciation with English as the source language. As a result, meter and rhyme of target language may not be forced into the source language without damaging the naturalness of meaning for target language readers. It can be said that the translation can meet the transparency criteria because source language appeared in poetry comes naturally to native speakers of English.

We study a new problem of cross-lingual transfer learning for event coreference resolution (ECR) where models trained on data from a source language are adapted for evaluations in different target languages. We introduce the first baseline model for this task based on XLM-RoBERTa, a state-of-the-art multilingual pre-trained language model. We also explore language adversarial neural networks (LANN) that present language discriminators to distinguish texts from the source and target languages to improve the language generalization for ECR. In addition, we introduce two novel mechanisms to further enhance the general representation learning of LANN, featuring: (i) multi-view alignment to penalize cross coreference-label alignment of examples in the source and target languages, and (ii) optimal transport to select close examples in the source and target languages to provide better training signals for the language discriminators. Finally, we perform extensive experiments for cross-lingual ECR from English to Spanish and Chinese to demonstrate the effectiveness of the proposed methods.

The ongoing debate on code switching (CS) in multilingual societies and Foreign Language (FL) classes is still relevant and worth considering. This linguistic phenomenon simply refers to altering from one language to another during speech and writing (Richards and Schmidt 2013). While some authorities propose that interactions between learners and instructors in the FL class be carried out only in the target language, other opinions favour a careful blend of both target and source languages. Ghana, a multilingual country in West Africa, adopted English as its official language during the colonial era. French was later introduced to promote regional integration among Ghana and her neighbours, especially since Ghana is greatly surrounded by francophone countries within the subregion (Gandolfi et al. 2021). French has since then been the second most widely taught FL in the country, with a presence in almost all stages of the Ghanaian academic trajectory. This multilingual position of Ghana makes it common to find elements of CS present in FL classrooms, where both instructors and learners occasionally resort to English for instructional conversations (Chachu 2016). This study focuses on university students’ perceptions of CS between English as the source language and French as a foreign language (FFL) during the instruction of the latter. Some frameworks for teaching and learning FFL in Ghana prefer that the language of instruction and interaction be limited to the target language only (Soku and Azanku 2015). The Common European Framework of References for Languages (CEFR) however, suggests the possible use of CS for purposes of instruction when there are favourable conditions (Council of Europe 2001). To identify the roles of CS and investigate FL students’ opinions on this, our data was gathered from classroom observations and questionnaires that had both Likert scale and open-ended questions. This was at the end of their first semester of studying university-level FFL. It was observed that CS plays a significant role in the understanding of linguistic concepts and greatly improves students’ engagement and participation in the FL class. Of the 36 respondents, 34 felt their academic needs were met when the language of instruction and interaction alternated between French and English. Twenty-nine students reported losing focus easily in the FFL class in instances where the medium of interaction was limited to French alone, while 26 reported feeling nervous whenever they were asked to only communicate in French. For metalinguistic aspects of the language, 33 students, constituting 91.7% were of the view that a careful blend of English and French could go a long way to aid their understanding and foster the learning of grammatical concepts. Generally, students attested to better retention of vocabulary in the target language when there was a good blend of both languages. It is recommended therefore that language instructors resort periodically to CS to ensure understanding and engagement from students. CS also helps learners have the opportunity to intervene with their limited linguistic structures when they are faced with linguistic challenges in the FL classroom. Click here to read the corresponding ISSOTL blog post.

We propose a new method to resolve ambiguity in translation and meaning interpretation using linguistic statistics extracted from dual corpora of source and target languages in addition to the logical restrictions described on dictionary and grammar rules for ambiguity resolution. It provides reasonable criteria for determining a suitable equivalent translation or meaning by making the dependency relation in the source language be reflected in the translated text. The method can be tractable because the required statistics can be computed semi-automatically in advance from a source language corpus and a target language corpus, while an ordinal corpus-based translation method needs a large volume of bilingual corpus of strict pairs of a sentence and its translation. Moreover, it also provides the means to compute the linguistic statistics on the pairs of meaning expressions.