Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework

There have been many studies on human-machine dialogue systems. To evaluate them accurately and fairly, many resort to human grading of system outputs. Unfortunately, this is time-consuming and expensive. The study of AM-FM (Adequacy Metric - Fluency Metric) suggests an automatic evaluation metric, that achieves good performance in terms of correlation with human judgements. AM-FM framework intends to measure the quality of dialogue generation along two dimensions with the help of gold references: (1) The semantic closeness of generated response to the corresponding gold references; (2) The syntactic quality of the sentence construction. However, the original formulation of both adequacy and fluency metrics face some technical limitations. The latent semantic indexing (LSI) approach to AM modeling is not scalable to large amount of data. The bag-of-words representation of sentences fails to capture the contextual information. As for FM modeling, the n-gram language model implementation is not able to capture long-term dependency. Many deep learning approaches, such as the long short-term memory network (LSTM) or transformer-based architectures, are able to address these issues well by providing better contextual-aware sentence representations than the LSI approach and achieving much lower perplexity on benchmarking datasets as compared to the n-gram language model. In this paper, we propose deep AM-FM, a DNN-based implementation of the framework and demonstrate that it achieves promising improvements in both Pearson and Spearman correlation w.r.t human evaluation on the bench-marking DSTC6 End-to-end Conversation Modeling task as compared to its original implementation and other popular automatic metrics.

Despite the recent success of automatic metrics for assessing translation quality, their application in evaluating the quality of machine-translated chats has been limited. Unlike more structured texts like news, chat conversations are often unstructured, short, and heavily reliant on contextual information. This poses questions about the reliability of existing sentence-level metrics in this domain as well as the role of context in assessing the translation quality. Motivated by this, we conduct a meta-evaluation of existing automatic metrics, primarily designed for structured domains such as news, to assess the quality of machine-translated chats. We find that reference-free metrics lag behind reference-based ones, especially when evaluating translation quality in out-of-English settings. We then investigate how incorporating conversational contextual information in these metrics for sentence-level evaluation affects their performance. Our findings show that augmenting neural learned metrics with contextual information helps improve correlation with human judgments in the reference-free scenario and when evaluating translations in out-of-English settings. Finally, we propose a new evaluation metric, Context-MQM, that utilizes bilingual context with a large language model (LLM) and further validate that adding context helps even for LLM-based evaluation metrics.

Standard automatic metrics, e.g. BLEU, are not reliable for document-level MT evaluation. They can neither distinguish document-level improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations. This paper introduces a novel automatic metric BlonDe to widen the scope of automatic MT evaluation from sentence to document level. BlonDe takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. We conduct extensive comparisons on a newly constructed dataset BWB. The experimental results show that BlonDe possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances. In a large-scale human study, BlonDe also achieves significantly higher Pearson’s r correlation with human judgments compared to previous metrics.

Estimating translation quality is a problem of growing importance as it has many potential applications. The quality of translation from Arabic to English is especially difficult to evaluate due to the languages being distant languages: different in syntax and low in lexical similarity. We propose a feature-based framework for estimating the quality of Arabic to English translations at the sentence level. The proposed method works without reference translations, considers both fluency and adequacy of translations, and does not imply assumptions on the source of translation (humans, machines, or post-edited machine translations); thus, making the solution applicable to increasingly more situations. This research solves the translation quality estimation problem by treating it as a supervised machine learning problem. The proposed model utilizes regression algorithms (SVR and Linear Regression) to predict quality scores of unseen translated texts at runtime. This is accomplished by training models on a labeled parallel corpus and mapping extracted features to the quality label. The prediction models succeeded in predicting fluency and adequacy of translations with a Mean Absolute Error of 0.84 and 1.02, respectively. Furthermore, we show that in a similar setting of our approach, fluency of an Arabic to English translated sentence on its own, is an appropriate indication of a translation’s overall quality.

Translation Environment Tools make translators’ work easier by providing them with term lists, translation memories and machine translation output. Ideally, such tools automatically predict whether it is more effortful to post-edit than to translate from scratch, and determine whether or not to provide translators with machine translation output. Current machine translation quality estimation systems heavily rely on automatic metrics, even though they do not accurately capture actual post-editing effort. In addition, these systems do not take translator experience into account, even though novices’ translation processes are different from those of professional translators. In this paper, we report on the impact of machine translation errors on various types of post-editing effort indicators, for professional translators as well as student translators. We compare the impact of MT quality on a product effort indicator (HTER) with that on various process effort indicators. The translation and post-editing process of student translators and professional translators was logged with a combination of keystroke logging and eye-tracking, and the MT output was analyzed with a fine-grained translation quality assessment approach. We find that most post-editing effort indicators (product as well as process) are influenced by machine translation quality, but that different error types affect different post-editing effort indicators, confirming that a more fine-grained MT quality analysis is needed to correctly estimate actual post-editing effort. Coherence, meaning shifts, and structural issues are shown to be good indicators of post-editing effort. The additional impact of experience on these interactions between MT quality and post-editing effort is smaller than expected.

This thesis focuses on the rigid-body registration of a three-dimensional model of an object to a set of its two-dimensional projections. The main contribution is the development of two registration algorithms that use a continuous model of the volume based on splines, either in the space domain or in the frequency domain. This allows for a well-defined gradient of the dissimilarity measure, which is a necessary condition for efficient and accurate registration. The first part of the thesis contains a review of the literature on volume-to- image registration. Then, we discuss data interpolation in the space domain and in the frequency domain. The basic concepts of our registration strategy are given in the second part of the thesis. We present a novel one-step approach for fast ray casting to simulate space-based volume projections. We also discuss the use of the central-slice theorem to simulate frequency-based volume projections. Then, we consider the question of the registration robustness. To improve the robustness of the space-based approach, we apply a multiresolution optimization strategy where spline-based data pyramids are processed in coarse-to-fine fashion, which improves speed as well. To improve the robustness of the frequency-based registration, we apply a coarse-to-fine strategy that involves weights in the frequency domain. In the third part, we apply our space-based algorithm to computer-assisted orthopedic surgery while adapting it to the perspective projection model. We show that the registration accuracy achieved using the orthopedic data is consistent with the current standards. Then, we apply our frequency-based registration to three-dimensional electron-microscopy application. We show that our algorithm can be used to obtain a refined solution with respect to currently available algorithms. The novelty of our approach is in dealing with a continuous space of geometric parameters, contrary to the standard methods which deal with quantized parameters. We conclude that our continuous parameter space leads to better registration accuracy. Last, we compare the performance of the frequency-based algorithm with that of the space-based algorithm in the context of electron microscopy. With these data, we observe that frequency-based registration algorithm outperforms the space-based one, which we attribute to the suitability of interpolation in the frequency domain when dealing with strictly space-limited data.

Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task.

In view of the fact that existing machine translation error correction tools used in English teaching do not pay enough attention to differences in translation quality, leading to problems such as over-correction or under-correction, this paper proposes an adaptive error correction system based on machine translation evaluation for intelligent auxiliary English teaching. The system builds a “quality perception-dynamic error correction” closed-loop mechanism, integrating a machine translation quality evaluation module, an error correction module, and an adaptive adjustment module. The quality evaluation module identifies error probabilities at the word level and overall quality scores at the sentence level in students' English translations. The error correction module, drawing on advanced language processing technologies, adjusts its focus according to quality information to target errors more precisely. The adaptive adjustment module optimizes its performance based on feedback from correction results. Experiments were conducted on English learning datasets, including students' compositions, translation exercises, and daily communication dialogues. The results show that the proposed system performs significantly better in correcting English errors than traditional tools: on standard English-Chinese translation practice materials, its correction accuracy is 2.8 % higher than common QE-guided correction tools, and the rate of unnecessary changes is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{2. 2 \%}$</tex> lower, in low-resource scenarios such as colloquial English dialogues, its accuracy is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{2. 7 \%}$</tex> higher than comparative tools. Control experiments show that removing the adaptive adjustment module leads to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{2}. \mathrm{3} \%$</tex> drop in correction accuracy. The system can accurately handle various English errors such as grammar, semantics, and terminology use. While ensuring correction accuracy, it processes long sentences 8.3 % faster than common large-scale correction tools, balancing accuracy and efficiency, making it suitable for multiple English teaching scenarios such as writing, translation, and oral practice.

We introduce WOKIE, an open-source, modular, and ready-to-use pipeline for the automated translation of SKOS thesauri. This work addresses a critical need in the Digital Humanities (DH), where language diversity can limit access, reuse, and semantic interoperability of knowledge resources. WOKIE combines external translation services with targeted refinement using Large Language Models (LLMs), balancing translation quality, scalability, and cost. Designed to run on everyday hardware and be easily extended, the application requires no prior expertise in machine translation or LLMs. We evaluate WOKIE across several DH thesauri in 15 mainly European languages with different parameters, translation services and LLMs, systematically analysing translation quality, performance, and ontology matching improvements. Our results show that WOKIE is suitable to enhance the accessibility, reuse, and cross-lingual interoperability of thesauri by hurdle-free automated translation and improved ontology matching performance, supporting more inclusive and multilingual research infrastructures.

So far the majority of Machine Translation (MT) research has focused on translation at the level of individual sentences.For sentence level translation, Machine Translation has addressed various divergence issues for large variety of languages; the issue of pronominal divergence has been presented only recently.Since the quality of translation as required by users follows coherent multi-sentence discourse structure in a specific context, the pronominal divergence helps us in understanding the nuances of translation arising out of disparity in the languages.Subsequently using clues from this divergence, the anaphora resolution system can find the correct interpretation for the given pronominal referents and other entities by resolving the inter-sentential context.In the literature, researchers have examined the issue and have proposed ways for their classification and resolution of anaphora.However for Indic languages, not many studies are available.In this paper, we discuss different aspects of pronominal divergence that affects the anaphora resolution in English Hindi Machine Translation (EHMT).The study shall be helpful in developing approaches that can explicitly use inter-sentential information in order to resolve specific types of ambiguity and which can generate coherent multi-sentence discourse structure in the target language to produce higher quality of translation Machine Translation.

In this paper, we define and assess a reference-based metric to evaluate the accuracy of pronoun translation (APT). The metric automatically aligns a candidate and a reference translation using GIZA++ augmented with specific heuristics, and then counts the number of identical or different pronouns, with provision for legitimate variations and omitted pronouns. All counts are then combined into one score. The metric is applied to the results of seven systems (including the baseline) that participated in the DiscoMT 2015 shared task on pronoun translation from English to French. The APT metric reaches around 0.993-0.999 Pearson correlation with human judges (depending on the parameters of APT), while other automatic metrics such as BLEU, METEOR, or those specific to pronouns used at DiscoMT 2015 reach only 0.972-0.986 Pearson correlation.

We intended to demonstrate the effect of using N -best speech recognition hypotheses for improving speech translation performance. A log-linear model, which integrated features from speech recognition and statistical machine translation, was used to rescore the translation candidates. Model parameters were estimated by optimizing an objectively measurable but subjectively relevant translation quality metric. Experimental results have shown that the proposed N -best approach improved translation quality over the conventional single-best approach. The improvements were confirmed consistently by several automatic translation evaluation metrics.

In recent years, the rapid advancements of Large Language Models (LLMs) such as ChatGPT and GPT-5 have ushered machine translation into a new era. This study examines the impact of different prompts - simple prompts, complex prompts, and few-shot prompts - on GPT-5s translation performance for the 2024 Chinese Government Work Report, finding that while complex prompts yielded better results in automatic evaluation metrics, human assessment showed no substantial differences in translation quality between simple and complex prompts. The few-shot prompting approach displayed potential in adapting to the text style, but still faced common machine translation challenges, underscoring the importance of thoroughly analyzing text requirements and providing targeted prompt instructions when utilizing large language models for translation, as well as the need for future translators to master the characteristics of these models and develop the ability to identify and adjust translation issues, in order to enhance the practical effectiveness of machine translation.

Often, the training procedure for statistical machine translation models is based on maximum likelihood or related criteria. A general problem of this approach is that there is only a loose relation to the final translation quality on unseen text. In this paper, we analyze various training criteria which directly optimize translation quality. These training criteria make use of recently proposed automatic evaluation metrics. We describe a new algorithm for efficient training an unsmoothed error count. We show that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.

The aim of the research is twofold: to evaluate the translation quality of the individual sentences of the MT output and also post-edited MT output on the basis of metrics of automatic MT evaluation from Slovak into the German language; and to compare the quality of MT output and post-edited MT output based on the same automatic metrics of MT evaluation. The icon graphs were used to visualize the results for individual sentences. A significant difference was found in sentence 36 in favor of the post-edited MT output and vice versa in sentence 5 in favor of MT output. Due to the error rate, a significant difference was in sentence 29 and 11 in favor of post-edited MT output and vice versa the sentence 26 in favor of MT output. Based on our results we can state that it is necessary to include into the evaluation of the quality of translation all automatic metrics for each sentence separately.