Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis

Machine translation (MT) has become increasingly important and popular in the past decade, leading to the development of MT evaluation metrics aiming at automatically assessing MT output. Most of these metrics use reference translations to compare systems output, therefore, they should not only detect MT errors but also be able to identify correct equivalent expressions so as not to penalize them when those are not displayed in the reference translations. With the aim of improving MT evaluation metrics a study has been carried out of a wide panorama of linguistic features and their implications. For that purpose a Spanish and an English corpora containing hypothesis and reference translations have been analysed from a linguistic point of view, so that common errors can be detected and positive equivalencies highlighted. This article focuses on this qualitative analysis describing the linguistic phenomena that should be considered when developing an automatic MT evaluation metric. The results of this analysis have been used to develop an automatic MT evaluation metric that takes into account different dimensions of language. A brief review of the metric and its evaluation are also provided.

Traditional automatic machine translation (MT) evaluation methods adopt the idea that calculates the similarity between machine translation output and human reference translations. However, in terms of the needs of many users, it is a key research issues to propose an evaluation method without references. As described in this paper, we propose a novel automatic MT evaluation method without human reference translations. Firstly, calculate average n-grams probability of source sentence with source language models, and similarly, calculate average n-grams probability of machine-translated sentence with target language models, finally, use the relative error of two average n-grams probabilities to mark machine-translated sentence. The experimental results show that our method can achieve high correlations with a few automatic MT evaluation metrics. The main contribution of this paper is that users can get MT evaluation reliability in the absence of reference translations, which greatly improving the utility of MT evaluation metrics.Keywordsmachine translation evaluationautomatic evaluationwithout reference translations

Automated metrics for machine translation (MT) such as BLEU are customarily used because they are quick to compute and sufficiently valid to be useful in MT assessment. Whereas the instantaneity and reliability of such metrics are made possible by automatic computation based on predetermined algorithms, their validity is primarily dependent on a strong correlation with human assessments. Despite the popularity of such metrics in MT, little research has been conducted to explore their usefulness in the automatic assessment of human translation or interpreting. In the present study, we therefore seek to provide an initial insight into the way MT metrics would function in assessing spoken-language interpreting by human interpreters. Specifically, we selected five representative metrics – BLEU, NIST, METEOR, TER and BERT – to evaluate 56 bidirectional consecutive English–Chinese interpretations produced by 28 student interpreters of varying abilities. We correlated the automated metric scores with the scores assigned by different types of raters using different scoring methods (i.e., multiple assessment scenarios). The major finding is that BLEU, NIST, and METEOR had moderate-to-strong correlations with the human-assigned scores across the assessment scenarios, especially for the English-to-Chinese direction. Finally, we discuss the possibility and caveats of using MT metrics in assessing human interpreting.

Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper that the majority of them tend to underestimate model uncertainty, and as a result, they often produce misleading confidence intervals that do not cover the ground truth. We propose as an alternative the use of conformal prediction, a distribution-free method to obtain confidence intervals with a theoretically established guarantee on coverage. First, we demonstrate that split conformal prediction can “correct” the confidence intervals of previous methods to yield a desired coverage level, and we demonstrate these findings across multiple machine translation evaluation metrics and uncertainty quantification methods. Further, we highlight biases in estimated confidence intervals, reflected in imbalanced coverage for different attributes, such as the language and the quality of translations. We address this by applying conditional conformal prediction techniques to obtain calibration subsets for each data subgroup, leading to equalized coverage. Overall, we show that, provided access to a calibration set, conformal prediction can help identify the most suitable uncertainty quantification methods and adapt the predicted confidence intervals to ensure fairness with respect to different attributes.1

This paper introduces current-day automated metrics for neural machine translation evaluation and discusses two ways of result evaluation, which include evaluation using specific automated metrics as well as evaluation by an expert translator. Neural machine translation systems were tested for analysis and evaluation purposes. Google Translate and DeepL Translate systems, which apply a neural network approach, are presented as candidates. The following metrics were considered for evaluation: METEOR as a traditional reference-based metric, COMET as a neural reference-based metric, and COMET-kiwi as a neural reference-free metric. It can be noted that even if these metrics are automated, there is a huge human impact in this automation. Even models with a neural network approach are trained on data provided by humans, as today it is impossible to get rid of reference translations or quality estimations made by experts. Metrics allow to understand and better investigate machine translation, its features and limitations, as well as to determine the direction of its development. As a part of the analysis, a piece of source text was selected for translation, translated into the target languages using selected neural machine translation systems to get candidate translations, and then a reference translation was specified for each of them. The results of the metrics evaluation were helpful for understanding how close machine translation is to human translation, and also helped to learn more about the current stage of machine translation systems development. Expert evaluation helped to understand how well such systems are doing in terms of translation performance.

Machine Translation (MT) quality is typically assessed using automatic evaluation metrics such as BLEU and TER. Despite being generally used in the industry for evaluating the usefulness of Translation Memory (TM) matches based on text similarity, fuzzy match values are not as widely used for this purpose in MT evaluation. We designed an experiment to test if this fuzzy score applied to MT output stands up against traditional methods of MT evaluation. The results obtained seem to confirm that this metric performs at least as well as traditional methods for MT evaluation.

In recent years, the rapid advancements of Large Language Models (LLMs) such as ChatGPT and GPT-5 have ushered machine translation into a new era. This study examines the impact of different prompts - simple prompts, complex prompts, and few-shot prompts - on GPT-5s translation performance for the 2024 Chinese Government Work Report, finding that while complex prompts yielded better results in automatic evaluation metrics, human assessment showed no substantial differences in translation quality between simple and complex prompts. The few-shot prompting approach displayed potential in adapting to the text style, but still faced common machine translation challenges, underscoring the importance of thoroughly analyzing text requirements and providing targeted prompt instructions when utilizing large language models for translation, as well as the need for future translators to master the characteristics of these models and develop the ability to identify and adjust translation issues, in order to enhance the practical effectiveness of machine translation.

Translating the Holy Quran into different languages is a crucial task. It is a daunting one, given the linguistic complexity and rich cultural context of this holy scripture. The recent advancements in technology, especially in artificial intelligence, brought about a pressing need to explore the potential of machine learning-based tools in easing the translation of religious texts, the Holy Quran in particular. This study explores the potential of adopting ChatGPT, a large language model (LLM) powered by machine learning, in English Quranic translation. The study aims to assess the accuracy, adequacy, and cultural sensitivity of ChatGPT's English Quranic translation and identify potential challenges and limitations of using this tool in this domain. The study employed a comparative analysis approach, evaluating ChatGPT's translation of the El Fatiha Chapter against five human-generated translations using BLEU (Bilingual Evaluation Understudy) and METEOR (Metric for Evaluation of Translation with Explicit ORdering) machine translation metrics. The study also utilized an interpretation of the chapter by El Qurtubi to ensure that ChatGPT has the entire linguistic and cultural context of the chapter. The study's findings indicated that ChatGPT's English translation of El Fatiha achieved a moderate level of accuracy and adequacy with mean BLEU and METEOR scores of 0,48 and 0,78, respectively. However, there were areas where ChatGPT's translation could be improved. The use of ML-based tools such as ChatGPT in English Quranic translation offers promising benefits as it can aid translators in the translation process.

On the role of the UMLS in supporting diagnosis generation proposed by Large Language Models

Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

We introduce WOKIE, an open-source, modular, and ready-to-use pipeline for the automated translation of SKOS thesauri. This work addresses a critical need in the Digital Humanities (DH), where language diversity can limit access, reuse, and semantic interoperability of knowledge resources. WOKIE combines external translation services with targeted refinement using Large Language Models (LLMs), balancing translation quality, scalability, and cost. Designed to run on everyday hardware and be easily extended, the application requires no prior expertise in machine translation or LLMs. We evaluate WOKIE across several DH thesauri in 15 mainly European languages with different parameters, translation services and LLMs, systematically analysing translation quality, performance, and ontology matching improvements. Our results show that WOKIE is suitable to enhance the accessibility, reuse, and cross-lingual interoperability of thesauri by hurdle-free automated translation and improved ontology matching performance, supporting more inclusive and multilingual research infrastructures.

Textual entailment (TE) relation determines whether a text can be inferred from another. Given two texts, one is called the “Text” denoted as T and the other one is called “Hypothesis” denoted as H, the process of textual entailment is to decide whether or not the meaning of H can be logically inferred from the meaning of T. Different semantic, lexical and vector based similarity metrics are used as features for different machine learning classifiers to take the entailment decision in this study. We also considered two machine translation evaluation metrics, namely BLEU and METEOR, as similarity metrics for this task. We carried out the experiments on the datasets released in the shared tasks on textual entailment organized in RTE-1, RTE-2, and RTE-3. We experimented with different feature combinations. Best accuracies were obtained on different feature combinations by different classifiers. The best classification accuracies obtained by our system on the RTE-1, RTE-2 and RTE-3 dataset are 55.91%, 58.88% and 63.38% respectively. MT evaluation metrics based feature alone produced the best classification accuracies of 53.9%, 59.3%, and 62.8% on the RTE-1, RTE-2, and RTE-3 datasets respectively.

Globalization is a trend in the future, and the exchanges among countries in the world in economy, politics, and culture are becoming more and more frequent. In order to overcome the increasingly prominent language barriers, translation has become an indispensable “bridge” for communication between countries. Manual translation has long been unable to meet the growing needs of translation. So, machine translation came into being, especially in recent years, machine translation has developed rapidly. Compared with manual translation, machine translation has unique advantages, that is, fast translation speed and free access. With the gradual improvement of the quality of machine translation, the quality of translation has basically met the requirements of rough reading, which makes machine translation one of the hotspots in recent years. However, to achieve the level of intensive reading, the quality of machine translation needs to be further improved. A thorough study of translation error analysis is an important step to improve the quality of machine translation. At present, there are few studies on error analysis of business English in academic circles, and most of them are just simple classification of translation errors, lacking in in-depth empirical research. This study aims to study the advantages and disadvantages of machine translation in translation practice based on wireless sensor network communication algorithms, analyze the typical errors in machine translation, put forward the idea that posttranslation editing can make up for the shortcomings of machine translation, treat English machine translation more objectively, make full use of machine translation tools, combine machine translation with post-manual editing perfectly, improve translation speed, and guarantee translation quality at the same time.

While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric, MacroF1, and study its applicability to MT evaluation. We find that MacroF1 is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that MacroF1 can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the methods' outputs.

This work aims to present an overview of using some artificial intelligence (AI) models in machine translation (MT). This work aims to integrate machine learning (ML), deep learning (DL), large language models (LLMs) to enhance machine translation between natural languages. The focus is directed to present the neural-based machine translation (NMT), and some DL models are presented. The bidirectional-encoder-representation from transformer (BERT) and LLMs are presented to utilize the big amount of textual data to learn translation patterns. The main measurable criteria that are used to evaluate the performance of MT and Arabic machine translation (AMT) are also presented. Some linguistic and technical challenges of MT and AMT are discussed. Some key points of future works in NMT are mentioned. A comparative study among some recent published related works is presented. A critical survey is presented to show the important role of DL and LLMs in MT. Some open-source toolkits, datasets and some commercial MT systems are collected and briefly presented. This work is expected to be useful for those people interested to know the up-to-date knowledge of MT and the potential role of DL and LLM in automatic translation.