Improved spoken language translation using n-best speech recognition hypotheses

In present Statistical Machine Translation (SMT) systems, alignment is trained in a previous stage as the translation model. Consequently, alignment model parameters are not tuned in function of the translation task, but only indirectly. In this paper, we propose a novel framework for discriminative training of alignment models with automated translation metrics as maximization criterion. In this approach, alignments are optimized for the translation task. In addition, no link labels at the word level are needed. This framework is evaluated in terms of automatic translation evaluation metrics, and an improvement of translation quality is observed.

In the modern era, the field of Machine Translation (MT) has seen a significant shift towards Neural Machine Translation (NMT) techniques, which have surpassed traditional Statistical Machine Translation (SMT) models in terms of the quality of translation. Despite this, the efficacy of these techniques may differ based on the language combination in consideration. While SMT is somewhat more flexible in this regard, NMT often needs sizable parallel corpora to attain high translation accuracy. As a result, a benchmark system capable of offering sufficient translation for languages with limited resources, like Nepali, remains a pipe dream. This paper focuses on translating text using statistical and neural MT techniques for the under-resourced English-Nepali language pair. As a part of this system development, we built a parallel corpus of English-Nepali in the tourism domain. We explore the impact of different tokenization techniques on translation outcomes. A substantial analysis is also done for the performance of both approaches using automatic evaluation metrics, BLEU and TER. This paper aims to provide insights into the applicability of SMT and NMT for the under-resourced English-Nepali language pair in light of two popular epitomes of tokenization and to determine the most effective approach for achieving accurate translations.

In this chapter we present a human-based evaluation of surface realisation alternatives. We examine the relative rankings of naturally occurring corpus sentences and automatically generated strings chosen by statistical models (language model, log-linear model), as well as the naturalness of the strings chosen by the log-linear model. We also investigate to what extent preceding context has an effect on choice. We show that native speakers do accept quite some variation in word order, but that there are clearly also factors that make certain realisation alternatives more natural than others. We then examine correlations between native speaker judgements of automatically generated German text and automatic evaluation metrics. We look at a number of metrics from the MT and Summarisation communities and find that for a relative ranking task, most automatic metrics perform equally well and have fairly strong correlations to the human judgements. In contrast, on a naturalness judgement task, the correlation between the human judgements and the automatic metrics was quite weak, the General Text Matcher (GTM) tool providing the only metric that correlates with the human judgements at a statistically significant level.Keywordsgeneration evaluationsurface realisationhuman evaluationGermanhuman judgementsautomatic metricscorrelation

Statistical Machine Translation (SMT) has been the dominant corpus-based machine translation (MT) approach in the last twenty years. While SMT has been studied in detail among European languages, it has not been studied sufficiently in language pairs including Turkish as source or target language, and its study has been limited mostly to English ↔ Turkish language pair. This study aims to broaden the perspective on Turkish corpus-based MT studies by training MT engines between Turkish and a wide variety of languages with different features. It surveys customized SMT between Turkish and 11 different languages. Twenty-two SMT engines have been trained in KantanMT with open parallel corpora using Turkish as both source and target language. Three automatic evaluation metrics F-Measure, BLEU, and TER have been used for evaluating MT quality. Due to the variations in the corpus quality and size, highly varying results have been achieved. While Turkish ↔ Catalan engines have had the highest automatic evaluation scores, Turkish ↔ Arabic engines have had the lowest automatic scores. While the quality results are highly varying across languages, we obtain baseline scores for a wide variety of languages coupled with Turkish. These results may provide a reference point for evaluating future MT systems including Turkish.

Often, the training procedure for statistical machine translation models is based on maximum likelihood or related criteria. A general problem of this approach is that there is only a loose relation to the final translation quality on unseen text. In this paper, we analyze various training criteria which directly optimize translation quality. These training criteria make use of recently proposed automatic evaluation metrics. We describe a new algorithm for efficient training an unsmoothed error count. We show that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.

The background of machine translation dates back to the 1950s when scientists began exploring the use of computers for translation. Motivated by various challenges and needs, such as the high cost of manual translation and cross-cultural communication, machine translation has become a pivotal field. This overview delves into the research background, content, methods, key figures, conclusions, and future prospects of machine translation. It summarizes automatic evaluation metrics, corpus construction, and transfer learning, all of which contribute to enhancing translation performance. Currently, there are three mainstream categories of methods, which include rule-based translation, statistical translation, and neural network-based translation. The rule-based translation method relies on language rules and dictionaries for translation. Statistical machine translation involves the use of extensive bilingual corpora for identification and translation. The conclusion emphasizes the potential of neural machine translation, yet acknowledges challenges in diverse languages, low-resource languages, and specialized terminology.

Patent documents involve many professional fields and include multiple paragraphs, which have brought great difficulties for patent machine translation. This paper presents a hybrid strategy for patent machine translation, which combines the template-based machine translation (TBMT), the semantic rule-based machine translation (RBMT) and the statistics-based machine translation (SMT). The TBMT and RBMT are combined to translate the claim texts in patent documents, and the RBMT and SMT are combined to translate the other texts. In order to find the optimal hybrid strategy, three larger patent fields are selected to train specific models, and then compared with the general model. The experiment results show that the hybrid strategy method outperforms each individual system across a varied set of automatic translation evaluation metrics, verifying the effectiveness of the proposed hybrid strategy method in patent machine translation.

The machine translation output requires evaluation to measure its relevancy to the reference for checking its quality. The evaluation of the output translations can be done manually by human translator, but it requires immense amount of time and labor cost. Therefore, the automatic evaluation metrics has been introduced which is widely used for machine translation evaluation task. In this paper, the translation quality of Digaru-English translation output of 10,500 sentence pair is measured using the automatic evaluation metrics. Further, the translation output of Digaru-English from the Moses SMT system has been evaluated by the BLEU, NIST, METEOR, and TER metric. Further, manual evaluation of Digaru-English has been examined and judged based on fluency and adequacy.KeywordsBLEUNISTTERMETEOREvaluation metrics

Evaluation metrics for video captioning: A survey

We examine correlations between native speaker judgements on automatically generated German text against automatic evaluation metrics. We look at a number of metrics from the MT and Summarisation communities and find that for a relative ranking task, most automatic metrics perform equally well and have fairly strong correlations to the human judgements. In contrast, on a naturalness judgement task, the General Text Matcher (GTM) tool correlates best overall, although in general, correlation between the human judgements and the automatic metrics was quite weak.

In this paper, we explore the generation of face images conditioned on a textual description, as well as the capabilities of the models in editing a machine-generated image on the basis of additional text prompts. We leverage open source state-of-the-art face image generators, StyleGAN models and couple these with the open source multimodal embedding space, CLIP, in an optimisation loop using the method in StyleCLIP to set up our experimental system. We make use of automatic metrics and human ratings to evaluate the results and, in addition, obtain insight into how much automatic metrics are correlated with human ratings. We found compelling evidence that both the text-to-image and editing models based on StyleGAN2 stand out as the better options. In addition, the automatic evaluation metrics are only weakly correlated with human ratings.

In recent years, the rapid advancements of Large Language Models (LLMs) such as ChatGPT and GPT-5 have ushered machine translation into a new era. This study examines the impact of different prompts - simple prompts, complex prompts, and few-shot prompts - on GPT-5s translation performance for the 2024 Chinese Government Work Report, finding that while complex prompts yielded better results in automatic evaluation metrics, human assessment showed no substantial differences in translation quality between simple and complex prompts. The few-shot prompting approach displayed potential in adapting to the text style, but still faced common machine translation challenges, underscoring the importance of thoroughly analyzing text requirements and providing targeted prompt instructions when utilizing large language models for translation, as well as the need for future translators to master the characteristics of these models and develop the ability to identify and adjust translation issues, in order to enhance the practical effectiveness of machine translation.

Machine translation (MT) has become increasingly important and popular in the past decade, leading to the development of MT evaluation metrics aiming at automatically assessing MT output. Most of these metrics use reference translations to compare systems output, therefore, they should not only detect MT errors but also be able to identify correct equivalent expressions so as not to penalize them when those are not displayed in the reference translations. With the aim of improving MT evaluation metrics a study has been carried out of a wide panorama of linguistic features and their implications. For that purpose a Spanish and an English corpora containing hypothesis and reference translations have been analysed from a linguistic point of view, so that common errors can be detected and positive equivalencies highlighted. This article focuses on this qualitative analysis describing the linguistic phenomena that should be considered when developing an automatic MT evaluation metric. The results of this analysis have been used to develop an automatic MT evaluation metric that takes into account different dimensions of language. A brief review of the metric and its evaluation are also provided.

Existing automatic evaluation metrics for open-domain dialogue response generation systems correlate poorly with human evaluation. We focus on evaluating response generation systems via response selection. To evaluate systems properly via response selection, we propose the method to construct response selection test sets with well-chosen false candidates. Specifically, we propose to construct test sets filtering out some types of false candidates: (i) those unrelated to the ground-truth response and (ii) those acceptable as appropriate responses. Through experiments, we demonstrate that evaluating systems via response selection with the test sets developed by our method correlates more strongly with human evaluation, compared with widely used automatic evaluation metrics such as BLEU.

Machine Translation (MT) quality is typically assessed using automatic evaluation metrics such as BLEU and TER. Despite being generally used in the industry for evaluating the usefulness of Translation Memory (TM) matches based on text similarity, fuzzy match values are not as widely used for this purpose in MT evaluation. We designed an experiment to test if this fuzzy score applied to MT output stands up against traditional methods of MT evaluation. The results obtained seem to confirm that this metric performs at least as well as traditional methods for MT evaluation.