CloudLM: a Cloud-based Language Model for Machine Translation

This paper presents example-based machine translation (MT) based on syntactic transfer, which selects the best translation by using models of statistical machine translation. Example-based MT sometimes generates invalid translations because it selects similar examples to the input sentence based only on source language similarity. The method proposed in this paper selects the best translation by using a language model and a translation model in the same manner as statistical MT, and it can improve MT quality over that of 'pure' example-based MT. A feature of this method is that the statistical models are applied after word re-ordering is achieved by syntactic transfer. This implies that MT quality is maintained even when we only apply a lexicon model as the translation model. In addition, translation speed is improved by bottom-up generation, which utilizes the tree structure that is output from the syntactic transfer.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Classifiers in natural language processing (NLP) often have a large number of output classes. For example, neural language models (LMs) and machine translation (MT) models both predict tokens from a vocabulary of thousands. The Softmax output layer of these models typically receives as input a dense feature representation, which has much lower dimensionality than the output. In theory, the result is some words may be impossible to be predicted via argmax, irrespective of input features, and empirically, there is evidence this happens in small language models (Demeter et al., 2020). In this paper we ask whether it can happen in practical large language models and translation models. To do so, we develop algorithms to detect such unargmaxable tokens in public models. We find that 13 out of 150 models do indeed have such tokens; however, they are very infrequent and unlikely to impact model quality. We release our algorithms and code so that others can test their models.

Natural Language Processing (NLP) involves the use of algorithms and models and various computational techniques to analyze, process, and generate natural language data, including speech and text. NLP helps computers interact with humans in a more natural way, which has become increasingly important as more humancomputer interactions take place. NLP allows machines to process and analyze voluminous unstructured data, including social media posts, newspaper articles, reviews from customers, emails, and others. It helps organizations extract insights, automate tasks, and improve decision-making by enabling machines to understand and generate human-like language. A linguistic background is essential for understanding NLP. Linguistic theories and models help in developing NLU systems, as NLP specialists need to understand the structure and rules of language. NLU systems are organized into different components, including language modelling, parsing, and semantic analysis. NLU systems may be assessed through the use of metrics that includes measures like precision and recall, as well as indicators that convey meaningful information that include F1 score and others. Semantics and knowledge representation are central to NLU, as they involve understanding the meaning of words and sentences and representing this information in a way that machines can use. Approaches to knowledge representation include semantic networks, ontologies, and vector embeddings. Language modelling is an essential step in NLP that sees usage in applications like speech recognition, text generation, and text completion and also in areas such as machine translation. Ambiguity Resolution remains a major challenge in NLP, as language is often ambiguous and context-dependent. Some common applications of NLP include sentiment analysis, chatbots, virtual assistants, machine translation, speech recognition, text classification, text summarization, and information extraction. In this chapter, we show the applicability of a popular unsupervised learning technique, viz., clustering through K-Means. The efficiency provided by the K-Means algorithm can be improved through the use of an optimization loop. The prospects for NLP are promising, with an increasing demand for AI-powered language technologies in various industries, including healthcare, finance, and e-commerce. There is also a growing need for ethical and responsible AI systems that are transparent and accountable.

This work aims to present an overview of using some artificial intelligence (AI) models in machine translation (MT). This work aims to integrate machine learning (ML), deep learning (DL), large language models (LLMs) to enhance machine translation between natural languages. The focus is directed to present the neural-based machine translation (NMT), and some DL models are presented. The bidirectional-encoder-representation from transformer (BERT) and LLMs are presented to utilize the big amount of textual data to learn translation patterns. The main measurable criteria that are used to evaluate the performance of MT and Arabic machine translation (AMT) are also presented. Some linguistic and technical challenges of MT and AMT are discussed. Some key points of future works in NMT are mentioned. A comparative study among some recent published related works is presented. A critical survey is presented to show the important role of DL and LLMs in MT. Some open-source toolkits, datasets and some commercial MT systems are collected and briefly presented. This work is expected to be useful for those people interested to know the up-to-date knowledge of MT and the potential role of DL and LLM in automatic translation.

Neural Network Language Models (NNLMs) are a successful approach to Natural Language Processing tasks, such as Machine Translation. We introduce in this work a Statistical Machine Translation (SMT) system which fully integrates NNLMs in the decoding stage, breaking the traditional approach based on [Formula: see text]-best list rescoring. The neural net models (both language models (LMs) and translation models) are fully coupled in the decoding stage, allowing to more strongly influence the translation quality. Computational issues were solved by using a novel idea based on memorization and smoothing of the softmax constants to avoid their computation, which introduces a trade-off between LM quality and computational cost. These ideas were studied in a machine translation task with different combinations of neural networks used both as translation models and as target LMs, comparing phrase-based and [Formula: see text]-gram-based systems, showing that the integrated approach seems more promising for [Formula: see text]-gram-based systems, even with nonfull-quality NNLMs.

Enhancing end-to-end speech translation via multi-stage knowledge distillation.

In the present information age, with the rapid development of computer software and hardware technology and the mature manufacturing system of English-speaking enterprises, it is no longer impossible to use statistics for machine translation. The level and quality of machine translation are expected to meet human expectations. This explains the meaning and realization value of the acquisition of English corpus machine translation, introduces the basic principles of speech recognition technology, and combines the characteristics of the English language on the basis of the original Chinese speech recognition system, adopts the technical means of speech recognition, and leads to in-depth research. In the new era of machine translation acquisition of English corpus, we use the combination of LabVIEW and MATLAB to complete the collection, editing, feature extraction, and speech recognition of speech signals and use VQ pattern matching technology to realize the English recognition of a large number of short vocabulary and individuals. In the application part, we applied the classic LabVIEW technology to the speech recognition technology, which actually realized the idea of “software instead of hardware” and achieved better translation results. Experiments show that the accuracy rate of English machine translation can be as high as 94% when using speech recognition technology. According to the results, it takes about 0.2 seconds to complete machine translation for a 30-second speech, which is basically okay to achieve the effect of real-time translation.

Statistical machine translation systems use a combination of one or more translation models and a language model. While there is a significant body of research addressing the improvement of translation models, the problem of optimizing language models for a specific translation task has not received much attention. Typically, standard word trigram models are used as an out-of-the-box component in a statistical machine translation system. In this paper we apply language modeling techniques that have proved beneficial in automatic speech recognition to the ACL05 machine translation shared data task and demonstrate improvements over a baseline system with a standard language model.

Machine translation is a computer-based translation process that receives a set of words from a particular humanreadable language as input and outputs a second set of words in the intended human-readable language. Machine translation has assisted linguists and sociologists all across the world. A machine translation model may be statistical, rule-based, or based on neural networks. The Neural Machine Translation (NMT) model was introduced in response to the numerous drawbacks of rule- based and statistical-based machine translation models. This study adds a parallel corpus of Assamese and English and develops the NMT system, a transformer model with a self- attention mechanism. For translation from English to Assamese, the system received a higher bilingual evaluation understudy (BLEU) score of 7.6, and for translation from Assamese to English,a BLEU score of 23.2.

Encoder pre-training is promising in end-to-end Speech Translation (ST), given the fact that speech-to-translation data is scarce. But ST encoders are not simple instances of Automatic Speech Recognition (ASR) or Machine Translation (MT) encoders. For example, we find that ASR encoders lack the global context representation, which is necessary for translation, whereas MT encoders are not designed to deal with long but locally attentive acoustic sequences. In this work, we propose a Stacked Acoustic-and-Textual Encoding (SATE) method for speech translation. Our encoder begins with processing the acoustic sequence as usual, but later behaves more like an MT encoder for a global representation of the input sequence. In this way, it is straightforward to incorporate the pre-trained models into the system. Also, we develop an adaptor module to alleviate the representation inconsistency between the pre-trained ASR encoder and MT encoder, and develop a multi-teacher knowledge distillation method to preserve the pre-training knowledge. Experimental results on the LibriSpeech En-Fr and MuST-C En-De ST tasks show that our method achieves state-of-the-art BLEU scores of 18.3 and 25.2. To our knowledge, we are the first to develop an end-to-end ST system that achieves comparable or even better BLEU performance than the cascaded ST counterpart when large-scale ASR and MT data is available.

In natural Language Processing, Machine Translation is still challenging for low resource regional languages like Hindi and Punjabi because of morphology, script variations and Semantic Ambiguity. Previous research on Hindi- Punjabi translation has mainly focused on distinct methodologies like dictionary-based machine translation, rule-based translation, statistical machine translation and pure neural machine translation. This research proposes a cascade Hybrid MT framework for Hindi-Punjabi Translation integrating Dictionary-based direct mapping for known words, Example based machine translation (EBMT) for similar sentence patterns and Neural Machine Translation for complex linguistic sentences. The proposed system optimizes computational efficiency through knowledge-driven pipelines that prioritize low-cost translation methods before invoking expensive neural networks, therefore proposed hybrid machine translation framework is significantly more computationally efficient than pure Neural Machine Translation (NMT) models. Experimental results demonstrate that the hybrid approach achieves superior performance compared to individual methods, with BLEU scores of 42.5 for dictionary matches, 38.2 for EBMT matches, and 35.7 for NMT translations.

This paper addresses challenges of Natural Language Processing (NLP) on non-canonical multilingual data in which two or more languages are mixed. It refers to code-switching which has become more popular in our daily life and therefore obtains an increasing amount of attention from the research community. We report our experience that cov- ers not only core NLP tasks such as normalisation, language identification, language modelling, part-of-speech tagging and dependency parsing but also more downstream ones such as machine translation and automatic speech recognition. We highlight and discuss the key problems for each of the tasks with supporting examples from different language pairs and relevant previous work.

Phrase identification plays an important role in medical English machine translation. However, the phrases in medical English are complicated in internal structure and semantic relationship, which hinders the identification of machine translation and thus affects the accuracy of translation results. With the aim of breaking through the bottleneck of machine translation in medical field, this paper designed a machine translation model based on the optimized generalized likelihood ratio (GLR) algorithm. Specifically, the model in question established a medical phrase corpus of 250,000 English and 280,000 Chinese words, applied the symbol mapping function to the identification of the phrase’s part of speech, and employed the syntactic function of the multioutput analysis table structure to correct the structural ambiguity in the identification of the part of speech, eventually obtaining the final identification result. According to the comprehensive verification, the translation model employing the optimized GLR algorithm was seen to improve the speed, accuracy, and update performance of machine translation and was seen to be more suitable for machine translation in medical field, therefore providing a new perspective for the employment of medical machine translation.

Transformer architectures have achieved state- of-the-art results on a variety of natural language processing (NLP) tasks. However, their attention mechanism comes with a quadratic complexity in sequence lengths, making the computational overhead prohibitive, especially for long sequences. Attention context can be seen as a random-access memory with each token taking a slot. Under this perspective, the memory size grows linearly with the sequence length, and so does the overhead of reading from it. One way to improve the efficiency is to bound the memory size. We show that disparate approaches can be subsumed into one abstraction, attention with bounded-memory control (ABC), and they vary in their organization of the memory. ABC reveals new, unexplored possibilities. First, it connects several efficient attention variants that would otherwise seem apart. Second, this abstraction gives new insights—an established approach (Wang et al., 2020b) previously thought to not be applicable in causal attention, actually is. Last, we present a new instance of ABC, which draws inspiration from existing ABC approaches, but replaces their heuristic memory-organizing functions with a learned, contextualized one. Our experiments on language modeling, machine translation, and masked language model finetuning show that our approach outperforms previous efficient attention models; compared to the strong transformer baselines, it significantly improves the inference time and space efficiency with no or negligible accuracy loss.