CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation

Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task.

Multi-modal simultaneous machine translation fusion of image information

To achieve high-quality translation with low latency, a Simultaneous Speech Translation (SimulST) system relies on a policy module to decide whether to translate immediately or wait for additional streaming input, along with a translation model capable of effectively handling partial speech input. Prior research has tackled these components separately, either using ``wait-k'' policies based on fixed-length segments or detected word boundaries, or dynamic policies based on different strategies (e.g., meaningful units), while employing offline models for prefix-to-prefix translation. In this paper, we propose Divergence-Guided Simultaneous Speech Translation (DiG-SST), a tightly integrated approach focusing on both translation quality and latency for streaming input. Specifically, we introduce a simple yet effective prefix-based strategy for training translation models with partial speech input, and develop an adaptive policy that makes read/write decisions for the translation model based on the expected divergence in translation distributions resulting from future input. Our experiments on multiple translation directions of the MuST-C benchmark demonstrate that our approach achieves a better trade-off between translation quality and latency compared to existing methods.

Simultaneous speech translation aims to maintain translation quality while minimizing the delay between reading input and incrementally producing the output. We propose a new general-purpose prediction action which predicts future words in the input to improve quality and minimize delay in simultaneous translation. We train this agent using reinforcement learning with a novel reward function. Our agent with prediction has better translation quality and less delay compared to an agent-based simultaneous translation system without prediction.

Speech translation systems commonly couple automatic speech recognition (ASR) and machine translation (MT) components. Hereby the automatic segmentation of the ASR output for the subsequent MT is critical for the overall performance. In simultaneous translation systems, which require a continuous output with a low latency, chunking of the ASR output into translatable segments is even more critical. This paper addresses the question how utterance chunking influences machine translation performance in an empirical study. In addition, the machine translation performance is also set in relation to the segment length produced by the chunking strategy, which is important for simultaneous translation. Therefore, we compare different relatively simple chunking/ segmentation strategies on speech recognition hypotheses as well as on reference transcripts.

Design of Translation Error Correction System Based on Improved Seq2Seq

As speech translation (ST) systems become increasingly prevalent, understanding their vulnerabilities is crucial for ensuring robust and reliable communication. However, limited work has explored this issue in depth. This paper explores methods of compromising these systems through imperceptible audio manipulations. Specifically, we present two approaches: (1) adapting perturbation-based techniques used for automatic speech recognition (ASR) attacks to the ST context, making our work the first to apply this approach to ST, and (2) proposing a novel music generation-based method to guide targeted translation, while also conducting more practical over-the-air attacks in the physical world. Our experiments reveal that carefully crafted audio perturbations can mislead translation models to produce targeted, harmful outputs, while adversarial music achieve this goal more covertly, exploiting the natural imperceptibility of music. These attacks have proven effective across multiple languages and translation models, highlighting a systemic vulnerability in current ST architectures. Beyond immediate security concerns, our findings highlight broader challenges in the robustness and interpretability of neural speech systems.

In the rapid development of information technology and network technology, the cultural knowledge of human society blends with each other, and people have more and more ways to obtain information. Faced with the continuous growth and various types of information composed of different languages, researchers have proposed to use machine translation technology to build auxiliary translation system, which can play an important role in daily learning and life work. After understanding the research and application status of speech translation system in the era of big data, this paper mainly explores the architecture of multi-user and multi-language collaborative speech translation system according to the application characteristics of multi-language assisted translation system in the new era, and puts forward practical application methods from the perspective of long-term development of the industry, so as to let the world economy and culture communicate with each other. Fully demonstrate the translation efficiency and application quality of the design system.

Timing and Specificity of Feed-Forward Inhibition within the LGN

In the typical speech translation system, the first-best speech recognizer hypothesis is segmented into sentence-like units which are then fed to the downstream machine translation component. The need for a sufficiently large context in this intermediate step and for the MT introduces delays which are undesirable in many application scenarios, such as real-time subtitling of foreign language broadcasts or simultaneous translation of speeches and lectures. In this paper, we propose a statistical machine translation decoder which processes a continuous input stream, such as that produced by a run-on speech recognizer. By decoupling decisions about the timing of translation output generation from any fixed input segmentation, this design can guarantee a maximum output lag for each input word while allowing for full word reordering within this time window. Experimental results show that this system achieves competitive translation performance with a minimum of translationinduced latency.

We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-language text). Thus, the speech translation model can learn from both unlabeled and labeled data, especially when the source-language text data is abundant. Beyond this, we present a denoising method to build a robust text encoder that can deal with both normal and noisy text data. Our system sets new state-of-the-arts on the MuST-C En-De, En-Fr, and LibriSpeech En-Fr tasks.

Today, we have entered an era of big data. Under the background of global economization, communication in all walks of life is becoming more and more frequent and cross-language communication is inevitable. Cross-language communication is difficult for many people. The online translation system can greatly reduce the communication barriers between people of different languages. As an efficient tool, the translation system can realize the translation of different languages under the conditions of retaining the original semantics equivalent conversion. The article adopts the Internet of Things technology and big data model to build an English intelligent translation system, which can realize intelligent translation between multiple languages and English. The research results of the article show the following: (1) For samples with high semantic feature values, the correlation coefficient and similarity coefficient will be higher. Therefore, it can be concluded that for different semantics, the similarity is generally positively correlated with its feature value and correlation coefficient. The translation speed of the system proposed in the article is the fastest among the three translation systems. When the number of sentences is 10,000, the translation speed of the translation system proposed in the article is 5.89 seconds, the translation speed of the network multilingual translation system is 6.74 seconds, and the translation speed of the traditional translation system is 10.53 seconds. (2) The translation accuracy of the big data intelligent translation model proposed in the article is the highest among the three models. The translation accuracy of simple sentences can reach 99%. The translation accuracy of general sentences is 98%, and the translation accuracy of complex sentences is 95%. The BLEU value of the method in this paper is basically the same as that of the RNN cyclic neural network translation model. When translating general sentences, the BLEU value of the method in this paper is slightly higher than that of the RNN cyclic neural network translation model; especially when translating complex sentences by machine, the BLEU value of the method in this paper is far higher than that of the RNN translation model. (3) The average response time will increase with the increase in the number of tests, and the success rate generally remains above 98%, close to 100%, indicating that the response time of the system operation is normal. The number of designed test cases for the data processing module is 90, the number of executed test cases is 90, and the execution rate can reach 100%. Normal operation means that in the process of operation, no fault occurs. In the system load test, the load of serial number 1 is normal, the average delay is 38 seconds, the average delay of serial number 2 is 48 seconds, the average delay of serial number 3 is 59 seconds, the average delay of serial number 4 is 62 seconds, and the average delay of serial number 5 is 47 seconds. The delay of data packets under all kinds of loads can meet the standard requirements.

Recent studies on end-to-end speech translation(ST) have facilitated the exploration of multilingual end-to-end ST and end-to-end simultaneous ST. In this paper, we investigate end-to-end simultaneous speech translation in a one-to-many multilingual setting which is closer to applications in real scenarios. We explore a separate decoder architecture and a unified architecture for joint synchronous training in this scenario. To further explore knowledge transfer across languages, we propose an asynchronous training strategy on the proposed unified decoder architecture. A multi-way aligned multilingual end-to-end ST dataset was curated as a benchmark testbed to evaluate our methods. Experimental results demonstrate the effectiveness of our models on the collected dataset. Our codes and data are available at: https://github.com/XiaoMi/TED-MMST.

Since Graham Bell first invented the telephone in 1876, it has become an indispensable means for communications. We can easily communicate with others domestically as well as internationally. However, another great barrier has not been overcome yet; communications between people speak- ing different languages. An interpreting telephone system, or a speech translation system, will solve this problem which has been annoying human-being from the beginning of their history. The first effort was made by

Nowadays automatic speech recognition (ASR) is an important task for machines. Several applications such as speech translation, virtual assistants and voice bot systems use ASR to understand human speech. Most of the research and available models are for widely used languages, such as English, German, French, Chinese and Spanish. This paper presents the Armenian speech recognition system. As a result of this research developed acoustic and language models for the Armenian language (modern ASR systems combine acoustic and language models to achieve higher accuracy). RNN-based Baidu’s Deep Speech deep neural network was used to train the acoustic model, and the KenLM toolkit was used to train the probabilistic language model. The acoustic model was trained and validated on ArmSpeech Armenian native speech corpus using transfer-learning and data augmentation techniques and tested on the Common Voice Armenian database. The language model was built based on the texts scraped from Armenian news websites. Final models are small in size and can be run and do real-time speech-to-text tasks on IoT devices. Testing on the Common Voice Armenian database the model gave 0.902565 WER and 0.305321 CER without the language model, and 0.552975 WER and 0.285904 CER with the language model. The paper aims to describe environment setup, data collection, acoustic and language models training processes, as well as final results and benchmarks.