Real Context Model for Tone Recognition in Mandarin Conversational Telephone Speech

Research in the speech recognition speech-to-text conversion area has been underway for a couple of decades, and a great deal of progress has been made in reducing the word error rate. In this paper, we attempt to summarize the state of the art in speech recognition algorithms. The algorithms we describe span the areas of lexicon design, feature extraction, classifier design, combination of hypotheses, and speaker adaptation of acoustic models. We will benchmark the algorithms on two main sources of speech, the first being Voicemail (conversational telephone speech from a single speaker) and the second being Switchboard (conversational telephone speech between two speakers). We also present the results of some cross-domain experiments which highlight the brittleness of speech recognition systems today and illustrates the need to focus research effort on improving cross-domain performance.

We report insights from translating Spanish conversational telephone speech into English text by cascading an automatic speech recognition (ASR) system with a statistical machine translation (SMT) system. The key new insight is that the informal register of conversational speech is a greater challenge for ASR than for SMT: the BLEU score for translating the reference transcript is 64%, but drops to 32% for translating automatic transcripts, whose word error rate (WER) is 40%. Several strategies are examined to mitigate the impact of ASR errors on the SMT output: (i) providing the ASR lattice, instead of the 1-best output, as input to the SMT system, (ii) training the SMT system on Spanish ASR output paired with English text, instead of Spanish reference transcripts, and (iii) improving the core ASR system. Each leads to consistent and complementary improvements in the SMT output. Compared to translating the 1-best output of an ASR system with 40% WER using an SMT system trained on Spanish reference transcripts, translating the output lattice of a better ASR system with 35% WER using an SMT system trained on ASR output improves BLEU from 32% to 38%.

This paper presents two different approaches utilizing sta-tisticallanguagemodel(SLM)andsupportvectormachines(SVM) for sentence boundary detection of spontaneousJapanese. In the SLM-based approach, linguistic likeli-hoods and occurrence of pause are used to determine sen-tence boundaries. To suppress false alarms, heuristic pat-terns of end-of-sentence expressions are also incorporated.On the other hand, SVM is adopted to realize robust classi-fication against a wide variety of expressions and speechrecognition errors. Detection is performed by an SVM-based text chunker using lexical and pause information asfeatures. We evaluated these approaches on manual and au-tomatic transcription of spontaneous lectures and speeches,and achieved F-measures of 0.85 and 0.78, respectively. Index Terms : sentence boundary detection, spontaneousspeech, statistical language model, support vector ma-chines. 1. Introduction Recent advance of automatic speech recognition (ASR)technology, especially for spontaneous speech, enables var-ious applications such as spoken document archiving andretrieval, speech summarization and speech translation. Toorganizeaspokendocumentinastructuredformandtogiveuseful indices, transcriptions should be segmented into ap-propriate units like sentences. Moreover, these applicationsare usually built by combining an ASR system with naturallanguage processing (NLP) systems such as a parser and amachine translator, which often assume that input text is asentence. However,sentencesinspontaneousspeechareill-formed, and sentence boundaries are indistinct. Output textby ASR systems is just a sequence of words and has no ex-plicitsentenceboundaries, sothefurtherstepofsegmentingthe ASR output is required for these applications.Automatic boundary detection of spoken sentences hasbeen explored mainly on broadcast news (BN)tasks[1,2,3]and conversational telephone speech (CTS) tasks[3, 4, 5] inEnglish. As features for detection, pause, prosodic and lin-guistic information is often used. Most popular approach isa combination of prosodic and linguistic information[2, 3],which realizes high performance on BN and CTS tasks.Prosody-based approaches[1, 5] have also been investi-gated. Meanwhile, linguistic information is not used by it-self, since most of these works were performed on Englishdata, where cue words or expressions of sentence bound-aries are not easily defined.In Japanese, cue expressions are typically observed atthe end of sentences and expected to be useful for bound-ary detection. However, variety of such expressions is solarge in spoken Japanese, that it is hard to collect sufficientamountofdatafortrainingstatisticalmodelssuchasamax-imum entropy (ME) model. Moreover, many of cue expres-sions consist of particles, which are apparently difficult tobedetectedinASR.Thus,robustnessforASRerrorsshouldalso be investigated.In this paper, we address two approaches of sen-tence boundary detection for spontaneous Japanese. Asframeworks of detection, we adopt and compare statisti-cal language model (SLM) and support vector machines(SVM). The proposed approaches are evaluated with reallectures and speeches included in the Corpus of Sponta-neous Japanese (CSJ).

Recognition of Hungarian conversational telephone speech is challenging due to the informal style and morphological richness of the language. Recurrent Neural Network Language Model (RNNLM) can provide remedy for the high perplexity of the task; however, two-pass decoding introduces a considerable processing delay. In order to eliminate this delay we investigate approaches aiming at the complexity reduction of RNNLM, while preserving its accuracy. We compare the performance of conventional back-off n-gram language models (BNLM), BNLM approximation of RNNLMs (RNN-BNLM) and RNN n-grams in terms of perplexity and word error rate (WER). Morphological richness is often addressed by using statistically derived subwords - morphs - in the language models, hence our investigations are extended to morph-based models, as well. We found that using RNN-BNLMs 40% of the RNNLM perplexity reduction can be recovered, which is roughly equal to the performance of a RNN 4-gram model. Combining morph-based modeling and approximation of RNNLM, we were able to achieve 8% relative WER reduction and preserve real-time operation of our conversational telephone speech recognition system.

The paper describes the use of several advanced acoustic modeling techniques for the 2004 CU-HTK large vocabulary speech recognition systems. These techniques include Gaussianization for speaker normalization, discriminative cluster adaptive training (CAT), subspace for precision and mean (SPAM) modeling of inverse covariances, and discriminative complexity control. Acoustic models featuring these techniques were integrated into a state-of-the-art 10 real-time multi-pass system with sophisticated adaptation for performance evaluation. Experimental results are presented on both broadcast news (BN) and conversational telephone speech (CTS) transcription tasks.

Keyword spotting becomes a very important branch of speech recognition. But the acoustic mismatch between training and testing environments often causes a severe degradation in the recognition performance. This paper presents an improved keyword spotting strategy. A fuzzy search algorithm is proposed to extract keyword hypotheses from a syllable confusion network (SCN). SCN is linear and naturally suitable for indexing. To accelerate search process, SCN are pruned to feasible sizes. As a post-processing method, minimum classification error (MCE) optimized confidence measure is adopted to reject false accepts. On Mandarin conversational telephone speech (CTS), the proposed algorithms reduce the equal error rate (EER) by 7.2% relative.

Maximum a posteriori (MAP) adaptation and its discriminative variants, such as MMI-MAP (maximum mutual information MAP) and MPE-MAP (minimum phone error MAP), have been widely applied to acoustic model adaptation. This paper introduces a new adaptation approach, fMPE-MAP, which is an extension to the original fMPE (feature minimum phone error) algorithm, with the enhanced ability in porting Gaussian models and fMPE transforms to a new domain. We applied this approach to the SRI-ICSI 2007 NIST meeting recognition system, for which we ported our conversational telephone speech (CTS) and broadcast news (BN) models to the meeting domain. Experiments showed that the proposed fMPE-MAP approach has comparable or better performance than simply training the fMPE transform on combined data, in addition to the obvious speed advantage. In combination with MPE-MAP, we obtained about 20% relative word error rate reduction on a lecture meeting evaluation test set, over the models trained with the standard MAP approach.

Probabilistic linear discriminant analysis (PLDA) is widely described as an effective model for text-independent speaker verification in the i-vector space. The PLDA scoring function is typically formulated as the likelihood ratio between the speaker-adapted and the universal PLDAs. In this case, the adaptation of PLDA was performed through the speaker factors. In this paper, we show that the channel factors of the PLDA could be equivalently exploited to deal with the multi-source conditions. In speaker verification, with the proposed method, a PLDAmodel trained on conversational telephone speech could be adequately adapted for interview-style microphone recordings. Experimental results on NIST SRE'08 and SRE'10 datasets confirm that the proposed method is effective, especially for the case whereby enrollment and test utterances were captured from different sources.

Phonotactic modelling, typically in the form of a PPRLM system, forms a key component in state-of-the-art Language Identification (LID) systems. Given the objective of PPRLM systems is to capture as accurately as possible the phonotactics which characterise a language, it is assumed that the minimisation of Phone Error Rate (PER) is a precursor to achieving this effectively. In this paper we examine the relevance of PER as a metric for determining eventual LID performance. In order to conduct this investigation we make use of the CallHome corpus, based on the premise it provides a better representation for the style of discourse and channel conditions encountered in the Conversational Telephone Speech (CTS), which is now the focus of current NIST LID evaluations. Using CallHome instead of the OGI-MLTS corpus to train phone recognisers, we obtained significantly improved results, with an average improvement of approximately 6% absolute across the 30, 10 and 3 seconds tasks for the NIST 1996 and 2003 evaluations. We also examine the impact of tuning the individual front-end recognisers, on both the resultant PER of other languages and against the resultant LID performance. We find that PER has a number of limitations in indicating both the degree and direction of changes to LID performance. Accordingly, we propose a new metric which is better suited for forecasting the impact on LID performance when the phone recogniser front-end is modified.

We propose a method for zero-resource domain adaptation of DNN acoustic models, for use in low-resource situations where the only in-language training data available may be poorly matched to the intended target domain. Our method uses a multi-lingual model in which several DNN layers are shared between languages. This architecture enables domain adaptation transforms learned for one well-resourced language to be applied to an entirely different low- resource language. First, to develop the technique we use English as a well-resourced language and take Spanish to mimic a low-resource language. Experiments in domain adaptation between the conversational telephone speech (CTS) domain and broadcast news (BN) domain demonstrate a 29% relative WER improvement on Spanish BN test data by using only English adaptation data. Second, we demonstrate the effectiveness of the method for low-resource languages with a poor match to the well-resourced language. Even in this scenario, the proposed method achieves relative WER improvements of 18-27% by using solely English data for domain adaptation. Compared to other related approaches based on multi-task and multi-condition training, the proposed method is able to better exploit well-resource language data for improved acoustic modelling of the low-resource target domain.

This paper reports on a simplified system for determining vocal tract normalization. Such normalization has led to significant gains in recognition accuracy by reducing variability among speakers and allowing the pooling of training data and the construction of sharper models. But standard methods for determining the warp scale have been extremely cumbersome, generally requiring multiple recognition passes. We present a new system for warp scale selection which uses a simple generic voiced speech model to rapidly select appropriate frequency scales. The selection is sufficiently streamlined that it can moved completely into the front-end processing. Using this system on a standard test of the Switchboard Corpus, we have achieved relative reductions in word error rates of 12% over unnormalized gender-independent models and 6% over our best unnormalized gender-dependent models.

DIHARD III was the third in a series of speaker diarization challenges intended to improve the robustness of diarization systems to variability in recording equipment, noise conditions, and conversational domain. Speaker diarization was evaluated under two speech activity conditions (diarization from a reference speech activity vs. diarization from scratch) and 11 diverse domains. The domains span a range of recording conditions and interaction types, including read audio-books, meeting speech, clinical interviews, web videos, and, for the first time, conversational telephone speech. A total of 30 organizations (forming 21teams) from industry and academia submitted 499 valid system outputs. The evaluation results indicate that speaker diarization has improved markedly since DIHARD I, particularly for two-party interactions, but that for many domains (e.g., web video) the problem remains far from solved.

Cascaded model adaptation for dialog act segmentation and tagging

Word usage is influenced by diverse factors, including topic, genre and various speaker/author characteristics. To characterize these aspects of language, we introduce the “Multi-Factor Sparse Plus Low Rank” exponential language model, which allows supervised joint training of arbitrary overlapping factor-specific model components. This flexible architecture has the advantage of being highly interpretable. The elements of sparse parameter matrices can be viewed as factor-dependent corrections (e.g. topic- or speaker-dependent phenomena). In topic modeling experiments on conversational telephone speech, we obtain modest perplexity reductions over an n-gram baseline and demonstrate topic-dependent keyword extraction that leads to a 13% (absolute) improvement in precision over TFIDF. We also show how keywords can be jointly learned for speakers, roles and topics in a study of Supreme Court oral arguments.

In this work, we present a complete framework of discriminative training using non-uniform criteria for keyword spotting, adaptive boosted non-uniform minimum classification error (MCE) for keyword spotting on spontaneous speech. To further boost the spotting performance and tackle the potential issue of over-training in the non-uniform MCE proposed in our prior work, we make two improvements to the fundamental MCE optimization procedure. Furthermore, motivated by AdaBoost, we introduce an adaptive scheme to embed error cost functions together with model combinations during the decoding stage. The proposed framework is comprehensively validated on two challenging large-scale spontaneous conversational telephone speech (CTS) tasks in different languages (English and Mandarin) and the experimental results show it can achieve significant and consistent figure of merit (FOM) gains over both ML and discriminatively trained systems.