Incorporating linguistic structure into maximum entropy language models

An improved maximum entropy language model (IMELM) is presented based on three respects of language modeling (LM) improvement: the solution of long dependences, the integration of language knowledge into LM, and the general framework that combines all kinds of language knowledge. The proposed model combines trigram with base phrase structure knowledge in this paper. Trigram is used to capture the local relation between words, while base phrase structure knowledge is considered to represent the long-distance relations between syntactical structures. The knowledge of syntax, semantics and word is integrated in the maximum entropy framework. The experimental results show that the proposed model has a 24% improvement in perplexity over the conventional trigram model.

A maximum entropy approach to adaptive statistical language modelling

Maximum Entropy (MaxEnt) language models [1, 2] are linear models that are typically regularized via well-known L1 or L2 terms in the likelihood objective, hence avoiding the need for the kinds of backoff or mixture weights used in smoothed ngram language models using Katz backoff [3] and similar techniques. Even though backoff cost is not required to regularize the model, we investigate the use of backoff features in MaxEnt models, as well as some backoff-inspired variants. These features are shown to improve model quality substantially, as shown in perplexity and word-error rate reductions, even in very large scale training scenarios of tens or hundreds of billions of words and hundreds of millions of features. Index Terms: maximum entropy modeling, language modeling, n-gram models, linear models

In this paper, Maximum Entropy (ME) framework is used to classify text documents. The ME framework has a lot of advantages when compared with other supervised learning algorithms, such as naive Bayes classifier. For example, it makes no inherent conditional independence assumptions between terms. With four labeled data sets, extensive experiments are made to compare the accuracy of ME algorithm with those of naive Bayes and Support Vector Machine (SVM), which are two popular and accurate algorithms in the domain of text classification. The final result is that ME method consistently outperforms naive Bayes and SVM algorithms in accuracy. On the WebKB and Industry Vector data sets, the accuracy of ME algorithm increases from 81.38% to 85.52% and from 85.73% to 89.78% respectively. On the third 20 Newsgroups data set, our experimental result is opposite to that of Nigam et al. For the last Reuters-21578 data set, the accuracy of ME algorithm increases from 94.76% to 96.16%.

This article introduces two new maximum entropy (ME) methods for modeling the distribution of time to an event. One method is within the classical ME framework and provides characterizations of change point models such as the piecewise exponential distribution. The second method uses the entropy of the equilibrium distribution (ED) for the objective function and provides new characterizations of the exponential, Weibull, Pareto, and uniform distributions. With the same moment constraints, the classical ME and the maximum ED entropy algorithms generate different models for the interarrival time. © 2014 Wiley Periodicals, Inc. Naval Research Logistics 61: 427–434, 2014

In large vocabulary continuous speech recognition (LVCSR) for agglutinative and inflectional languages, we encounter problems due to theoretically infinite full-word lexicon size. Sub-word lexicon units may be utilized to dramatically reduce the out-of-vocabulary rate in test data. One can develop language models based on sub-word units to perform LVCSR. However, it has not always been beneficial to use sub-word lexicon units, since shorter units have higher acoustic confusability among them and language model history is effectively shorter as compared to the history in full-word language models. To reduce the aforementioned problems, we propose using the longest possible sub-word units in our lexicon, namely half-words and full-words only. We also incorporate linguistic rules of half-word combination into our statistical language model. The language constraints are represented with a rule-based WFSM which can be combined with an N-gram language model to yield a better and smaller language model. We study the performance of the proposed system for Turkish LVCSR, when the language constraint takes the form of enforcing vowel harmony between stems and endings. We also introduce novel error-rate metrics that are more appropriate than word-error-rate for agglutinative languages. Using half-words with a bi-gram model yields a significant reduction in word-error-rate as compared to a bi-gram full-word model. In addition, combining a tri-gram half-word language model with the vowel-harmony WFSM improves the accuracy further when rescoring the bi-gram lattices.

Ngram language models are ubiquitous in speech applications and many other natural language systems. One issue with n-gram language models is that the language is not completely represented in the model. When words appear that are not in the model, we may need to provide a smoothing method to distribute the model probabilities over the unknown values. Many techniques exist for language model smoothing with many different performance characteristics. Often the performance of smoothing algorithms may depend on the application of the language model (so, for example, unigram models with interpolation smoothing may perform better with information retrieval applications, but trigram models with backoff smoothing might perform better for speech). This paper examines the relative performance of some selected smoothing methods with bigram language models created using chat data. The language models are used for machine translation of chat data and for creating text classification models.

This paper deals with the use of a stochastic context‐free grammar (SCFG) for large vocabulary continuous speech recognition; in particular, an SCFG with phrase‐level dependency rules is built. Unlike n‐gram models, the SCFG can describe not only local constraints but also global constraints pertaining to the sentence as a whole, thus making possible language models with great expressive power. However, the inside‐outside algorithm must be used for estimation of the SCFG parameters, which involves a great amount of calculation, proportional to the third power of the number of nonterminal symbols and of the input string length. Hence, due to problems in dealing with extensive text corpora, the SCFG has hardly been applied as a language model for very large vocabulary continuous speech recognition. The proposed phrase‐level dependency SCFG allows a significant reduction of the computational load. In experiments with the EDR corpus, the proposed method proved effective. In experiments with the Mainichi corpus, a large‐scale phrase‐level dependency SCFG was built for a very large vocabulary continuous speech recognition system. Speech recognition tests with a vocabulary of about 5000 words showed that the proposed method could not compare with the trigram model in performance; however, when it was used in combination with a trigram model, the error rate was reduced by 14% compared to the trigram model alone. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(13): 48–59, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.1172

Code-switching is a commonly occurring phenomenon in many multilingual communities, wherein a speaker switches between languages within a single utterance. Conventional Word Error Rate (WER) is not sufficient for measuring the performance of code-mixed languages due to ambiguities in transcription, misspellings and borrowing of words from two different writing systems. These rendering errors artificially inflate the WER of an Automated Speech Recognition (ASR) system and complicate its evaluation. Furthermore, these errors make it harder to accurately evaluate modeling errors originating from code-switched language and acoustic models. In this work, we propose the use of a new metric, transliteration-optimized Word Error Rate (toWER) that smoothes out many of these irregularities by mapping all text to one writing system and demonstrate a correlation with the amount of code-switching present in a language. We also present a novel approach to acoustic and language modeling for bilingual code-switched Indic languages using the same transliteration approach to normalize the data for three types of language models, namely, a conventional n-gram language model, a maximum entropy based language model and a Long Short Term Memory (LSTM) language model, and a state-of-the-art Connectionist Temporal Classification (CTC) acoustic model. We demonstrate the robustness of the proposed approach on several Indic languages from Google Voice Search traffic with significant gains in ASR performance up to 10% relative over the state-of-the-art baseline.

A randomisation test is described for assessing relative abundance predictions from the maximum entropy approach to biodiversity. The null model underlying the test randomly allocates observed abundances to species, but retains key aspects of the structure of the observed communities; site richness, species composition, and trait covariance. Three test statistics are used to explore different characteristics of the predictions. Two are based on pairwise comparisons between observed and predicted species abundances (RMSE, RUSE Sqrt ). The third statistic is novel and is based on community-level abundance patterns, using an index calculated from the observed and predicted community entropies (E Diff ). Validation of the test to quantify type I and type II error rates showed no evidence of bias or circularity, confirming the dependencies quantified by Roxburgh and Mokany (2007) and Shipley (2007) have been fully accounted for within the null model. Application of the test to the vineyard data of Shipley et al. (2006) and to an Australian grassland dataset indicated significant departures from the null model, suggesting the integration of species trait information within the maximum entropy framework can successfully predict species abundance patterns. The paper concludes with some general comments on the use of maximum entropy in ecology, including a discussion of the mathematics underlying the Maxent optimisation algorithm and its implementation, the role of absent species in generating biased predictions, and some comments on determining the most appropriate level of data aggregation for Maxent analysis.

The evolutionary approach to entropy: Reconciling Georgescu-Roegen's natural philosophy with the maximum entropy framework

In Chinese speech processing, word dividing still remains a challenging problem, but it is one of the key problems for Chinese speech understanding, text-to-speech (TTS) systems and vocoders based on speech recognition and synthesis (SRSB vocoders). This paper proposes a word dividing algorithm based on statistical language models and Markov chain theory, which find the optimal word dividing solution without making use of grammatical and semantic knowledge.

We derive general distribution tests based on the method of maximum entropy (ME) density. The proposed tests are derived from maximizing the differential entropy subject to given moment constraints. By exploiting the equivalence between the ME and maximum likelihood (ML) estimates for the general exponential family, we can use the conventional likelihood ratio (LR), Wald, and Lagrange multiplier (LM) testing principles in the maximum entropy framework. In particular, we use the LM approach to derive tests for normality. Monte Carlo evidence suggests that the proposed tests are compatible with and sometimes outperform some commonly used normality tests. We show that the proposed tests can be extended to tests based on regression residuals and non-i.i.d. data in a straightforward manner. An empirical example on production function estimation is presented.

Japanese large-vocabulary continuous-speech recognition using a newspaper corpus and broadcast news

We model Chinese pitch accent prediction as a classification problem with six C-ToBI pitch accent types, and apply conditional Maximum Entropy (ME) classification to this problem. We acquire multiple levels of linguistic knowledge from natural language processing to make well-integrated features for ME framework. Five kinds of features were used to represent various linguistic constraints including phonetic features, POS tag features, phrase break features, position features, and length features.