Risk-Aware Information Retrieval

In this paper, an important question, whether a small language model can be practically accurate enough, is raised. Afterwards, the purpose of a language model, the problems that a language model faces, and the factors that affect the performance of a language model, are analyzed. Finally, a novel method for language model compression is proposed, which makes the large language model usable for applications in handheld devices, such as mobiles, smart phones, personal digital assistants (PDAs), and handheld personal computers (HPCs). In the proposed language model compression method, three aspects are included. First, the language model parameters are analyzed and a criterion based on the importance measure of n-grams is used to determine which n-grams should be kept and which removed. Second, a piecewise linear warping method is proposed to be used to compress the uni-gram count values in the full language model. And third, a rank-based quantization method is adopted to quantize the bi-gram probability values. Experiments show that by using this compression method the language model can be reduced dramatically to only about 1M bytes while the performance almost does not decrease. This provides good evidence that a language model compressed by means of a well-designed compression technique is practically accurate enough, and it makes the language model usable in handheld devices.

Invited Commentary

Despite the superior performance of large language models to generate natural language texts, it is hard to generate texts with correct logic according to a given task, due to the difficulties for neural models to capture strict logic from free-form texts. In this paper, we propose a novel graph-based language model, Logical-GLM, to extract strict logic from free-form texts and then infuse into language models. Specifically, we first capture information from natural language instructions and construct logical probability graphs that generally describe domains. Next, we generate logical skeletons to guide language model training, infusing domain knowledge into language models. At last, we alternately optimize the searching policy of graphs and language models until convergence. The experimental results show that Logical-GLM is both effective and efficient compared with traditional language models, despite using smaller-scale training data and fewer parameters. Our approach can generate instructional texts with more correct logic owing to the internalized domain knowledge. Moreover, the search of logical graphs reflects the inner mechanism of the language models, which improves the interpretability of black-box models.

Unsupervised adaptation has evolved as a popular approach for tuning the acoustic models of speaker-independent speech recognition systems to specific speakers, speaker groups or channel conditions while making use of only untranscribed data. This study focuses on procedures for unsupervised adaptation of other probabilistic models that are involved in state-of-the-art speech recognizers and on the joint adaptation of multiple knowledge sources. In particular, we outline and evaluate approaches for adapting both the language model and the pronunciation model (lexicon) without supervision. Initial experiments on off-line lecture speech transcription achieved small but promising word error rate improvements with each approach applied separately. The experimental results on the joint application of acoustic, language and pronunciation model adaptation indicate that the individually achievable performance improvements are additive.

In this paper, we propose a contextual retrieval framework which incorporates the user and global contextual information into the probabilistic retrieval model. We investigate different techniques of using contextual information to improve information retrieval performance in details. In particular, (1) we use the related text contextual information for query expansion; (2) we use the granularity information to construct the document level index and paragraph level index; (3) we use the geographic information for filtering. In addition, a new term weighting function BM5O is proposed based on the global context information. This framework is adaptable and extensible. If there is a new context category, we can extend the existing search system to accommodate it. Finally, we report our experimental findings on TREC data sets.

Nowadays automatic speech recognition (ASR) is an important task for machines. Several applications such as speech translation, virtual assistants and voice bot systems use ASR to understand human speech. Most of the research and available models are for widely used languages, such as English, German, French, Chinese and Spanish. This paper presents the Armenian speech recognition system. As a result of this research developed acoustic and language models for the Armenian language (modern ASR systems combine acoustic and language models to achieve higher accuracy). RNN-based Baidu’s Deep Speech deep neural network was used to train the acoustic model, and the KenLM toolkit was used to train the probabilistic language model. The acoustic model was trained and validated on ArmSpeech Armenian native speech corpus using transfer-learning and data augmentation techniques and tested on the Common Voice Armenian database. The language model was built based on the texts scraped from Armenian news websites. Final models are small in size and can be run and do real-time speech-to-text tasks on IoT devices. Testing on the Common Voice Armenian database the model gave 0.902565 WER and 0.305321 CER without the language model, and 0.552975 WER and 0.285904 CER with the language model. The paper aims to describe environment setup, data collection, acoustic and language models training processes, as well as final results and benchmarks.

The design and construction of a language model for minority languages is a hard task. By minority language, we mean a language with small available resources, especially for the statistical learning problem. In this paper, a new methodology for fast language model construction in minority languages is proposed. It is based on the use of Web resources to collect and make efficient textual corpora. By using some filtering techniques, this methodology allows a quick and efficient construction of a language model with a small cost in term of computational and human resources. Our primary experiments have shown excellent performance of the Web language models vs newspaper language models using the proposed filtering methods on a majority language (French). Following the same way for a minority language (Vietnamese), a valuable language model was constructed in 3 month with only 15% new development to modify some filtering tools.

A statistically designed Automatic Speech Recognition (ASR) system extracts features from speech signals using feature extraction methods, links the extracted features with the expected phonetics of the hypothesis using acoustic models, uses language model to add prior information about the structure of the target language. For many years, Mel-frequency Cepstral coefficients (MFCC), n-gram, and Hidden Markov Model (HMM) approaches have been used predominantly for feature extraction, language modeling and acoustic modeling, respectively. However, performance degradation of MFCC in noisy conditions and inaccuracy of HMMs while handling large vocabularies have made researchers to propose more efficient methods. The proposed work uses noise robust method Gammatone Frequency Cepstral Coefficients(GFCC) for feature extraction, trigram language modeling, and HMM-Gaussian mixture model (GMM) based acoustic modeling to implement a continuous Hindi language ASR system. Also, it applies Differential Evolution (DE) technique to refine the GFCC features and discriminative techniques to enhance performance of the acoustic model. The performance of the implemented system has been evaluated by using different feature extraction methods, variants of n-gram language modeling techniques and different discriminative techniques in clean as well as noisy conditions. Initially, the results reveal that DE optimized GFCC with HMM-Gaussian Mixture Model (GMM) acoustic modeling performs better than MFCC, PLP and MF-PLP feature extraction methods. Secondly, the experimental results show that the Minimum Phone Error (MPE) outperforms Maximum Mutual Information (MMI) and Maximum Likelihood Estimation (MLE) and trigram based language modeling gives more accurate results than unigram and bigram language modeling. Finally, it has been concluded that the continuous Hindi language ASR system implemented using DE refined GFFC feature extraction method with MPE discriminative training technique and trigram based language modeling gives better accuracy in clean as-well-as noisy environments.

Probabilistic language models have gained popularity in Natural Language Processing due to their ability to successfully capture language structures and constraints with computational efficiency. Probabilistic language models are flexible and easily adapted to language changes over time as well as to some new languages. Probabilistic language models can be trained and their accuracy strongly related to the availability of large text corpora. In this paper, we investigate the usability of grapheme probabilistic models, specifically grapheme n-grams models in spellchecking as well as augmentative typing systems. Grapheme n-gram models require substantially smaller training corpora and that is one of the main drivers for this thesis in which we build grapheme n-gram language models for the Albanian language. There are presently no available Albanian language corpora to be used for probabilistic language modeling. Our technique attempts to augment spellchecking and typing systems by utilizing grapheme n-gram language models in improving suggestion accuracy in spellchecking and augmentative typing systems. Our technique can be implemented in a standalone tool or incorporated in another tool to offer additional selection/scoring criteria.

The ease and speed of spreading misinformation and propaganda on the Web motivate the need to develop trustworthy technology for detecting fallacies in natural language arguments. However, state-of-the-art language modeling methods exhibit a lack of robustness on tasks like logical fallacy classification that require complex reasoning. In this paper, we propose a Case-Based Reasoning method that classifies new cases of logical fallacy by language-modeling-driven retrieval and adaptation of historical cases. We design four complementary strategies to enrich input representation for our model, based on external information about goals, explanations, counterarguments, and argument structure. Our experiments in in-domain and out-of-domain settings indicate that Case-Based Reasoning improves the accuracy and generalizability of language models. Our ablation studies suggest that representations of similar cases have a strong impact on the model performance, that models perform well with fewer retrieved cases, and that the size of the case database has a negligible effect on the performance. Finally, we dive deeper into the relationship between the properties of the retrieved cases and the model performance.

In this study, it is aimed to create a high-quality dataset to improve the performance of Turkish language models and to investigate the effect of this dataset on language model training. The irregular, context-free and noisy structure of webbased data can negatively affect the success of large language models. To address this issue, a comprehensive cleaning and filtering process has been performed on the existing CulturaX dataset. In this process, page-based and content-based filtering methods have been applied to make the dataset more consistent and meaningful. In addition, supervised machine learning models have been trained using various feature sets such as FastText and BERT embeddings, and the contributions of these feature sets to model performance have been compared. Experimental results have shown that the model that uses FastText embeddings and heuristic features together has achieved the highest accuracy and F1 scores. The resulting 125 GB cleaned Turkish web corpus provided lower perplexity and higher accuracy rates in the training of language models. This study provides a significant contribution to the development of more reliable and effective large language models for Turkish, thus providing a solid foundation for use in language model research.

The impact of distribution on value-at-risk measures

By positing a relationship between naturalistic reading times and information-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008) provides a natural interface between language models and psycholinguistic models. This paper re-evaluates a claim due to Goodkind and Bicknell (2018) that a language model’s ability to model reading times is a linear function of its perplexity. By extending Goodkind and Bicknell’s analysis to modern neural architectures, we show that the proposed relation does not always hold for Long Short-Term Memory networks, Transformers, and pre-trained models. We introduce an alternate measure of language modeling performance called predictability norm correlation based on Cloze probabilities measured from human subjects. Our new metric yields a more robust relationship between language model quality and psycholinguistic modeling performance that allows for comparison between models with different training configurations.

The system described generates a language model for use in the APRIL natural-language parser. The APRIL language model takes the form of a set of typical productions (pairings of mother label with sequence of daughter labels) for each non-terminal category; the system analyses an input by seeking a labelled tree over the words of the input which offers the best fit between each production in the tree and some production of the language model. Previous versions of APRIL used hand-designed language models, but this became a serious research bottleneck. This paper presents a system which uses stochastic optimization to reduce a large set of observed productions to a set of “prototype productions” which are few in number but which nevertheless successfully typify the large observed set.KeywordsLanguage ModelNoun PhraseLabel TreePrototype ProductionHuman AnalystThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

BackgroundPredictive modeling presents an opportunity to contain the expansion of medical expenditures by focusing on very few people. Evaluation of how risk adjustment models perform in predictive modeling in Taiwan or Asia has been rare. The aims of this study were to evaluate the performance of different risk adjustment models (the ACG risk adjustment system and prior expenditures) in predictive modeling, using Taiwan's National Health Insurance (NHI) claims data, and to compare characteristics of potentially high-expenditure subjects identified through different models.MethodsA random sample of NHI enrollees continuously enrolled in 2002 and 2003 (n = 164,562) was selected. Health status measures and total expenditures derived from 2002 NHI claims data were used to predict the possibility of becoming 2003 top users. Statistics-based indicators (C-statistics, sensitivity, & Predictive Positive Value) and characteristics of identified top groups by different models (expenditures and prevalence of manageable diseases) were presented.ResultsBoth diagnosis-based and prior expenditures models performed much better than the demographic model. Diagnosis-based models were better in identifying top users with manageable diseases; prior expenditures models were better in statistics-based indicators and identifying people with higher average expenditures. Prior expenditures status could correctly identify more actual top users than diagnosis-based or demographic models. The proportions of actual top users that could be identified by diagnosis-based models alone were much lower than that identified by prior expenditures status.ConclusionsPredicted top users identified by different models have different characteristics and there is little agreement between modes regarding which groups would be potentially top users; therefore, which model to use should depend on the purpose of predictive modeling. Prior expenditures are a more powerful tool than diagnosis-based risk adjusters in terms of correctly identifying more actual high expenditures users. There is still much room left for improvement of diagnosis-based models in predictive modeling.