Combining Topic Specific Language Models

Statistical language models improve the performance of speech recognition systems by providing estimates of a priori probabilities of word sequences. The commonly used trigram language models obtain the conditional probability estimate of a word given the previous two words, from a large corpus of text. The text corpus is often a collection of several small diverse segments such as newspaper articles, or conversations on different topics. Knowledge of the current topic could be utilized to adapt the general trigram language models to match that topic closely. For example, an interpolation of the general language model with one built on the topic data could be used. The authors first discuss the adaptation of general trigram language models to a known topic using the minimum discrimination information (MDI) method. They then present results on the switchboard corpus which consists of telephone conversations on several topics.

BackgroundUniversal HIV screening programs are costly, labor-intensive, and in practice unable to identify all individuals at risk of HIV infection. Automated risk assessment methods that leverage longitudinal electronic health records (EHRs) could catalyze targeted screening programs in Emergency Departments and across public health jurisdictions. While information on social and behavioral determinants of health are typically collected in unstructured fields, previous analyses have only considered structured EHR data. We sought to characterize whether clinical notes can improve predictive models of HIV diagnosis.Methods181 individuals who received care at an academic medical center in New York City prior to a confirmatory HIV diagnosis were included in the study cohort. 543 HIV- controls with similar utilization patterns were selected using propensity score matching. Demographics, laboratory tests, and diagnosis codes were extracted from longitudinal records. Clinical notes were preprocessed using both topic modeling and an n-grams approach. We fit 3 predictive models using Random Forests including a baseline model which included only structured EHR data, the baseline model plus topic modeling, and baseline model plus clinical keywords.ResultsPredictive models demonstrated a range of performance with F-measures of 0.59 for the baseline model, 0.63 for the baseline plus topic modeling and 0.74 for the baseline plus clinical keyword model. The baseline plus topic model displayed low precision but high recall while the baseline plus clinical keyword model displayed high precision but low recall. Clinical keywords including ‘msm’, ‘unprotected’, ‘hiv’, and ‘methamphetamine’ were indicative of elevated risk.ConclusionClinical notes improved the performance of predictive models for automated HIV risk assessment. Future studies should explore novel techniques for extracting social and behavioral determinants from unstructured text in longitudinal EHRs.DisclosuresAll authors: No reported disclosures.

It is estimated that approximately 60% of the natural vegetative cover of the Brazilian savanna, locally known as the Cerrado and the second largest biome in South America, have already been converted. Despite this rapid conversion pace, there have only been limited attempts to operationally monitor this major farming frontier with remote sensing data. In this study, we evaluated the performance of spectral linear mixture models (SLMM) for the mapping of the major Cerrado physiognomies. Two SLMMs were considered: a general model, comprising the vegetation, soil and shade components, and a specific model, restricted to the ‘true’ Cerrado physiognomies. We also considered the potential effects of atmospheric contamination, and the influence of endmember sources on the fraction images derived from the general and specific models, respectively. The general model, apparently resistant to the atmosphere with respect to land cover discrimination, primarily enhanced forested domains and non‐vegetated targets (water bodies and bare soils). By contrast, the specific model, regardless of the endmember source, significantly distinguished the major Cerrado physiognomies. Such contrasting and complementary behavior suggests a potential synergism between the general and specific models for the mapping and monitoring of a complex environment such as the Cerrado biome.

National Forest Inventories are important primary data sources for large-scale forest resource surveys, in which volume estimates of sampled trees are essential for quantitative analysis. Volume prediction models in natural forests are scarce in Brazil due to legal restrictions for cutting trees, especially in the Atlantic Forest. This study aimed to fit volume models for the main forest types and timber species of the Atlantic Forest in Rio de Janeiro state, considering two hypotheses: (I) generic volume models provide greater generalizability of estimates; however, (II) they may reduce the accuracy of forest type- and species-specific predictions. Four linear models with logarithmic transformation of variables were evaluated to fit volume models for generic and specific datasets, which correspond to the main forest types and timber species. Goodness-of-fit statistics were calculated to compare the accuracy and efficiency of the models, and selected models were validated through leave-one-out cross-validation procedures. The estimates obtained by generic and specific models were compared by non-parametric hypothesis tests. Generic models showed similar predictions to the specific models for forest types and timber species, with similar potential for stem and total volume predictions. Therefore, generic models can be used for Atlantic Forests in Rio de Janeiro state, while specific models are recommended to obtain more detailed local estimates.

We explore unsupervised language model adaptation techniques for Statistical Machine Translation. The hypotheses from the machine translation output are converted into queries at different levels of representation power and used to extract similar sentences from very large monolingual text collection. Specific language models are then build from the retrieved data and interpolated with a general background model. Experiments show significant improvements when translating with these adapted language models.

Suppose you are on a mobile device with no keyboard (e.g., a cell or PDA). How can you enter text quickly? T9? Graffiti? This demo will show how language modeling can be used to speed up data entry, both in the mobile context, as well as the desk-top. The Wild Thing encourages users to use wildcards (*). A language model finds the k-best expansions. Users quickly figure out when they can get away with wildcards. General purpose trigram language models are effective for the general case (unrestricted text), but there are important special cases like searching over popular web queries, where more restricted language models are even more effective.

Recent climate change is altering the migratory behaviour of many bird species. An advancement in the timing of spring events and a shift in the geographical distribution have been detected for birds around the world. In particular, intra-Palearctic migratory birds have advanced arrivals in spring and shortened migratory distances by shifting northward their wintering grounds. These changes in migratory patterns are considered adaptive responses facilitating the adjustment of the life cycle to the phenological changes found in their breeding areas. However, in some cases, populations exposed to the same selective pressures do not show any appreciable adaptive change in their behaviour. Basing on the comparison of realized and non-realized adaptive changes, I propose here the formulation of a qualitative model that predicts the potential of migratory birds populations to change adaptively their migratory behaviour. The model assumes that the adaptive potential of migratory behaviour is fuelled by both genetic diversity and phenotypic plasticity. Populations of long-distance migrants are exposed to strong environmental canalization that largely eroded their phenotypic plasticity and reduced genetic variability, so that they show a very low amount of adaptive potential regarding migratory behaviour. On the contrary, partial-migrant populations have a highly varied genetic profile and are more plastic at the phenotypic level, and consequently show the highest amount of adaptive potential. Species with mainly social and mainly genetic determination of the migratory behaviour are separately treated in the model. Specific empirical models to foresee the adaptive strategies of wild bird populations that face to climate change can be derived from the general theoretical model. As example, a specific model about the shortening of migratory distances in Western European migratory bird is presented. Finally, a number of future research lines on the topic of adaptive potential of migratory behaviour are discussed, including some examples of concrete study cases. In conclusion, partial-migration emerge as the less known system and future research efforts on this topic are expected to be especially fruitful.

An efficient, scalable speech recognition architecture is proposed for multidomain dialog systems by combining topic detection and topic-dependent language modeling. The inferred domain is automatically detected from the user's utterance, and speech recognition is then performed with an appropriate domain-dependent language model. The architecture improves accuracy and efficiency over current approaches and is scaleable to a large number of domains. In this paper, a novel framework using a multilayer hierarchy of language models is introduced in order to improve robustness against topic detection errors. The proposed system provides a relative reduction in WER of 10.5% over a single language model system. Furthermore it achieves an accuracy that is comparable to using multiple language models in parallel while using only a fraction of the computational cost.

Language model pruning is an essential technology for speech applications running on resource-constrained devices, and many pruning algorithms have been developed for conventional word n-gram models. However, while exponential language models can give superior performance, there has been little work on the pruning of these models. In this paper, we propose several pruning algorithms for general exponential language models. We show that our best algorithm applied to an exponential n-gram model outperforms existing n-gram model pruning algorithms by up to 0.4% absolute in speech recognition word-error rate on Wall Street Journal and Broadcast News data sets. In addition, we show that Model M, an exponential class-based language model, retains its performance improvement over conventional word n-gram models when pruned to equal size, with gains of up to 2.5% absolute in word-error rate.

The recent surge in large language models (LLMs) across various fields has yet to be fully realized in traditional Chinese medicine (TCM). This study aims to bridge this gap by developing a large language model tailored to TCM knowledge, enhancing its performance and accuracy in clinical reasoning tasks such as diagnosis, treatment, and prescription recommendations. This study harnessed a wide array of TCM data resources, including TCM ancient books, textbooks, and clinical data, to create 3 key datasets: the TCM Pre-trained Dataset, the Traditional Chinese Patent Medicine (TCPM) Question Answering Dataset, and the Spleen and Stomach Herbal Prescription Recommendation Dataset. These datasets underpinned the development of the Lingdan Pre-trained LLM and 2 specialized models: the Lingdan-TCPM-Chat Model, which uses a Chain-of-Thought process for symptom analysis and TCPM recommendation, and a Lingdan Prescription Recommendation model (Lingdan-PR) that proposes herbal prescriptions based on electronic medical records. The Lingdan-TCPM-Chat and the Lingdan-PR Model, fine-tuned on the Lingdan Pre-trained LLM, demonstrated state-of-the art performances for the tasks of TCM clinical knowledge answering and herbal prescription recommendation. Notably, Lingdan-PR outperformed all state-of-the-art baseline models, achieving an improvement of 18.39% in the Top@20 F1-score compared with the best baseline. This study marks a pivotal step in merging advanced LLMs with TCM, showcasing the potential of artificial intelligence to help improve clinical decision-making of medical diagnostics and treatment strategies. The success of the Lingdan Pre-trained LLM and its derivative models, Lingdan-TCPM-Chat and Lingdan-PR, not only revolutionizes TCM practices but also opens new avenues for the application of artificial intelligence in other specialized medical fields. Our project is available at https://github.com/TCMAI-BJTU/LingdanLLM.

This paper deals with the use of a stochastic context‐free grammar (SCFG) for large vocabulary continuous speech recognition; in particular, an SCFG with phrase‐level dependency rules is built. Unlike n‐gram models, the SCFG can describe not only local constraints but also global constraints pertaining to the sentence as a whole, thus making possible language models with great expressive power. However, the inside‐outside algorithm must be used for estimation of the SCFG parameters, which involves a great amount of calculation, proportional to the third power of the number of nonterminal symbols and of the input string length. Hence, due to problems in dealing with extensive text corpora, the SCFG has hardly been applied as a language model for very large vocabulary continuous speech recognition. The proposed phrase‐level dependency SCFG allows a significant reduction of the computational load. In experiments with the EDR corpus, the proposed method proved effective. In experiments with the Mainichi corpus, a large‐scale phrase‐level dependency SCFG was built for a very large vocabulary continuous speech recognition system. Speech recognition tests with a vocabulary of about 5000 words showed that the proposed method could not compare with the trigram model in performance; however, when it was used in combination with a trigram model, the error rate was reduced by 14% compared to the trigram model alone. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(13): 48–59, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.1172

In this paper we present a scheme to select relevant subsets of sentences from a large generic corpus such as text acquired from the web. A relative entropy (R.E) based criterion is used to incrementally select sentences whose distribution matches the domain of interest. Experimental results show that by using the proposed subset selection scheme we can get significant performance improvement in both Word Error Rate (WER) and Perplexity (PPL) over the models built from the entire web-corpus by using just 10% of the data. In addition incremental data selection enables us to achieve significant reduction in the vocabulary size as well as number of n-grams in the adapted language model. To demonstrate the gains from our method we provide a comparative analysis with a number of methods proposed in recent language modeling literature for cleaning up text.

The public release of ChatGPT, a generative artificial intelligence language model, caused wide-spread public interest in its abilities but also concern about the implications of the application on academia, depending on whether it was deemed benevolent (e.g., supporting analysis and simplification of tasks) or malevolent (e.g., assignment writing and academic misconduct). While ChatGPT has been shown to provide answers of sufficient quality to pass some university exams, its capacity to write essays that require an exploration of value concepts is unknown. This paper presents the results of a study where ChatGPT-4 (released May 2023) was tasked with writing a 1500-word essay to discuss the nature of values used in the assessment of cultural heritage significance. Based on an analysis of 36 iterations, ChatGPT wrote essays of limited length with about 50% of the stipulated word count being primarily descriptive and without any depth or complexity. The concepts, which are often flawed and suffer from inverted logic, are presented in an arbitrary sequence with limited coherence and without any defined line of argument. Given that it is a generative language model, ChatGPT often splits concepts and uses one or more words to develop tangential arguments. While ChatGPT provides references as tasked, many are fictitious, albeit with plausible authors and titles. At present, ChatGPT has the ability to critique its own work but seems unable to incorporate that critique in a meaningful way to improve a previous draft. Setting aside conceptual flaws such as inverted logic, several of the essays could possibly pass as a junior high school assignment but fall short of what would be expected in senior school, let alone at a college or university level.

Ngram language models are ubiquitous in speech applications and many other natural language systems. One issue with n-gram language models is that the language is not completely represented in the model. When words appear that are not in the model, we may need to provide a smoothing method to distribute the model probabilities over the unknown values. Many techniques exist for language model smoothing with many different performance characteristics. Often the performance of smoothing algorithms may depend on the application of the language model (so, for example, unigram models with interpolation smoothing may perform better with information retrieval applications, but trigram models with backoff smoothing might perform better for speech). This paper examines the relative performance of some selected smoothing methods with bigram language models created using chat data. The language models are used for machine translation of chat data and for creating text classification models.

Along with the exponential growth of text data on the Web, particularly of the user-generated content, comes an increasing need for hierarchically organizing documents, retrieving documents accurately, and discovering evolutionary trends of various popular topics from the data. However, all of these are challenging due to the diversity, heterogeneity, noisiness and time-sensitivity of Web 2.0 data. Motivated by this, we tackle the challenges at a fundamental level, by proposing a novel topic modeling method with ontological guidance. It may be used to discover topic language models formalizing various terms relevant to given topics using the Web data. The topic model takes into account both the ontological relationships amongst the topics defined in a topic taxonomy and also word co-occurrence patterns in the data to automatically identify the portions in the data relevant to the topics. Then, it estimates language models for these topics from these relevant portions. At an application level, we use the topic model to propose novel approaches for three different tasks, namely hierarchical text classification without labeled data, information retrieval with pseudo-relevance feedback, and discovering topic evolutionary trends. Our classification experiment on the IPTC (International Press and Telecommunications Council) taxonomy, containing more than 1100 topics, shows that our approach achieves a performance of 67% in terms of the hierarchical version of the F-1 measure, without using any labeled data. Our retrieval experiments on five benchmark datasets show that compared to baseline retrieval (without pseudo-relevance feed-back), our approach improves on average 39% in terms of mean average precision. Finally, for the last task, using blog data, our approach discovers meaningful insights on how the crowd responds to various news topics such as the language used to discuss each topic, how this language drifts over time, and when the crowd’s focus on a topic increases, reaches a peak, and declines.