Planning with Logical Graph-Based Language Model for Instruction Generation

In this paper, we describe the work developed in the creation of syllable-based language model for continuous speech recognition system for Myanmar language. Speech recognition systems contain language model as the part of prediction word order sequence. In English and other languages, speech recognition system can use word-based language model. Since Myanmar is monosyllabic and syllable-timed language, the syllable-based language model is more suitable in speech recognition system. At the first step, this paper explains the structure of traditional (Phrase based) language model and describes how to normalized Myanmar sentences and building of normalized Myanmar language model. In the second step, the evaluation of the two language models is expressed. Finally, the comparison of these two language models with the perplexity values is described. The syllable-based Myanmar language model has lower perplexity value than the traditional language model. This language model can support the creation of word-based Myanmar language model.

Recent studies have shown that integrating constructional information can improve the performance of pre-trained language models (PLMs) in natural language understanding.However, exploration into leveraging constructional information to enhance generative language models for natural language generation has been limited.Additionally, probing studies indicate that PLMs primarily grasp the syntactic structure of constructions but struggle to capture their semantics.In this work, we encode constructions as inductive biases to explicitly embed constructional semantics and guide the generation process.We begin by presenting a construction grammar induction framework designed to automatically identify constructions from corpora.Subsequently, we propose the Construction-Enhanced Language Model (CoELM).It introduces a construction-guided language modeling approach that employs a dynamic sequence reassembly strategy during pre-training.Extensive experiments have demonstrated the superiority of CoELM across various benchmarks.

The design and construction of a language model for minority languages is a hard task. By minority language, we mean a language with small available resources, especially for the statistical learning problem. In this paper, a new methodology for fast language model construction in minority languages is proposed. It is based on the use of Web resources to collect and make efficient textual corpora. By using some filtering techniques, this methodology allows a quick and efficient construction of a language model with a small cost in term of computational and human resources. Our primary experiments have shown excellent performance of the Web language models vs newspaper language models using the proposed filtering methods on a majority language (French). Following the same way for a minority language (Vietnamese), a valuable language model was constructed in 3 month with only 15% new development to modify some filtering tools.

• We propose a new method for embedding dimensions interpretation based on assigning concepts to the individual dimensions that we apply to a variety of LLMs. • Our innovation includes 1) using ontologies within a domain to extract the interpretation concepts, 2) a new metric that allows us to measure the goodness of allocation of the concepts within dimension in terms of interpretability, and 3) a thorough study of statistical distances identifying KL as the best one. • We empirically demonstrate that the quality of interpretability within equally complex LLMs is not neccesarily associated with the language model performance. • We illustrate the feasibility and applicability of our method through experiments with two ontologies, HPO and GO, and four different LLMs. As language models are used in more applications, a key problem has become clear: their numerical embeddings are hard to interpret because it is unclear how each part of the vector relates to real-world meanings in specific fields. The prevailing embedding methods are inadequate in their current state, as they are unable to effectively bridge the gap between mathematical representations and human-understandable knowledge structures. The present study proposes a novel framework that explicitly links ontology classes to specific embedding dimensions through a dual-component architecture combining a text encoder that produces the target embedding dimensions with domain knowledge graphs. The Area Under the Interpretability Curve (AUIC) metric is introduced as a means to systematically evaluate model-alignment with ontological concepts. The analysis reveals that targeted dimensional mapping enables direct interpretation of individual vector components through ontological terms. The practical applications of this framework are illustrated through case studies in biomedical contexts, demonstrating enhanced model transparency without compromising performance. This approach establishes a measurable pathway for reconciling statistical language representations with structured domain knowledge, particularly benefiting fields requiring precise concept alignment like biomedicine. The implementation is publicly available at: https://github.com/Mellandd/DEIBO .

We introduce a novel approach for building language models based on a systematic, recursive exploration of skip n-gram models which are interpolated using modified Kneser-Ney smoothing. Our approach generalizes language models as it contains the classical interpolation with lower order models as a special case. In this paper we motivate, formalize and present our approach. In an extensive empirical experiment over English text corpora we demonstrate that our generalized language models lead to a substantial reduction of perplexity between 3.1% and 12.7% in comparison to traditional language models using modified Kneser-Ney smoothing. Furthermore, we investigate the behaviour over three other languages and a domain specific corpus where we observed consistent improvements. Finally, we also show that the strength of our approach lies in its ability to cope in particular with sparse training data. Using a very small training data set of only 736 KB text we yield improvements of even 25.7% reduction of perplexity.

Statistical language models improve the performance of speech recognition systems by providing estimates of a priori probabilities of word sequences. The commonly used trigram language models obtain the conditional probability estimate of a word given the previous two words, from a large corpus of text. The text corpus is often a collection of several small diverse segments such as newspaper articles, or conversations on different topics. Knowledge of the current topic could be utilized to adapt the general trigram language models to match that topic closely. For example, an interpolation of the general language model with one built on the topic data could be used. The authors first discuss the adaptation of general trigram language models to a known topic using the minimum discrimination information (MDI) method. They then present results on the switchboard corpus which consists of telephone conversations on several topics.

Robot grasping under the instruction of natural language has attracted increasing attention in various applications for its advantages in enabling natural and smooth human-robot interaction. At present, mainstream algorithms mainly solve problems of utilizing simple natural language instructions to guide the robot arm to perform some specific grasping. However, for two natural language instructions with different temporal logic and the same semantics, it is usually difficult for the robot to achieve semantic disambiguation, which further leads to the failure of the grasping task. In order to address this problem, we propose a new natural language instruction disambiguation method for robot grasping by combining sentence vector similarity calculation model and sentence temporal logic model. Firstly, the word vector is obtained through the Skip-gram model in Word2vec and a sentence vector is constructed. The semantic similarity of the sentence is then calculated by using the proposed cost function. Based on the semantic similarity of the sentence, the correct temporal logic form of the sentence is then extracted according to the temporal adverbial priority to further guide the grabbing process of the robot arm. The experimental results show that our method can successfully realize the semantic disambiguation for natural language instructions with different temporal logics and the same semantics, and further guide the robot arm to complete more complicated tasks than previous tasks.

The advent of powerful transformer-based discriminative language models and, more recently, generative GPT-family models, has led to notable advancements in natural language processing (NLP), particularly in commonsense reasoning tasks. One such task is commonsense reasoning, where performance is usually evaluated through multiple-choice question-answering benchmarks. Till date, many such benchmarks have been proposed and `leaderboards' tracking state-of-the-art performance on those benchmarks suggest that transformer-based models are approaching human-like performance. However, due to documented problems such as hallucination and bias, the research focus is shifting from merely quantifying accuracy on the task to an in-depth, context-sensitive probing of LLMs' generalization and robustness. To gain deeper insight into diagnosing these models' performance in commonsense reasoning scenarios, this thesis addresses three main studies: the generalization ability of transformer-based language models on commonsense reasoning, the trend in confidence distribution of these language models confronted with ambiguous inference tasks, and a proposed risk-centric evaluation framework for both discriminative and generative language models.

Representing multiple compositions of human language has been a difficult task due to the complex hierarchical and compositional nature of language. Hierarchical structures are one of the architectures which can be used to capture such compositionalities. In this paper, we introduce temporal hierarchies to the Neural Language Model (NLM) with the help of a Deep Gated Recurrent Neural Network with adaptive timescales to help represent multiple compositions of language. We demonstrate that by representing multiple compositions of language in a deep recurrent neural network architecture, we can improve the performance of Language Models without complex hierarchical architectures. We report the performance of the proposed model using the popular Penn Treebank (PTB) dataset. The results show that by using the multiple timescale concept in an NLM, we can achieve better perplexities compared to the existing baselines.

To improve the accuracy of lip-reading recognition, an emotions and topic-related mixed language model has been researched. On the basis of the key words, the topic is divided by subject words, improved scene training corpus design and parameter estimation methods are used, the scene training corpus of different topics is expressed as the fuzzy subset of the whole scene training corpus, parameter estimated which can be got is also based on the fuzzy training set of different topics. The problem of sparse data which is introduced by less of training corpus in traditional language model has been eased by improved methods, quantitative description about the relationship of scene training corpus and topics has been presented, and full use of the image identification techniques for expression recognition in lipreading recognition area, auxiliary emotional factors language model to carry out lip-reading recognition.

The public release of ChatGPT, a generative artificial intelligence language model, caused wide-spread public interest in its abilities but also concern about the implications of the application on academia, depending on whether it was deemed benevolent (e.g., supporting analysis and simplification of tasks) or malevolent (e.g., assignment writing and academic misconduct). While ChatGPT has been shown to provide answers of sufficient quality to pass some university exams, its capacity to write essays that require an exploration of value concepts is unknown. This paper presents the results of a study where ChatGPT-4 (released May 2023) was tasked with writing a 1500-word essay to discuss the nature of values used in the assessment of cultural heritage significance. Based on an analysis of 36 iterations, ChatGPT wrote essays of limited length with about 50% of the stipulated word count being primarily descriptive and without any depth or complexity. The concepts, which are often flawed and suffer from inverted logic, are presented in an arbitrary sequence with limited coherence and without any defined line of argument. Given that it is a generative language model, ChatGPT often splits concepts and uses one or more words to develop tangential arguments. While ChatGPT provides references as tasked, many are fictitious, albeit with plausible authors and titles. At present, ChatGPT has the ability to critique its own work but seems unable to incorporate that critique in a meaningful way to improve a previous draft. Setting aside conceptual flaws such as inverted logic, several of the essays could possibly pass as a junior high school assignment but fall short of what would be expected in senior school, let alone at a college or university level.

In this study, it is aimed to create a high-quality dataset to improve the performance of Turkish language models and to investigate the effect of this dataset on language model training. The irregular, context-free and noisy structure of webbased data can negatively affect the success of large language models. To address this issue, a comprehensive cleaning and filtering process has been performed on the existing CulturaX dataset. In this process, page-based and content-based filtering methods have been applied to make the dataset more consistent and meaningful. In addition, supervised machine learning models have been trained using various feature sets such as FastText and BERT embeddings, and the contributions of these feature sets to model performance have been compared. Experimental results have shown that the model that uses FastText embeddings and heuristic features together has achieved the highest accuracy and F1 scores. The resulting 125 GB cleaned Turkish web corpus provided lower perplexity and higher accuracy rates in the training of language models. This study provides a significant contribution to the development of more reliable and effective large language models for Turkish, thus providing a solid foundation for use in language model research.

Code completion is one of the most useful features in the Integrated Development Environments (IDEs), which can accelerate software development by suggesting the next probable token based on the contextual code in real-time. Recent studies have shown that statistical language modeling techniques can improve the performance of code completion tools through learning from large-scale software repositories. However, these models suffer from two major drawbacks: a) Existing research uses static embeddings, which map a word to the same vector regardless of its context. The differences in the meaning of a token in varying contexts are lost when each token is associated with a single representation; b) Existing language model based code completion models perform poor on completing identifiers, and the type information of the identifiers is ignored in most of these models. To address these challenges, in this paper, we develop a multi-task learning based pre-trained language model for code understanding and code generation with a Transformer-based neural architecture. We pre-train it with hybrid objective functions that incorporate both code understanding and code generation tasks. Then we fine-tune the pre-trained model on code completion. During the completion, our model does not directly predict the next token. Instead, we adopt multi-task learning to predict the token and its type jointly and utilize the predicted type to assist the token prediction. Experiments results on two real-world datasets demonstrate the effectiveness of our model when compared with state-of-the-art methods.

Although n-gram models are still the de facto standard in language modeling for speech recognition, it has been shown that more sophisticated models achieve better accuracy by taking additional information, such as syntactic rules, semantic relations or domain knowledge into account. Unfortunately, most of the effort in developing such models goes into the implementation of handcrafted inference routines. What lacks is a generic mechanism to introduce background knowledge into a language model. We propose the use of dynamic Bayesian networks for this purpose. Dynamic Bayesian networks can be seen as a generalization of the n-gram models and hmms traditionally used in language modeling and speech recognition. Whereas those models use a single random variable to represent state, Bayesian networks can have any number of variables. As such they are particularly well-suited for the construction of models that take additional information into account. In this paper language modeling with belief networks is discussed. Examples of belief network implementations of well-known language models are given and a new model is presented that models dependencies between the content words in a sentence.KeywordsBayesian NetworkSpeech RecognitionLanguage ModelContent WordDynamic Bayesian NetworkThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this study, we employed a large language model to evaluate the diagnostic efficacy of radiology reports of bone scintigraphy in the context of identifying SAPHO syndrome, and further examined the potential of such a model to augment the diagnostic procedure. Imaging data and clinical information of 151 patients (105/46 women/men, mean age: 53.5 years) who underwent bone scintigraphy for suspected Synovitis, Acne, Pustulosis, Hyperostosis, and Osteitis (SAPHO) syndrome between January 2007 and December 2022 were retrospectively reviewed. ChatGPT-4.0 was used as the large language model. The diagnostic performance of the large language model was verified by comparing the cases judged to have SAPHO syndrome that fulfilled Kahn's classification criteria based on a combination of concise radiology reports and skin lesions such as palmoplantar pustulosis, with cases diagnosed with SAPHO syndrome by rheumatologists based on all clinical information. The diagnostic accuracy of a large language model for analysing bone scintigraphy radiology reports in conjunction with information about skin symptoms, such as palmoplantar pustulosis, achieved a sensitivity of 83.5%, specificity of 69.4%, and an overall accuracy of 76.8%. This research indicates the prospective value of extensive language models in scrutinizing radiology accounts from bone scintigraphy for the diagnosis of SAPHO syndrome.