A Formal Framework for Evaluating Reasoning Integrity in Language Models

The world has been facing the COVID-19 pandemic, which has come with an unprecedented impact on general physical health and financial and social repercussions. The adopted mitigation measures also present significant challenges to the population’s mental health and health-related programs. It is complex for public organizations to measure the population’s mental health to incorporate its feedback into their decision-making process. A significant portion of the population has turned to social media to express the details of their daily life, making these public data a rich field for understanding emotional and mental well-being. To this end, by using open sentiment analysis tools, we analyzed 760,064,879 public domain tweets collected from a public access repository to examine the collective shifts in the general mood about the pandemic evolution, news cycles, and governmental policies. Several modern language models were evaluated and compared using intrinsic and extrinsic tasks, that is, the sentiment analysis evaluation of public domain tweets related to the COVID-19 pandemic in Mexico. This study provides a fair evaluation of state-of-the-art language models, such as BERT and VADER, showcasing their metrics and comparing their performance against a real-world task. Results show the importance of selecting the correct language model for large projects such as this one, for there is a need to balance costs with the model’s performance.

The increasing complexity and scale of language models have introduced significant challenges related to computational efficiency, inference speed, and the generation of factually accurate responses. Addressing these issues, a novel architectural enhancement known as the Dynamic Sequential Transformer (DST) has been proposed, offering a groundbreaking approach to token processing through dynamic token sequencing and adaptive attention recalibration. By introducing a more flexible and context-aware method for handling tokenized inputs, DST improves both accuracy and speed while reducing the occurrence of hallucinations, which are a common problem in language generation tasks. The experimental results have demonstrated that the DST architecture significantly reduces memory usage, accelerates inference times, and improves model convergence rates, making it an efficient solution for large-scale deployments without sacrificing output quality. Additionally, DST's modular design ensures scalability and robustness across diverse language processing tasks, positioning it as a promising advancement for practical real-world applications. Through extensive evaluation, the study confirms that DST not only optimizes resource utilization but also enhances the linguistic coherence and factual consistency of generated text, establishing a new benchmark for architectural efficiency in modern language models.

Do large multilingual language models learn language universals?We consider a much discussed candidate universal, the Final-over-Final Condition (Sheehan et al., 2017b).This Condition is syntactic in the sense that it can only be stated by reference to abstract sentence properties such as nested phrases and head direction.A study of typologically diverse "mixed head direction" languages confirms that the Condition holds in corpora.But in a targeted syntactic evaluation, Gemini Pro only seems to respect the Condition in German, Russian, Hungarian and Serbian.These relatively high-resource languages contrast with Basque, where Gemini Pro does not seem to have learned the Condition at all.This result suggests that modern language models may need additional sources of bias in order to become truly human-like, within a developmentallyrealistic budget of training data.

This paper starts from the remarkable habit of yoik singing among the Sámi people and the notion of togetherness in ‘singing together’, being confronted with the role of technology in the contemporary manifestations of ‘singing’. An important link appears to exist between several forms of grooming and the social dimensions of language and singing, as previously shown by Robin Dunbar. These types of interaction surpass the physiological, hormonal and psychological roles of grooming in humans, in non-human primates and in other mammals. For they also determine the constraints of group-forming processes in mammals, and social structures in humans too. The characteristic group size of grooming mammals has been reported to determine the necessity of social stratification. Number constraints therefore form a silver lining throughout this paper. The analogy with artificial language models, and modern language models (MLM) in particular, has been chosen following the suggested applicability of AI for the deciphering of the language of Whales (Cetaceans). The polemic discussion between Noam Chomsky’s so-called classical generative linguistics and propagators of the nonlinguistic MLM approach (referring to Steven T. Piantadosi) is found an instructive heuristic for analyzing the future impact of AI on human culture. It is suggested that AI-generated analogues of ‘singing together’ will profoundly interact (or already do so) with social stratification and with the conservation of human values.

Abstract—Artificial intelligence–based conversational agents have recently gained attention as scalable solutions for mental health assistance. Large Language Models (LLMs) demonstrate strong language understanding and generation capabilities; how- ever, their responses in counseling contexts often lack emotional grounding, empathy, and philosophical reasoning required for sensitive mental health interactions. This research proposes a Spiritual-Emotion Guided Mental Health Dialogue System developed using parameter-efficient fine- tuning of the Mistral-7B-Instruct large language model. The proposed system integrates philosophical teachings from the Bhagavad Gita to improve emotional stability, reflective reasoning, and empathetic response generation in mental health conversations. Bhagavad Gita scriptures are extracted from PDF sources using the PDFPlumber framework and converted into structured question–answer instruction pairs for supervised fine-tuning. The base model is adapted using Low-Rank Adaptation (LoRA), enabling efficient domain-specific learning while preserving the general capabilities of the pretrained model. The proposed system is evaluated using a mental health counseling dialogue dataset to assess its effectiveness in generating supportive and contextually meaningful responses. The expected outcomes indicate improved emotional coherence, contextual understanding, and response quality compared to the baseline language model. The findings aim to demonstrate that integrating philosophical knowledge with modern language models can enhance the effectiveness and empathy of AI-driven mental health dialogue systems. Index Terms—Mental Health AI, Mistral-7B, LoRA, Bhagavad Gita, Psych8K, Emotional Intelligence, Parameter Efficient Fine- Tuning

Neural language models (NLMs) are susceptible to producing inconsistent output. This paper proposes a new diagnosis as well as a novel remedy for NLMs' incoherence. We train NLMs on synthetic text corpora that are created by simulating text production in a society. For diagnostic purposes, we explicitly model the individual belief systems of artificial agents (authors) who produce corpus texts. NLMs, trained on those texts, can be shown to aggregate the judgments of individual authors during pre-training according to sentence-wise vote ratios (roughly, reporting frequencies), which inevitably leads to so-called discursive dilemmas: aggregate judgments are inconsistent even though all individual belief states are consistent. As a remedy for such inconsistencies, we develop a self-training procedure—inspired by the concept of reflective equilibrium—that effectively reduces the extent of logical incoherence in a model's belief system, corrects global mis-confidence, and eventually allows the model to settle on a new, epistemically superior belief state. Thus, social choice theory helps to understand why NLMs are prone to produce inconsistencies; epistemology suggests how to get rid of them.

In this paper, we consider the problem of controlling a partially observed Markov decision process (POMDP) in order to actively estimate its state trajectory over a fixed horizon with minimal uncertainty. We pose a novel active smoothing problem in which the objective is to directly minimise the smoother entropy, that is, the conditional entropy of the (joint) state trajectory distribution of concern in fixed-interval Bayesian smoothing. Our formulation contrasts with prior active approaches that minimise the sum of conditional entropies of the (marginal) state estimates provided by Bayesian filters. By establishing a novel form of the smoother entropy in terms of the POMDP belief (or information) state, we show that our active smoothing problem can be reformulated as a (fully observed) Markov decision process with a value function that is concave in the belief state. The concavity of the value function is of particular importance since it enables the approximate solution of our active smoothing problem using piecewise-linear function approximations in conjunction with standard POMDP solvers. We illustrate the approximate solution of our active smoothing problem in simulation and compare its performance to alternative approaches based on minimising marginal state estimate uncertainties.

This study investigates the linguistic competence of modern language models. Artificial neural networks demonstrate high quality in many natural language processing tasks. However, their implicit grammar knowledge remains unstudied. The ability to judge a sentence as grammatical or ungrammatical is regarded as key property of human’s linguistic competence. We suppose that language models’ grammar knowledge also occurs in their ability to judge the grammaticality of a sentence. In order to test neural networks’ linguistic competence, we probe their acquisition of number predicate agreement in Russian. A dataset consisted of artificially generated grammatical and ungrammatical sentences was created to train the language models. Automatic sentence generation allows us to test the acquisition of particular language phenomenon, to detach from vocabulary and pragmatic differences. We use transfer learning of pre-trained neural networks. The results show that all the considered models demonstrate high accuracy and Matthew's correlation coefficient values which can be attributed to successful acquisition of predicate agreement rules. The classification quality is reduced for sentences with inanimate nouns which show nominative-accusative case syncretism. The complexity of the syntactic structure turns out to be significant for Russian models and a model for Slavic languages, but it does not affect the errors distribution of multilingual models.

We present ConflLlama, demonstrating how efficient fine-tuning of large language models can advance automated classification tasks in political science research. While classification of political events has traditionally relied on manual coding or rigid rule-based systems, modern language models offer the potential for more nuanced, context-aware analysis. However, deploying these models requires overcoming significant technical and resource barriers. We demonstrate how to adapt open-source language models to specialized political science tasks, using conflict event classification as our proof of concept. Through quantization and efficient fine-tuning techniques, we show state-of-the-art performance while minimizing computational requirements. Our approach achieves a macro-averaged AUC of 0.791 and a weighted F1-score of 0.753, representing a 37.6% improvement over the base model, with accuracy gains of up to 1463% in challenging classifications. We offer a roadmap for political scientists to adapt these methods to their own research domains, democratizing access to advanced NLP capabilities across the discipline. This work bridges the gap between cutting-edge AI developments and practical political science research needs, enabling broader adoption of these powerful analytical tools.

Recent breakthroughs in scale have enabled the emergence of powerful generative language models, and the ability to fine-tune these models on various tasks by casting them into prompts or instructions. In this landscape, the problem of Unsupervised Domain Adaptation (UDA), or the problem of leveraging knowledge from a labeled source domain to an unlabeled target domain, has been left behind, with recent UDA methods still addressing discriminative classification. In particular, two popular UDA approaches, involving Continued Pre-Training (CPT) and learning domain invariant representations, have been under-explored in the generative setting, signaling a gap. In this work, we evaluate the utility of CPT for generative UDA. We first perform an empirical evaluation to measure the trade-offs between CPT and strong methods promoting domain invariance. We further evaluate how well the benefits of CPT extend to different architectures, tuning methods and data regimes. We then motivate the use of CPT by studying to what degree it benefits classification performance on the target domain. Finally, we attempt to understand the mechanism behind which CPT improves classification performance on the unlabeled target domain. Our findings suggest that a implicitly learns the downstream task while predicting masked words informative to that task. Our work connects the body of UDA research with that of instruction tuning, enabling an initial step towards a wider applicability of modern language models.

In this paper we discuss three issues in modern language modeling. The first one is the question of a quality measure for language models, the second is language model smoothing and the third is the question of how to build good long-range language models. In all three cases some results are given indicating possible directions of further research.

BackgroundShoulder injuries are relatively common among professional rugby players and result in a large proportion of days absent from training and competition. No instrument exists that is designed and validated...

ObjectivesThis study was conducted to develop the Mobility to Participation Assessment Scale for Stroke (MPASS) and assess its content validity, internal consistency, inter-rater and intra-rater reliability, and convergent validity in people with stroke living in the community.MethodsThe MPASS was developed using published data on mobility-related activity and participation timing in elderly individuals, and then reviewed by community physical therapists. Content validity was established by reaching a consensus of experienced physical therapists in a focus group. The MPASS was scored for 32 participants with stroke (mean age 61.75±4.92 years) by 3 individual testers. Reliability was examined using the intraclass correlation coefficient (ICC), internal consistency using the Cronbach alpha coefficient (α), and convergent validity using the Pearson correlation coefficient (r) to compare the MPASS to the Modified Rivermead Mobility Index as a referent test of mobility.ResultsThe MPASS consists of 8 items, and its scoring system provides information on the ability of people with stroke to reach a movement level enabling them to live in society, including interactions with other people and safe living in the community. The inter-rater and intra-rater reliability were excellent (ICC, 0.948; 95% confidence interval [CI], 0.893 to 0.982 and ICC, 0.967; 95% CI, 0.933 to 0.989, respectively). Internal consistency was good (α=0.877). The convergent validity was moderate (r=0.646; p<0.001).ConclusionsThe newly developed MPASS showed acceptable construct validity and high reliability. The MPASS is suitable for use in people with stroke, especially those who have been discharged and live in the community with the ability to initiate sitting.

Validation of the Spine Oncology Study Group—Outcomes Questionnaire to assess quality of life in patients with metastatic spine disease

This article studies a composer style classification task based on raw sheet music images. While previous works on composer recognition have relied exclusively on supervised learning, we explore the use of self-supervised pretraining methods that have been recently developed for natural language processing. We first convert sheet music images to sequences of musical words, train a language model on a large set of unlabeled musical “sentences”, initialize a classifier with the pretrained language model weights, and then finetune the classifier on a small set of labeled data. We conduct extensive experiments on International Music Score Library Project (IMSLP) piano data using a range of modern language model architectures. We show that pretraining substantially improves classification performance and that Transformer-based architectures perform best. We also introduce two data augmentation strategies and present evidence that the model learns generalizable and semantically meaningful information.