Mental model shifts in human-LLM interactions

Large Language Models (LLMs) can solve some undergraduate-level to graduate-level physics textbook problems and are proficient at coding. Combining these two capabilities could one day enable AI systems to simulate and predict the physical world. We present an evaluation of state-of-the-art (SOTA) LLMs on PhD-level to research-level computational physics problems. We condition LLM generation on the use of well-documented and widely-used packages to elicit coding capabilities in the physics and astrophysics domains. We contribute ∼50 original and challenging problems in celestial mechanics (with REBOUND), stellar physics (with MESA), 1D fluid dynamics (with Dedalus) and non-linear dynamics (with SciPy). Since our problems do not admit unique solutions, we evaluate LLM performance on several soft metrics: counts of lines that contain different types of errors (coding, physics, necessity and sufficiency) as well as a more educational’ Pass-Fail metric focused on capturing the salient physical ingredients of the problem at hand. As expected, today's SOTA LLM (GPT4) zero-shot fails most of our problems, although about 40% of the solutions could plausibly get a passing grade. About 70%–90% of the code lines produced are necessary, sufficient and correct (coding & physics). Physics and coding errors are the most common, with some unnecessary or insufficient lines. We observe significant variations across problem class and difficulty. We identify several failure modes of GPT4 in the computational physics domain, such as poor physical units handling, poor code versioning, tendency to hallucinate plausible sub-modules, lack of physical justification for global run parameters (e.g., simulation time, or upper-lower bounds for parametric exploration) and inability to define steady-state or stopping conditions reliably. Our reconnaissance work provides a snapshot of current computational capabilities in classical physics and points to obvious improvement targets if AI systems are ever to reach a basic level of autonomy in physics simulation capabilities.

The mathematical capabilities of AI systems are complex and multifaceted. Most existing research has predominantly focused on the correctness of AI-generated solutions to mathematical problems. In this work, we argue that beyond producing correct answers, AI systems should also be capable of, or assist humans in, developing novel solutions to mathematical challenges. This study explores the creative potential of Large Language Models (LLMs) in mathematical reasoning, an aspect that has received limited attention in prior research. We introduce a novel framework and benchmark, CreativeMath, which encompasses problems ranging from middle school curricula to Olympic-level competitions, designed to assess LLMs' ability to propose innovative solutions after some known solutions have been provided. Our experiments demonstrate that, while LLMs perform well on standard mathematical tasks, their capacity for creative problem-solving varies considerably. Notably, the Gemini-1.5-Pro model outperformed other LLMs in generating novel solutions. This research opens a new frontier in evaluating AI creativity, shedding light on both the strengths and limitations of LLMs in fostering mathematical innovation, and setting the stage for future developments in AI-assisted mathematical discovery.

The emergence of large-language models (LLMs) in everyday applications has highlighted their capability to provide human-like dialog and answer complex questions. However, LLMs often exhibit a tendency to "hallucinate," generating responses that appear factual but are incorrect, even when trained on ground truth data. In high-stakes applications such as medical diagnostics and mechanical troubleshooting, the need for accurate, trustworthy, and explainable AI systems is paramount. Without these qualities, such systems cannot be fully relied upon for critical decision-making. In this paper, we propose a novel AI system designed to address these challenges. The system integrates an AI agent, a logic model, an LLM, and a prompt engine to enhance accuracy and trustworthiness in a sequence of dialog. Experimental results demonstrate that the proposed system achieves an average accuracy of 86.49% across three diagnostic and troubleshooting tasks in healthcare, automotive maintenance, and digital equipment applications. In contrast, the baseline LLM only approach achieves an average accuracy of 63.42%, representing an absolute accuracy improvement of 23.07% with our approach. Additionally, the proposed system achieves a 4.26-fold speedup in response time across the experiments. These results highlight the potential of the proposed system to enhance reliability and efficiency in critical real-world applications.

Background: The deployment of large language models (LLMs) in mental health therapy presents a compelling yet deeply fraught opportunity to address widespread disparities in access to psychological care. Recent empirical evidence reveals that these AI systems exhibit substantial shortcomings when confronted with complex clinical contexts. Methods: This paper synthesizes key findings from a critical analysis of LLMs operating in therapeutic roles and argues for the urgent establishment of comprehensive risk management frameworks, policy interventions, and ethical protocols governing their use. Results: LLMs tested in simulated therapeutic settings frequently exhibited stigmatizing attitudes toward mental health conditions and responded inappropriately to acute clinical symptoms such as suicidal ideation, psychosis, and delusions. Real-world evaluations reinforce these concerns. Some studies found that therapy and companion bots endorsed unsafe or harmful suggestions in adolescent crisis vignettes, while others reported inadequate chatbot responses to self-harm and sexual assault queries, prompting concern from clinicians, disappointment from patients, and calls for stronger oversight from policymakers. These failures contravene fundamental principles of safe clinical practice, including non-maleficence, therapeutic alliance, and evidence-based care. Moreover, LLMs lack the emotional intelligence, contextual grounding, and ethical accountability that underpin the professional responsibilities of human therapists. Their propensity for sycophantic or non-directive responses, driven by alignment objectives rather than clinical efficacy, further undermines their therapeutic utility. Conclusions: This analysis highlights barriers to the replacement of human therapists with autonomous AI systems. It also calls attention to the regulatory vacuum surrounding LLM-based wellness and therapy applications, many of which are widely accessible and unvetted. Recommendations include professional standards, transparency in training and deployment, robust privacy protections, and clinician oversight. The findings underscore the need to redefine AI as supportive, not substitutive.

Are large language models valid tools for patient information on lumbar disc herniation? The spine surgeons' perspective

As Large Language Models (LLMs) become increasingly integrated into everyday life as general-purpose multimodal AI systems, their capabilities to simulate human understanding are under examination. This study investigates LLMs’ ability to interpret linguistic pragmatics, which involves context and implied meanings. Using Grice’s communication principles, we evaluated both LLMs (GPT-2, GPT-3, GPT-3.5, GPT-4, and Bard) and human subjects (N = 147) on dialogue-based tasks. Human participants included 71 primarily Serbian students and 76 native English speakers from the United States. Findings revealed that LLMs, particularly GPT-4, outperformed humans. GPT-4 achieved the highest score of 4.80, surpassing the best human score of 4.55. Other LLMs performed well: GPT-3.5 scored 4.10, Bard 3.75, and GPT-3 3.25; GPT-2 had the lowest score of 1.05. The average LLM score was 3.39, exceeding the human cohorts’ averages of 2.80 (Serbian students) and 2.34 (U.S. participants). In the ranking of all 155 subjects (including LLMs and humans), GPT-4 secured the top position, while the best human ranked second. These results highlight significant progress in LLMs’ ability to simulate understanding of linguistic pragmatics. Future studies should confirm these findings with more dialogue-based tasks and diverse participants. This research has important implications for advancing general-purpose AI models in various communication-centered tasks, including potential application in humanoid robots in the future.

Forgetting is the complementary process to memory, enabling both biological and artificial systems to prioritize relevant information while discarding outdated or redundant data. In large language models (LLMs), forgetting occurs through computational mechanisms such as attention bottlenecks, catastrophic interference, and optimization dynamics, mirroring neurobiological processes such as synaptic decay and memory consolidation. While existing research has extensively studied memory mechanisms in LLMs, there remains a critical gap in systematically analyzing forgetting phenomena and their implications for AI system design. This study presents a comprehensive review of forgetting mechanisms in LLMs through a neuroscience-inspired lens. This study first establishes parallels between human forgetting processes and computational forgetting patterns in LLMs. Building on these foundations, this study proposes a novel taxonomy that classifies LLM forgetting across four key dimensions. This taxonomy is further supported by the introduction of quantitative evaluation metrics to provide additional evidence. Furthermore, this study demonstrates how controlled forgetting can be transformed from a limitation into a design feature through bio-inspired optimization strategies. These approaches enable critical applications in privacy-compliant AI, adaptive education systems, and sustainable computing. The study concludes by identifying open challenges in generalization, ethical governance of machine forgetting, and computational efficiency, while outlining future directions for developing LLMs with human-like memory adaptability.

This chapter critically examines the ethical, legal, and societal implications of integrating large language models (LLMs) into political and economic structures. By drawing from interdisciplinary perspectives in psychology, AI ethics, and legal scholarship, it explores the evolving landscape of AI governance and its impact on society. The discussion focuses on the challenges of granting AI autonomy and agency, assessing how LLMs influence decision-making and governance. A key aspect of the chapter is its proposal for a framework of “Constitutional AI,” which seeks to align AI decision-making with constitutional values such as fairness, justice, and transparency. It highlights the need for explainable AI (XAI) techniques, robust governance policies, and ethical considerations in AI system design. The potential risks of AI misuse, manipulation, and opacity are also addressed, emphasizing accountability and user empowerment. The chapter further examines psychological and philosophical concepts like agency, normativity, and the metaphysics of self-constitution, linking them to AI’s role in human decision-making. Ultimately, it advocates for AI systems that operate in a safe, secure, and trustworthy manner, ensuring their development benefits society while maintaining ethical integrity and legal compliance.

While large language models have transformed how we interact with AI systems, they have a critical weakness: they confidently state false in- formation that sounds entirely plausible. This ”hallucination” problem has become a major barrier to using these models where accuracy matters most. Existing solutions either require retraining the entire model, add significant computational costs, or miss the root causes of why these hallucinations occur in the first place. We present Causal Guard, a new approach that combines causal reasoning with symbolic logic to catch and prevent hallucinations as they happen. Unlike previous methods that only check outputs after generation, our system understands the causal chain that leads to false statements and intervenes early in the process. Causal Guard works through two complementary paths: one that traces causal relationships between what the model knows and what it generates, and an- other that checks logical consistency using auto- mated reasoning. Testing across twelve different benchmarks, we found that Causal Guard correctly identifies hallucinations 89.3% of the time while missing only 8.3% of actual hallucinations. More importantly, it reduces false claims by nearly 80% while keeping responses natural and helpful. The system performs especially well on complex reasoning tasks where multiple steps of logic are required. Because Causal Guard shows its reasoning process, it works well in sensitive areas like medical diagnosis or financial analysis where understanding why a decision was made matters as much as the decision itself.

This study explores the growing transition in healthcare from rigid, rule-based algorithms to intelligent, self-directed “agentic” AI systems. These agents rely on Large Language Models (LLMs) to think step-by-step, make decisions, and carry out tasks using a simple yet powerful four-part structure: planning, action, reflection, and memory. The paper reviews their current uses in medical diagnosis, hospital workflow automation, and collaborative multi-agent setups. It also highlights a major challenge — most testing still happens in controlled lab settings rather than real hospitals — and discusses what is needed for safe everyday adoption. Keywords: Medical AI Agents, Autonomous Clinical Systems, Large Language Models, Chain-of-Thought Reasoning, Multimodal Integration, Agentic Framework, Clinical Decision Support, Human-in-the-Loop, Healthcare Automation, Physician Burnout, Precision Medicine, Scoping Review, Simulation Gap, PRISMA-ScR. Healthcare AI agents support several key areas. They strengthen diagnostic accuracy in critical situations such as predicting sepsis or identifying cancer on scans. They lighten the administrative load by automatically creating summaries from patient records, which helps reduce doctor burnout. They also improve patient involvement by offering personalised guidance, voice-based recovery check-ins, and easier remote care. In addition, groups of specialist agents can work together on complicated cases, such as building complete treatment plans.

The computational significance of consciousness is an important and potentially more tractable research theme than the hard problem of consciousness, as one could look at the correlation of consciousness and computational capacities through, e.g., algorithmic or complexity analyses. In the literature, consciousness is defined as what it is like to be an agent (i.e., a human or a bat), with phenomenal properties, such as qualia, intentionality, and self-awareness. The absence of these properties would be termed "unconscious." The recent success of large language models (LLMs), such as ChatGPT, has raised new questions about the computational significance of human conscious processing. Although instances from biological systems would typically suggest a robust correlation between intelligence and consciousness, certain states of consciousness seem to exist without manifest existence of intelligence. On the other hand, AI systems seem to exhibit intelligence without consciousness. These instances seem to suggest possible dissociations between consciousness and intelligence in natural and artificial systems. Here, I review some salient ideas about the computational significance of human conscious processes and identify several cognitive domains potentially unique to consciousness, such as flexible attention modulation, robust handling of new contexts, choice and decision making, cognition reflecting a wide spectrum of sensory information in an integrated manner, and finally embodied cognition, which might involve unconscious processes as well. Compared to such cognitive tasks, characterized by flexible and ad hoc judgments and choices, adequately acquired knowledge and skills are typically processed unconsciously in humans, consistent with the view that computation exhibited by LLMs, which are pretrained on a large dataset, could in principle be processed without consciousness, although conversations in humans are typically done consciously, with awareness of auditory qualia as well as the semantics of what are being said. I discuss the theoretically and practically important issue of separating computations, which need to be conducted consciously from those which could be done unconsciously, in areas, such as perception, language, and driving. I propose conscious supremacy as a concept analogous to quantum supremacy, which would help identify computations possibly unique to consciousness in biologically practical time and resource limits. I explore possible mechanisms supporting the hypothetical conscious supremacy. Finally, I discuss the relevance of issues covered here for AI alignment, where computations of AI and humans need to be aligned.

Large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks. Understanding how LLMs internally represent knowledge remains a significant challenge. Despite Sparse Autoencoders (SAEs) have emerged as a promising technique for extracting interpretable features from LLMs, SAE features do not inherently align with human-understandable concepts, making their interpretation cumbersome and labor-intensive. To bridge the gap between SAE features and human concepts, we present ConceptViz, a visual analytics system designed for exploring concepts in LLMs. ConceptViz implements a novel Identification ⇒ Interpretation ⇒Validation pipeline, enabling users to query SAEs using concepts of interest, interactively explore concept-to-feature alignments, and validate the correspondences through model behavior verification. We demonstrate the effectiveness of ConceptViz through two usage scenarios and a user study. Our results show that ConceptViz enhances interpretability research by streamlining the discovery and validation of meaningful concept representations in LLMs, ultimately aiding researchers in building more accurate mental models of LLM features. Our code and user guide are publicly available at https://github.com/Happy-Hippo209/conceptViz.

This paper examines the potential of integrating Large Language Models (LLMs) with knowledge bases to imrpove the accuracy and reliability of their responses. The advantages of such a combination are evaluated, particularly in reducing the risk of hallucinations – the phenomenon where models generate erroneous or fabricated information. Various methodologies for combining LLMs with knowledge bases are analyzed, along with their respective advantages and limitations. The prospects and challenges of implementing this technology in diverse fields—such as information retrieval, decision support, and automated content creation—are discussed. The paper presents an overview of the current state of research in this domain and delineates directions for future investigation.The integration of LLMs with knowledge bases represents a significant advancement in artificial intelligence, aiming one of the key concerns regarding LLMs—their tendency to generate inaccurate or fabricated information, commonly referred to as hallucinations. This approach leverages the vast language understanding and generation capabilities of LLMs while grounding their outputs in structured and verified information from knowledge bases. The synergy between these two technologies has the potential to significantly enhance the reliability and factual accuracy of AI-generated responses across a wide range of applications. The methodologies for combining LLMs with knowledge bases differ in their implementation and effectiveness. Some approaches involve pre-training LLMs on curated knowledge bases, while others reference knowledge bases externally during the inference process. Each method presents its own set of advantages and challenges, such as balancing computational efficiency against accuracy and maintaining the fluency of LLM outputs while adhering strictly to factual information. The application of this integrated technology extends beyond mere information retrieval, showing promise in complex decision support systems, automated content creation for specialized domains, and contributing to the advancement of explainable AI by providing traceable sources for generated information. As research in this area progresses, it is expected to open new avenues for developing more trustworthy and capable AI systems across various industries and academic disciplines.

Across the sciences, each discipline develops its own domain ontology—its structural assumptions about what exists, from particles and fields to molecules, cells, minds, and social agents. Yet beneath this disciplinary diversity lies a deeper unity. This paper presents an empirically grounded structural ontology showing that the universe organizes itself through four Simple Interaction Rules (SIRs): Distinctions, Systems, Relationships, and Perspectives (DSRP). Using the Unified Calculus of Organization, we show that these operators are not independent categories but projections of one generative polarity expressed through eight elemental roles and three universal dynamics (4–8–3). These structures appear empirically in quantum state boundaries, molecular bonding symmetries, cellular architectures, ecological networks, social systems, and human cognition. They constitute the structural grammar of reality. Two lines of scientific evidence support this claim. First, existence evidence demonstrates that the DSRP invariants—polarity, part–whole recursion, relational interaction, and perspectival orientation—are instantiated in nature independent of human observers. Second, effect evidence shows that human cognition uses the same invariants but with measurable structural bias. When these biases are corrected, individuals exhibit large and replicable gains in systems thinking, metacognition, causal reasoning, and model accuracy. In an age dominated by large language models, this structural ontology becomes more than a philosophical insight—it becomes a practical necessity. Current LLM-based AI systems lack internal ontology: they do not represent boundaries, part–whole structure, causal or conceptual relationships, or perspectives. As a result, they hallucinate, contradict themselves, collapse distinctions, fabricate causal chains, and drift semantically over time. Without structure, there is no coherence. By formalizing the generative grammar of reality and cognition, the ontology advanced here provides the missing structural foundation that both humans and AI systems require to think, model, and understand reliably. It clarifies why humans reason the way they do, why AI systems fail the way they do, and why a shared structural framework—one that makes thinking visible, measurable, and governable—is essential for science, education, and human–AI intelligence. DSRP is not a theory of systems; it is the structural ontology that makes systems possible—and the generative grammar future human and artificial intelligence will require.

As China rapidly advances in AI innovation and development, especially in frontier AI, its regulatory and ethical frameworks are under increasing pressure to ensure that technological progress aligns with human interests and societal values. This Article argues that AI value alignment—the process of ensuring AI systems act in accordance with human values, norms, and ethical principles—should be adopted as a strategic pillar in China’s evolving AI governance architecture. While China has already established a comprehensive legal, ethical, and self-regulatory landscape to address AI risks, these mechanisms often rely on reactive enforcement and external compliance. In contrast, AI value alignment offers a proactive, intrinsic approach that embeds safety and ethical constraints directly into AI systems, making them safer, more trustworthy, and responsive to human needs. This study begins by mapping China’s current AI governance landscape, including national legislation such as the Cybersecurity Law, Personal Information Protection Law, and a growing set of regulations targeted at algorithms and generative AI. It also evaluates China’s normative commitments, such as the “human-centric” and “tech for good” principles articulated in national policy documents, and the increasing role of corporate self-regulation among major technology firms. While commendable in scope and ambition, these governance mechanisms often fall short in ensuring that AI behavior aligns with safety constraints and ethical intent—particularly when AI systems (such as agentic AI) become more autonomous and capable. This gap highlights the urgent need for a systematic value alignment strategy. The Article then delves into the conceptual and technical foundations of AI value alignment, identifying both engineering challenges—such as reward misspecification, data bias, and model deception—and normative dilemmas, including moral pluralism, value aggregation, and dynamic ethics. Special attention is paid to frontier models like large language models and artificial general intelligence (AGI), which pose alignment challenges at a scale previously unseen. Drawing on contemporary alignment techniques such as RLHF (Reinforcement Learning from Human Feedback) and principle-based alignment, such as Anthropic’s Constitutional AI, the Article explores their limitations and calls for a more diversified, interdisciplinary, and forward-looking alignment research agenda. Finally, the Article offers a roadmap for operationalizing AI value alignment across three key governance domains: Law and regulation, ethical norms, and industry self-regulation. Recommendations include the incorporation of alignment assessments into regulatory filings, the development of technical standards for value alignment and ethics-by-design guidelines, and institutional investments in safety and alignment research. The Article concludes by asserting that value alignment is not merely a technical safeguard but a governance imperative for the age of autonomous AI and agentic AI. By integrating alignment into its AI governance strategy, China can not only enhance domestic safety and public trust but also better coordinate with global AI ethics and safety initiatives—ultimately contributing to the shared goal of human-aligned and beneficial artificial intelligence.