Efficient Thought Space Exploration Through Strategic Intervention

The rise of Large Language Model (LLM) chatbots, such as ChatGPT and Gemini, has revolutionized how we access information. These LLMs can answer a wide array of questions on nearly any topic. When humans answer questions, especially difficult or uncertain questions, they often accompany their responses with metacognitive confidence judgments indicating their belief in their accuracy. LLMs are certainly capable of providing confidence judgments, but it is currently unclear how accurate these confidence judgments are. To fill this gap in the literature, the present studies investigate the capability of LLMs to quantify uncertainty through confidence judgments. We compare the absolute and relative accuracy of confidence judgments made by four LLMs (ChatGPT, Bard/Gemini, Sonnet, Haiku) and human participants in both domains of aleatory uncertainty-NFL predictions (Study 1; n = 502) and Oscar predictions (Study 2; n = 109)-and domains of epistemic uncertainty-Pictionary performance (Study 3; n = 164), Trivia questions (Study 4; n = 110), and questions about life at a university (Study 5; n = 110). We find several commonalities between LLMs and humans, such as achieving similar levels of absolute and relative metacognitive accuracy (although LLMs tend to be slightly more accurate on both dimensions). Like humans, we also find that LLMs tend to be overconfident. However, we find that, unlike humans, LLMs-especially ChatGPT and Gemini-often fail to adjust their confidence judgments based on past performance, highlighting a key metacognitive limitation.

Token-level steering has emerged as a pivotal approach for inference-time alignment, enabling fine grained control over large language models by modulating their output distributions without parameter updates. While effective, existing methods rely on dense intervention at every decoding step. This persistent manipulation not only incurs substantial computational overhead but also risks compromising generation quality by excessively drifting from the model's intrinsic distribution. In this work, we show that dense intervention is unnecessary and propose Sparse Inference time Alignment (SIA), which performs sparse junction steering by intervening only at critical decision points along the generation trajectory. Our key insight is that high entropy junctions mark pivotal decision points in the generation trajectory and are particularly susceptible to misalignment, indicating the need to introduce alignment related reward signals at these points. Extensive experiments across different model families and alignment objectives show that steering only 20% to 80% of tokens achieves superior alignment-efficiency trade offs. For strong base models such as Qwen3, intervening on as few as 20% of tokens matches or even surpasses heavily post-trained instruct models. This sparsity enables stronger guidance while better preserving the model's native distribution, integrates seamlessly with search based methods such as Best-of-N, and reduces computational cost by up to 6x.

PurposeCurrent benchmarks evaluating large language models (LLMs) in ophthalmology are narrow and disproportionately prioritize accuracy. We introduce BEnchmarking LLMs for Ophthalmology (BELO), a standardized evaluation benchmark developed through multiple rounds of expert checking by 13 ophthalmologists. BEnchmarking LLMs for Ophthalmology assesses ophthalmology-related knowledge and reasoning quality.SubjectsThis study did not involve human participation.DesignCross-sectional study.MethodsUsing keyword matching and a fine-tuned PubMed Bidirectional Encoder Representations from Transformers model, we curated ophthalmology-specific multiple-choice questions (MCQs) from diverse medical data sets (Basic and Clinical Science Course [BCSC], Multi-Subject Multi-Choice Dataset for Medical domain [MedMCQA], Medical Question Answering [MedQA], Biomedical Semantic Indexing and Question Answering [BioASQ], and PubMed Question Answering [PubMedQA]). The data set underwent multiple rounds of expert checking. Duplicate and substandard questions were systematically removed. Ten ophthalmologists refined the explanations of each MCQ's correct answer. This was further adjudicated by 3 senior ophthalmologists. To illustrate BELO's utility, we evaluated 8 LLMs (OpenAI o1, o3-mini, GPT-5, GPT-4o, DeepSeek-R1, MedGemma-4B, Llama-3-8B, and Gemini 1.5 Pro).Main Outcome MeasuresThe 8 LLMs were evaluated in terms using accuracy, macro-F1, and 5 text-generation metrics (Recall-Oriented Understudy for Gisting Evaluation, BERTScore, BARTScore, Metric for Evaluation of Translation with Explicit Ordering, and AlignScore). In a further evaluation involving human experts, 2 ophthalmologists qualitatively reviewed 50 randomly selected outputs for accuracy, comprehensiveness, and completeness.ResultsBEnchmarking LLMs for Ophthalmology consists of 900 high-quality, expert-reviewed questions aggregated from 5 sources: BCSC (260), BioASQ (10), MedMCQA (572), MedQA (40), and PubMedQA (18). To demonstrate BELO's utility, we conducted a series of benchmarking exercises. In the quantitative evaluation, GPT-5 achieved the highest accuracy (0.90, 95% confidence interval [CI]: 0.89–0.92) and macro-F1 score (0.91, 95% CI: 0.89–0.93). On the other hand, the models' performance on text-generation metrics varied and were generally suboptimal, with scores ranging from 20.4 to 72.0 (out of 100, excluding the BARTScore metric), indicating room for improvement in clinical reasoning. In expert evaluations, GPT-4o was rated highest for accuracy and readability, while Gemini 1.5 Pro scored highest for completeness. A public leaderboard has been established to promote transparent evaluation and reporting. Importantly, the BELO data set will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models.ConclusionsBEnchmarking LLMs for Ophthalmology provides a robust clinically relevant benchmark for evaluating both the accuracy and reasoning capabilities of current and emerging LLMs in ophthalmology. Future BELO benchmarking efforts will expand to include vision-based question answering and clinical scenario management tasks.Financial Disclosure(s)The authors have no proprietary or commercial interest in any materials discussed in this article.

IntroductionMarine data is typically large-scale and complex, requiring effective visualization recommendation systems for data filtering and value extraction. The primary challenge in visualization automatic recommendation lies in the conflict between the inherent ambiguity of user intentions and the limitations of precise interaction methods. In the initial phase, users often lack well-defined analytical goals for the dataset, necessitating a cold-start and iterative interaction to clarify their goals. Moreover, although existing interaction methods are diverse, their precise control fails to effectively convey users' ambiguous intentions.MethodsTo address these issues, we introduce a novel cold-start visualization recommendation system that integrates a Large Language Model (LLM) and a Grammar Variational Autoencoder (GVAE). The LLM generates initial exploratory goals and visualization recommendations based on data descriptions, while the GVAE produces visual summary projections to verify the extent of user intent fulfillment. Additionally, users can roll back to previous record point to establish new analytical paths. This forms a comprehensive analysis framework for observation, reasoning, and backtracking. Users can adjust their exploration goals and refine their intent expressions based on projections through the LLM, iterating until the analysis is complete. The GVAE analyzes chart correlations and latent patterns, while the LLM converts ambiguous intentions into precise representations, with both working together to address the cold-start problem.Results and discussionThe effectiveness of this method in cold-start visualization recommendations and semantic-driven interactions has been validated through case studies and evaluations.

Most if not all twin engine Transport Category helicopters are designed to FAR Part 29 Category standards, but few are actually certificated for Transport Category Operations. Although recent FAR Airworthiness and Operations legislation sets minimum one engine inoperative (OEI) enroute performance standards, helicopter operators are still not required to follow FAA Certified Category takeoff and landing procedures, and it is doubtful many operators utilize helicopter manufacturers' published procedures which habe historically been inflexible with respect to optimization of payload and field lengths. This paper discusses the development and qualification of a Transport Category takeoff and landing procedure for the S-76B helicopter which provides the flexibility necessary to encourage helicopter operators to use the Rotorcraft Flight Manual zero exposure time takeoff and landing procedures thus improving overall safety. The takeoff procedure features variable critical decision point (CDP) and takeoff safety speed (V2). Rejected and continued takeoff distances are directly proportional to CDP and V2 speeds respectively, therefore lower CDP and V2 speeds equate to shorter field lengths. Maximum takeoff and landing gross weight is also directly proportional to CDP/V2 speeds, therefore higher CDP/V2 speeds can equate to higher maximum takeoff gross weights. This coupled with published headwind influence permits payload to be traded off against available field length in such a manner that Category OEI climb performance minima are maintained over a wide range of environmental conditions. The landing procedure features two different approach profiles. The primary landing approach path is intended for VFR approaches to airfields with minimum available space, with an alternate approach path providing a more shallow profile to address transition from instrument approach or where landing space is not critically short. A reduced gross weight and power single engine pilot training procedure was developed whereby the basic aircraft response and handling characteristics are preserved to provide a realistic training scenario by duplicating the all-up Category WAT limited power/weight ratio, but with the added safety advantage inherent when operating at a reduced gross weight. Timely development/qualification of this procedure was facilitated by newly developed real-time telemetry capabilities also discussed, Copyright @ American Institute of Aeronautics and Astronautics, Inc., 198f. All rights resewed. which include a microwave ranging aircraft position and tracking system (APATS) and CRT displays which permitted instantaneous flight maneuver validation and pilot technique modification decisioning. Also discussed are the potential for further technique enhancements for added operational flexibility, and the adaptation of the FAA certified procedure to CAA Group operations including the effect on paylbad segment APATS BCAR BL CAA CDP CG CRT CTO EAPS FADEC FAA FM FPM Gross IFR IGE ILS LDP MCP Net N1 OEI PCM Q RFM RTO T5 V ~ P P Vblss VFR VY Vl v2 Av WAT ofthe more stringkt 1st and 2nd CAA OEI climb performance minima.

The task of long-term action anticipation demands solutions that can effectively model temporal dynamics over extended periods while deeply understanding the inherent semantics of actions. Traditional approaches, which primarily rely on recurrent units or Transformer layers to capture long-term dependencies, often fall short in addressing these challenges. Large Language Models (LLMs), with their robust sequential modeling capabilities and extensive commonsense knowledge, present new opportunities for long-term action anticipation. In this work, we introduce the ActionLLM framework, a novel approach that treats video sequences as successive tokens, leveraging LLMs to anticipate future actions. Our baseline model simplifies the LLM architecture by setting future tokens, incorporating an action tuning module, and reducing the textual decoder layer to a linear layer, enabling straightforward action prediction without the need for complex instructions or redundant descriptions. To further harness the commonsense reasoning of LLMs, we predict action categories for observed frames and use sequential textual clues to guide semantic understanding. In addition, we introduce a Cross-Modality Interaction Block, designed to explore the specificity within each modality and capture interactions between vision and textual modalities, thereby enhancing multimodal tuning. Extensive experiments on benchmark datasets demonstrate the superiority of the proposed ActionLLM framework, encouraging a promising direction to explore LLMs in the context of action anticipation.

How can we accelerate large language models (LLMs) without sacrificing accuracy? The slow inference speed of LLMs hinders us to benefit from their remarkable performance in diverse applications. This is mainly because numerous sublayers are stacked together in LLMs. Sublayer pruning compresses and expedites LLMs via removing unnecessary sublayers. However, existing sublayer pruning algorithms are limited in accuracy since they naively select sublayers to prune, overlooking the different characteristics of each sublayer. In this paper, we propose SPRINT (Sublayer Pruning with Latency and Tunability Information), an accurate sublayer pruning method for LLMs. SPRINT accurately selects a target sublayer to prune by considering 1) the amount of latency reduction after pruning and 2) the tunability of sublayers. SPRINT iteratively prunes redundant sublayers and swiftly tunes the parameters of remaining sublayers. Experiments show that SPRINT achieves the best accuracy-speedup trade-off, exhibiting up to 23.88%p higher accuracy on zero-shot commonsense reasoning benchmarks compared to existing pruning algorithms.

The recent developments in artificial intelligence (AI), particularly in light of the impressive capabilities of transformer-based Large Language Models (LLMs), have reignited the discussion in cognitive science regarding whether computational devices could possess semantic understanding or whether they are merely mimicking human intelligence. Recent research has highlighted limitations in LLMs’ reasoning, suggesting that the gap between mere symbol manipulation (syntax) and deeper understanding (semantics) remains wide open. While LLMs overcome certain aspects of the symbol grounding problem through human feedback, they still lack true semantic understanding, struggling with common-sense reasoning and abstract thinking. This paper argues that while adding sensory inputs and embodying AI through sensorimotor integration with the environment might enhance its ability to connect symbols to real-world meaning, this alone would not close the gap between syntax and semantics. True meaning-making also requires a connection to subjective experience, which current AI lacks. The path to AGI must address the fundamental relationship between symbol manipulation, data processing, pattern matching, and probabilistic best guesses with true knowledge that requires conscious experience. A transition from AI to AGI can occur only if it possesses conscious experience, which is closely tied to semantic understanding. Recognition of this connection could furnish new philosophical insights into longstanding practical and philosophical questions for theories in biology and cognitive science and provide more meaningful tests of intelligence than the Turing test.

Mathematical reasoning ability objectively reflects a language model's understanding of implicit knowledge in contexts, with logic being a prerequisite for exploring, articulating and establishing effective reasoning. Large language models (LLMs) have shown great potential in complex reasoning tasks represented by mathematical reasoning. However, existing mathematical datasets either focus on commonsense reasoning, assessing the model's knowledge application ability, or arithmetic problems with fixed calculation rules, evaluating the model's rapid learning capability. There is a lack of datasets that require solving problems solely through logical reasoning. As a result, the performance of LLMs in accurately understanding the implicit logical relationships in problems and deriving conclusions based solely on given conditions is hindered. To address this challenge, we construct a dataset specifically for multiple step reasoning tasks: Reasoning-Math (RMath). This dataset focuses on evaluating logical reasoning abilities with mathematical reasoning problems, covering typical problem types, including direct reasoning problems, hypothetical reasoning problems, and nested reasoning problems. Additionally, we design a standardized annotation scheme that transforms natural language descriptions of conditions into formal propositions. Other annotation contents include problem categories, proposition truth values, and proposition relationship types. This not only reduces biases caused by semantic misunderstandings during problem-solving, but also facilitates the incorporation of theoretically grounded logical reasoning methods to enhance reasoning abilities. Furthermore, we propose a normalization problem-solving framework based on propositional logic for RMath and design the problem-solving process for prompt tuning to guide LLMs to absorb mathematical logical theories and improving reasoning abilities. Finally, we evaluate RMath on several popular LLMs and present the corresponding results.

<p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d624655e60"> <b>Objectives:</b> Manual implementation of pharmacometric analyses requires significant expertise, is time-intensive, and prone to inconsistencies across analysts. While large language models (LLMs) show promise for automation, their direct application lacks the stability and validation required for regulatory submissions. We developed a novel multi-agent framework that combines structured constraints with iterative refinement to generate high quality pharmacometric analyses while maintaining transparency and reproducibility. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d624655e65"> <b>Methods:</b> Our tool employs an agent-based LLM architecture: 1) An agent that reads and processes analysis plans using AWS bedrock hosted LLMs to identify required analyses and success criteria with structured output constraints; 2) An agent that iteratively generates and validates R code with built-in quality checkpoints, applying human-like reasoning for error resolution and code refinement; and 3) A systematic validation framework that executes generated code, provides feedback, and documents the complete analytical thought process. The framework includes automated handling of data validation, population filtering, missing data patterns, and model diagnostics. The system was evaluated using example pharmacometric analyses from multiple therapeutic areas. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d624655e70"> <b>Results:</b> The framework successfully automated complex analyses while maintaining rigorous submission quality standards. The first agent achieved &gt;95% accuracy in identifying required analyses from analysis plans. Generated code included comprehensive data validation, proper population filtering, and standardized output formats. The iterative refinement process enabled stable convergence to valid solutions within 8-10 cycles on average. Quality control checkpoints effectively identified potential issues in data handling, model specification, and results interpretation. The system maintained complete reproducibility through detailed documentation of analysis decisions and systematic version control. Example applications demonstrated successful automation of analyses like pharmacokinetic-pharmacodynamic characterization and exposure-response relationships. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d624655e75"> <b>Conclusions:</b> This work represents a significant advance in automating pharmacometric analyses by combining the flexibility of LLMs with rigorous quality requirements. The multi-agent architecture with iterative refinement enables reliable automation while maintaining transparency and reproducibility. This approach has broad applications in clinical pharmacology and could significantly accelerate drug development while improving analysis consistency. The framework's ability to generate high quality, well-documented analyses with built-in QC steps provides a scalable solution for standardizing pharmacometric workflows.

Large Language Models (LLMs) have demonstrated revolutionary potential in autonomous systems. Studies have demonstrated that LLMs can be leveraged to support various robotic operations including visual navigation, planning, and drone controls. However, when facing tasks with complex reasoning, concerns and challenges are raised about the reliability of solutions produced by LLMs. This thesis proposed a closed-loop reasoning approach to enhance the reliability of LLM-driven drone control. The proposed method introduces a structured prompt framework designed with Guidelines, Skills, Constraints, and Examples integrated with a closed-loop feedback mechanism that iteratively refines the generated code based on its execution outcomes. The framework is featured by iteratively refining LLM-generated solutions, thereby improving reasoning quality and control reliability. We performed thorough experiments of our approach for drone control with a wide range of task complexities. The experiment result demonstrates that our framework significantly improves task success rates and completeness compared to baseline approaches, highlighting its potential for reliable LLM-driven autonomous drone systems.

Edge deployments of large language models (LLMs) often suffer from significant latency due to the overhead of high-level client runtimes on resource-constrained hardware. To address this challenge, we conducted a side-by-side performance analysis of four quantised LLMs – Llama 3.2:1b, Gemma 3:1b, Granite 3.1-MoE:1b, and Qwen 2.5:0.5b – on a Raspberry Pi 4 Model B (8 GB LPDDR4, quad-core ARM Cortex-A72) using both Python and Rust API clients. Each model was served via a local Ollama inference server, and a fixed suite of twenty prompts – covering factual retrieval, arithmetic reasoning, translation, code synthesis, and creative generation – was executed sequentially with a two-second inter-request delay, yielding 160 measurements per client. Rust markedly reduces cold-start delays: mean model load times fall from 1 648.7 ms (Python) to 52.8 ms (Rust) for Llama 3.2:1b, and from 607.0 ms to 171.3 ms for Qwen 2.5:0.5b. Corresponding end-to-end latencies decrease by 1.4-2.0 s across models. In warm-start conditions, both clients deliver nearly identical decoding throughput – ≈2.7 tokens/s for Llama 3.2:1b, 4.4 tokens/s for Gemma 3:1b, 7.4 tokens/s for Granite 3.1-MoE, and 8.6 tokens/s for Qwen 2.5:0.5b – indicating that runtime overhead is negligible once models are loaded. Rigorous statistical testing, including paired t-tests, Mann-Whitney U tests, and bootstrap confidence intervals, confirms that Rust’s coldstart advantages are highly significant (p &lt; 0.01). At the same time, throughput differences in steady-state inference are not statistically meaningful. We discuss limitations in platform specificity, quantisation approaches, and prompt diversity, and outline future work on heterogeneous accelerators, adaptive scheduling, and ondevice fine-tuning. Finally, we highlight practical applications in smart agriculture, healthcare monitoring, industrial IoT, autonomous robotics, and offline educational tools. This benchmark furnishes actionable guidelines for selecting client languages and quantised models in edge AI scenarios.

<title>Abstract</title> While continued pretraining has been shown to improve many-shot in-context learning (ICL) in large language models, the underlying representation dynamics that support task generalization remain insufficiently explored. This paper introduces a task-conditioned continued pretraining strategy that enhances ICL by explicitly disentangling task-specific and task-invariant representations during pretraining. The method augments standard language modeling objectives with lightweight task-conditioning signals derived from latent task clusters inferred using contrastive embedding similarity. The model is pretrained on a 150-billion-token mixed-domain corpus containing over 3,200 instruction-defined tasks, with each training sequence incorporating up to 128 demonstrations. Empirical evaluations across arithmetic reasoning, code generation, and information extraction tasks indicate that the proposed approach yields consistent gains in many-shot ICL performance, achieving up to 9.3% accuracy improvement over baseline continued pretraining methods. Representation probing further shows a 17% increase in task-separability scores while preserving general linguistic coherence. These findings suggest that task-conditioned representation adaptation during continued pretraining provides a scalable and data-efficient pathway for improving many-shot in-context learning across heterogeneous task distributions.

ABSTRACTGenerative AI including large language models (LLMs) has recently gained significant interest in the geoscience community through its versatile task‐solving capabilities including programming, arithmetic reasoning, generation of sample data, time‐series forecasting, toponym recognition, or image classification. Existing performance assessments of LLMs for spatial tasks have primarily focused on ChatGPT, whereas other chatbots received less attention. To narrow this research gap, this study conducts a zero‐shot correctness evaluation for a set of 76 spatial tasks across seven task categories assigned to four prominent chatbots, that is, ChatGPT‐4, Gemini, Claude‐3, and Copilot. The chatbots generally performed well on tasks related to spatial literacy, GIS theory, and interpretation of programming code and functions, but revealed weaknesses in mapping, code writing, and spatial reasoning. Furthermore, there was a significant difference in the correctness of results between the four chatbots. Responses from repeated tasks assigned to each chatbot showed a high level of consistency in responses with matching rates of over 80% for most task categories in the four chatbots.

Large language models (LLMs) sometimes demonstrate poor performance on knowledgeintensive tasks, commonsense reasoning is one of them.Researchers typically address these issues by retrieving related knowledge from knowledge graphs or employing selfenhancement methods to elicit knowledge in LLMs.However, noisy knowledge and invalid reasoning issues hamper their ability to answer questions accurately.To this end, we propose a novel method named eLiciting, fIltering and iNtegrating Knowledge in large languagE moDel (LINKED).In it, we design a reward model to filter out the noisy knowledge and take the marginal consistent reasoning module to reduce invalid reasoning.With our comprehensive experiments on four complex commonsense reasoning benchmarks, our method outperforms SOTA baselines (up to 9.0% improvement of accuracy).Besides, to measure the positive and negative impact of the injected knowledge, we propose a new metric called effectiveness-preservation score for the knowledge enhancement works.Finally, through extensive experiments, we conduct an in-depth analysis and find many meaningful conclusions about LLMs in commonsense reasoning tasks.