OmniTester: Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

Handling long tail corner cases is a major challenge faced by autonomous vehicles (AVs). While large language models (LLMs) hold great potentials to handle the corner cases with excellent generalization and explanation capabilities and received increasing research interest on application to autonomous driving, there are still technical barriers to be tackled, such as strict model performance and huge computing resource requirements of LLMs, which are difficult to be met locally at AVs. In this paper, we investigate a new approach of applying remote or edge LLMs to support autonomous driving. With this approach connected autonomous vehicles (CAVs) send driving assistance requests to the LLMs. LLMs deployed at the edge of the networks or remote clouds process the requests and generate driving assistance instructions for the CAVs. A key issue for such LLM assisted driving system is the assessment of LLMs on their understanding of driving theory and skills, ensuring they are qualified to undertake safety critical driving assistance tasks for CAVs. As there is no published work on assessing LLM of driving theory and skills, we design and run driving theory tests for several proprietary LLM models (OpenAI GPT models, Baidu Ernie and Ali QWen) and open-source LLM models (Tsinghua MiniCPM-2B and MiniCPM-Llama3-V2.5) with more than 500 multiple-choices theory test questions. These questions are close to the official UK driving theory test ones. Model accuracy, cost and processing latency are measured from the experiments. Experiment results show that while model GPT-4 passes the test with improved domain knowledge and Ernie has an accuracy of 85% (just below the 86% passing threshold), other LLM models including GPT-3.5 fail the test. For the test questions with images, the multimodal model GPT4-o has an excellent accuracy result of 96%, and the MiniCPM-Llama3-V2.5 achieves an accuracy of 76%. While GPT-4 holds stronger potential for CAV driving assistance applications, the cost of using model GPT4 is much higher, almost 50 times of that of using GPT3.5. The results can help make decision on the use of the existing LLMs for CAV applications and balancing on the model performance and cost.

This study aimed to evaluate the diagnostic accuracy of multimodal large language models in classifying superior labial frenulum attachments from intraoral photographs using expert consensus as the reference standard. Five experts (two periodontists and three orthodontists) established the consensus standard by classifying frenulum attachments in 117 intraoral images as mucosal, gingival, papillary, and papilla penetrating. The same photographs were then presented to three multimodal large language models (ChatGPT 4o, Gemini 2.5 Pro, and Microsoft Copilot GPT-4), and their diagnostic performance was evaluated using accuracy, sensitivity, specificity, and F1 score. Reliability was assessed using Fleiss' and Cohen's Kappa, and diagnostic performances were compared using Cochran's Q test. Human raters demonstrated almost perfect agreement (κ = 0.838, p < 0.001), whereas large language models showed poor inter-model agreement (κ = -0.124, p < 0.001). ChatGPT achieved slight significant agreement with the consensus (κ = 0.114, p = 0.019), although its clinical relevance was negligible. Gemini (κ = 0.099) and Copilot (κ = 0.027) showed no significant agreement (p > 0.05). Copilot yielded the highest overall accuracy (46.2%), followed by Gemini (44.5%) and ChatGPT (35.0%). The performance of the large language models varied across frenulum types. Current multimodal large language models demonstrate inconsistent and clinically insufficient accuracy in the classification of superior labial frenulum attachments from photographs. Domain-specific training is essential before large language models can be considered reliable diagnostic tools in dentistry.

Accurate geospatial data are essential for intelligent transportation systems and automated reporting applications, as location precision directly impacts safety analysis and decision-making. GPS devices are now routinely employed by law enforcement officers when filing vehicle crash reports, yet our investigation reveals that significant data quality issues persist. The high apparent precision of GPS coordinates belies their actual accuracy as we find that approximately 20% of crash sites need correction—results consistent with existing research. To address this challenge, we present a novel credibility scoring and correction algorithm that leverages a state-of-the-art multimodal large language model (LLM) capable of integrated visual and textual reasoning. Our framework synthesizes information from structured coordinates, crash diagrams, and narrative text, employing advanced artificial intelligence techniques for comprehensive geospatial validation. In addition to the LLM, our system incorporates open geospatial data from Overture Maps, an emerging collaborative mapping initiative, to enhance the spatial accuracy and robustness of the validation process. This solution was developed as part of research leading to a patent for autonomous vehicle routing systems that require high-precision crash location data. Applied to a dataset of 5000 crash reports, our approach systematically identifies records with location discrepancies requiring correction. By uniting the latest developments in multimodal AI and open geospatial data, our solution establishes a foundation for intelligent data validation in electronic reporting systems, with broad implications for automated infrastructure management and autonomous vehicle applications.

AI-enabled language models (LMs) to large language models (LLMs) and multimodal large language models (MLLMs) in drug discovery and development

The swift development of digital technologies, especially Large Language Models (LLMs), offers a revolutionary opportunity to improve urban mobility. LLMs are used widely in various areas, like traffic prediction, autonomous vehicle (AV) communication, and customized trip assistance. General speaking, users now consider the integration of LLMs in mobility to be intelligent friends. The study's primary objective is to demonstrate how an integrated LLM in mobility services can significantly increase network efficiency, lessen congestion, and improve each traveler's experience. Practically speaking, deploying a suitable LLM for providing various services to users is imperative. Additionally, this process is an endeavor due to uncertainty and contradictory criteria. To provide a solid technique for evaluating LLMs in mobility applications, this study attempts to suggest a novel evaluation framework that combines uncertainty theory and Multi-Criteria Decision-Making (MCDM). Hence, entropy and weighted sum model (WSM) of MCDM techniques with Triangular Neutrosophic Sets (TriNSs) of uncertainty theory to support the expert panel in vague situations.

The task of long-term action anticipation demands solutions that can effectively model temporal dynamics over extended periods while deeply understanding the inherent semantics of actions. Traditional approaches, which primarily rely on recurrent units or Transformer layers to capture long-term dependencies, often fall short in addressing these challenges. Large Language Models (LLMs), with their robust sequential modeling capabilities and extensive commonsense knowledge, present new opportunities for long-term action anticipation. In this work, we introduce the ActionLLM framework, a novel approach that treats video sequences as successive tokens, leveraging LLMs to anticipate future actions. Our baseline model simplifies the LLM architecture by setting future tokens, incorporating an action tuning module, and reducing the textual decoder layer to a linear layer, enabling straightforward action prediction without the need for complex instructions or redundant descriptions. To further harness the commonsense reasoning of LLMs, we predict action categories for observed frames and use sequential textual clues to guide semantic understanding. In addition, we introduce a Cross-Modality Interaction Block, designed to explore the specificity within each modality and capture interactions between vision and textual modalities, thereby enhancing multimodal tuning. Extensive experiments on benchmark datasets demonstrate the superiority of the proposed ActionLLM framework, encouraging a promising direction to explore LLMs in the context of action anticipation.

The Ministry of Food and Drug Safety (MFDS) of the Republic of Korea, similar to the United States Food and Drug Administration and the United Kingdom's Medicines and Healthcare products Regulatory Agency, issued specific regulatory guidelines on January 24, 2025, for approving generative artificial intelligence (AI) technologies as medical devices [1].Although these guidelines use the term 'generative AI,' they predominantly focus on the approval of AI software tools based on large language models (LLMs) and large multimodal models (LMMs) [2], the latter of which can process various types of input data, such as texts, images, videos, audio, and bio-signals.Unlike the broader guidelines for approving AI models as medical devices, specific regulatory guidelines for LLMs/LMMs have arguably not yet been proposed in other countries.

Background: Multimodal large language models (LLMs) are increasingly being explored as clinical support tools, yet their capacity for orthodontic biomechanical reasoning has not been systematically evaluated. This retrospective study assessed their ability to analyze treatment mechanics and explored their potential role in supporting orthodontic decision-making. Methods: Five publicly available models (GPT-o3, Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.0, and Grok) analyzed 56 standardized intraoral photographs illustrating a diverse range of active orthodontic force systems commonly encountered in clinical practice. Three experienced orthodontists independently scored the outputs across four domains—observation, interpretation, biomechanics, and confidence—using a 5-point scale. Inter-rater agreement and consistency were assessed, and statistical comparisons were made between models. Results: GPT-o3 achieved the highest composite score (3.34/5.00; 66.8%), significantly outperforming all other models. The performance ranking was followed by Claude (57.8%), Gemini (52.6%), GPT-4.0 (48.8%), and Grok (38.8%). Inter-rater reliability among the expert evaluators was excellent, with ICC values ranging from 0.786 (Confidence Evaluation) to 0.802 (Observation). Model self-reported confidence showed poor calibration against expert-rated output quality. Conclusions: Multimodal LLMs show emerging potential for assisting orthodontic biomechanical assessment. With expert-guided validation, these models may contribute meaningfully to clinical decision support across diverse biomechanical scenarios encountered in routine orthodontic care.

A previous study investigated the application of Large Language Models (LLM) in Agent-Based Simulations for System of Systems studies, where the LLM acted as an Incident Commander during a wildfire event. Firefighting Concept of Operations were utilized as an initial prompt to limit the LLM's answers. The findings were promising, indicating that LLMs may effectively drive simulation agents. Building on these insights, the goal is to integrate an LLM directly into a particular agent's decision-making loop. The present work will investigate whether LLMs perform well in cyber-physical simulation environment. The LLM will be used with a cyber-physical simulation to fly an Unmanned Aerial Vehicle and study its adherence to mission objectives. The LLM's judgments and the computational resources utilized throughout the simulation will be discussed with special interest in future improvements and implications for designing autonomous aerial firefighting systems.

A previous study investigated the application of Large Language Models (LLM) in Agent-Based Simulations for System of Systems studies, where the LLM acted as an Incident Commander during a wildfire event. Firefighting Concept of Operations were utilized as an initial prompt to limit the LLM's answers. The findings were promising, indicating that LLMs may effectively drive simulation agents. Building on these insights, the goal is to integrate an LLM directly into a particular agent's decision-making loop. The present work will investigate whether LLMs perform well in cyber-physical simulation environment. The LLM will be used with a cyber-physical simulation to fly an Unmanned Aerial Vehicle and study its adherence to mission objectives. The LLM's judgments and the computational resources utilized throughout the simulation will be discussed with special interest in future improvements and implications for designing autonomous aerial firefighting systems.

The rise of Large Language Models (LLMs) has impacted research in robotics and automation. While progress has been made in integrating LLMs into general robotics tasks, a noticeable void persists in their adoption in more specific domains such as surgery, where critical factors such as reasoning, explainability, and safety are paramount. Achieving autonomy in robotic surgery, which entails the ability to reason and adapt to changes in the environment, remains a significant challenge. In this work, we propose a multi-modal LLM integration in robot-assisted surgery for autonomous blood suction. The reasoning and prioritization are delegated to the higher-level task-planning LLM, and the motion planning and execution are handled by the lower-level deep reinforcement learning model, creating a distributed agency between the two components. As surgical operations are highly dynamic and may encounter unforeseen circumstances, blood clots and active bleeding were introduced to influence decision-making. Results showed that using a multi-modal LLM as a higher-level reasoning unit can account for these surgical complexities to achieve a level of reasoning previously unattainable in robot-assisted surgeries. These findings demonstrate the potential of multi-modal LLMs to significantly enhance contextual understanding and decision-making in robotic-assisted surgeries, marking a step toward autonomous surgical systems.

We present a systematic evaluation of modern multimodal large language models (LLMs) for the classification of mean-motion and secular resonances from images of resonant arguments. Four benchmark datasets (RB-TEST, RB-PILOT, RB-SMALL, RB-FULL) were constructed to cover clear, ambiguous, and transient cases, with both binary and three-class outputs. Using standardized prompts (a full prompt for large models and a simplified variant for small models that cannot process complex instructions), we tested flagship commercial models, large open-source models, and small locally runnable models. Commercial LLMs reach [Formula: see text] on simple cases and up to [Formula: see text] on the three-class RB-SMALL dataset, while the best open-source models also reach [Formula: see text] on unambiguous cases and [Formula: see text] on the complex ones. On the full binary benchmark, open-source models approach commercial performance ([Formula: see text]-[Formula: see text]). Most errors occur in transient and resonance-sticking regimes. The results show that LLMs can perform resonance classification at levels comparable to those of classical or machine-learning methods without training or fine-tuning, and that even small open-source models achieve practically useful accuracy. The released benchmarks establish a reproducible standard for evaluating LLMs on dynamical astronomy tasks.

BackgroundDiagnosing oral lesions from benign conditions to oral cancer remains challenging due to overlapping visual features and reliance on histopathology. Large language models (LLMs) can integrate textual and visual cues, but their diagnostic accuracy and clinical utility in real decision-making contexts remain uncertain. To systematically evaluate the diagnostic performance, clinical usefulness, and limitations of LLMs in identifying oral lesions.MethodsPubMed, CINAHL, Embase, Web of Science, and Google Scholar were searched to 20 July 2025. Eligible studies applied LLMs (e.g., ChatGPT, Gemini, DeepSeek, Copilot, Claude) for diagnosis or differential diagnosis of oral lesions using text, images, or multimodal inputs. Outcomes included diagnostic accuracy, agreement metrics, and qualitative assessments of explanation quality and clinical applicability. Risk of bias was assessed using an adapted QUADAS-2. Narrative synthesis was performed due to heterogeneity.ResultsSeventeen studies (>1,200 cases) were included. Diagnostic accuracy ranged from 25%–96%, varying by model version, input modality, and lesion complexity. Multimodal inputs consistently improved performance, with Cohen's κ up to 0.85–0.90. Advanced models (GPT-4o, DeepSeek-R1, o1-preview) outperformed earlier versions and approached expert performance in some tasks, although specialists generally retained superior Top-1 accuracy. Clinical utility was highest when LLMs were used to structure differential reasoning, highlight red-flag features, and support communication, but limited in tasks requiring fine morphological interpretation or severity grading. Overall risk of bias was low to moderate.ConclusionsLLMs demonstrate variable diagnostic performance and context-dependent supportive utility as adjunctive tools in oral lesion assessment, particularly in multimodal settings. They should complement, rather than replace, expert clinical judgment. Future research should prioritize real-world workflow evaluation, standardized prompting strategies, and prospective clinical validation.Systematic Review Registrationhttps://www.crd.york.ac.uk/PROSPERO/view/CRD420251090315, identifier CRD420251090315.

Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multimodal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, LiDAR) as input and makes driving decisions and provide explanations. This model can plug-and-play in existing AD systems such as Autopilot and Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that replacing the decision-making modules of the Autopilot and Apollo with DriveMLM resulted in significant improvements of 3.2 and 4.7 points on the CARLA Town05 Long, respectively, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs.

Recently, Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have shown promise in instruction following and image understanding. While these models are powerful, they have not yet been developed to comprehend the more challenging 3D geometric and physical scenes, especially when it comes to the sparse outdoor LiDAR data. In this paper, we introduce LiDAR-LLM, which takes raw LiDAR data as input and harnesses the remarkable reasoning capabilities of LLMs to gain a comprehensive understanding of outdoor 3D scenes. The central insight of our LiDAR-LLM is the reformulation of 3D outdoor scene cognition as a language modeling problem, encompassing tasks such as 3D captioning, 3D grounding, 3D question answering, etc. Specifically, due to the scarcity of 3D LiDAR-text pairing data, we introduce a three-stage training strategy and generate relevant datasets, progressively aligning the 3D modality with the language embedding of LLM. Furthermore, we design a Position-Aware Transformer (PAT) to connect the 3D encoder with the LLM, which effectively bridges the modality gap and enhances the LLM's spatial orientation comprehension of visual features. Our experiments demonstrate that LiDAR-LLM effectively comprehends a wide range of instructions related to 3D scenes, achieving a 40.9 BLEU-1 score on the 3D captioning dataset, a Grounded Captioning accuracy of 63.1%, and a BEV mIoU of 14.3%.