Does GPT-4 surpass human performance in linguistic pragmatics?

The paper studies the problem of software code migration during the transition from the old version of the SAP ERP 6.0 system to modern ERP platforms, in particular SAP S/4HANA. The necessity of automation of the migration process using artificial intelligence methods and tools is substantiated. A comprehensive analysis of existing academic and industrial case studies confirmed the feasibility of using large language models for code migration. A number of advantages of the approach based on large language models are revealed, in particular, the ability to handle various migration scenarios. The key risks and limitations common in the enterprise environment are highlighted, including data security, privacy, high cost of API calls and limited knowledge of models regarding the subject area. A method is developed for selecting a large open source language model that can be safely used in the enterprise landscape, thereby mitigating the risks associated with cloud solutions. The method is based on the developed system of criteria, including: model size, context window size, license type, support for fine-tuning training, focus on code quality (measured using a reference dataset), support for model quantization and community support. The hybrid AHP-TOPSIS method is proposed for systematic ranking of candidate models. The AHP method was used to check the weights of the criteria, while TOPSIS was used to rank the models based on their proximity to the ideal solution. Based on the developed method, three large language models were selected: Qwen 2.5 Coder 14B, DeepSeek-Coder-V2 16B and Llama 3.1 8B. A comprehensive testing plan for the selected models was developed for further experimental studies. The scientific novelty of the work lies in the development of a method for selecting large language models for custom code migration tasks, which differs from the existing system of model selection criteria and the use of a hybrid AHP-TOPSIS approach for ranking candidate models. The practical value of the work is that the developed method allows you to select a model taking into account the constraints of the corporate environment and information security requirements, which will increase the level of automation in the migration of AVAP when switching to new versions of ERP systems from SAP SE. The developed method can be applied to the selection of large language models when upgrading any large computer systems developed in both open source and proprietary programming languages.

Large language models (LLMs) are advanced artificial intelligence (AI) tools capable of generating human-like text and are increasingly used in education, clinical care, and research. Little is known about their use within oral and maxillofacial surgery (OMFS) training. This study investigates LLM usage trends, perceived value, and educational integration among OMFS residents in the United States. A national, anonymous cross-sectional survey was distributed to OMFS residents via program directors. It gathered demographic data, LLM usage patterns, applications, perceived limitations, and attitudes toward incorporating LLMs into formal education. Eighty-one residents responded, 79.0% (64/81) reported having used an LLM, and of that group, 96.9% (62/64) use ChatGPT. 51.9% (42/81) of respondents used LLMs at least monthly in residency; however, 97.5% (79/81) reported having received no formal LLM education during residency. Residents used LLMs for clinical decision support, board preparation, research, and career planning. Free-text responses revealed a wide spectrum of views. Some advocated for curricular integration and patient education applications, while others questioned the need for formal instruction. Some respondents supported integrating LLMs into curriculums and patient education while others questioned the need for formal instruction. LLMs are used frequently by OMFS residents for a variety of purposes. As AI and LLMs become embedded in healthcare, understanding how OMFS residents interact with LLMs is vital. These findings may guide curriculum development, fostering responsible and effective use of LLMs in surgical training and practice.

Large language models (LLMs) increasingly find their way into the most diverse areas of our everyday lives. They indirectly influence people's decisions or opinions through their daily use. Therefore, understanding how and which moral judgements these LLMs make is crucial. However, morality is not universal and depends on the cultural background. This raises the question of whether these cultural preferences are also reflected in LLMs when prompted in different languages or whether moral decision-making is consistent across different languages. So far, most research has focused on investigating the inherent values of LLMs in English. While a few works conduct multilingual analyses of moral bias in LLMs in a multilingual setting, these analyses do not go beyond atomic actions. To the best of our knowledge, a multilingual analysis of moral bias in dilemmas has not yet been conducted. To address this, our paper builds on the moral machine experiment (MME) to investigate the moral preferences of five LLMs, Falcon, Gemini, Llama, GPT, and MPT, in a multilingual setting and compares them with the preferences collected from humans belonging to different cultures. To accomplish this, we generate 6500 scenarios of the MME and prompt the models in ten languages on which action to take. Our analysis reveals that all LLMs inhibit different moral biases to some degree and that they not only differ from the human preferences but also across multiple languages within the models themselves. Moreover, we find that almost all models, particularly Llama 3, divert greatly from human values and, for instance, prefer saving fewer people over saving more.

This paper investigates the utilization of large language models (LLMs) for the comprehensive control of humanoid robot locomotion. Traditional reinforcement learning (RL) approaches for robot locomotion are resource-intensive and rely heavily on manually designed reward functions. To address these challenges, we propose a method that employs LLMs as the primary designer to handle key aspects of locomotion control, such as trajectory planning, inverse kinematics solving, and reward function design. By using user-provided prompts, LLMs generate and optimize code, reducing the need for manual intervention. Our approach was validated through simulations in Unity, demonstrating that LLMs can achieve human-level performance in humanoid robot control. The results indicate that LLMs can simplify and enhance the development of advanced locomotion control systems for humanoid robots.

This paper discusses a multimodal AI system applied to legal reasoning for tax law. The results given here are very general and apply to systems developed for other areas besides tax law. A central goal of this work is to gain a better understanding of the relationships between LLMs (Large Language Models) and automated theorem-proving methodologies. To do this, we suppose (1) two cases for the theorem-proving system: one where it has a countable number of total meanings for its countable number of atoms and the other is where it has an uncountable number of total meanings for its countable number of atoms, and (2) LLMs can have an uncountable number of token meanings. With this in mind, the results given in this paper use the downward and upward Löwenheim–Skolem theorems and logical model theory to contrast these two AI modalities. One modality focuses on syntactic proofs and the other focuses on logical semantics based on LLMs. Particularly, one modality uses a rule-based first-order logic theorem-proving system to perform legal reasoning. The objective of this theorem-proving system is to provide proofs as evidence of valid legal reasoning when enacted laws are applied to particular situations. These proofs are syntactic structures that can be presented in the form of narrative explanations of how the answer to the legal question was determined. The second modality uses LLMs to analyze and transform a user’s tax query so this query can be sent to a first-order logic theorem-proving system to perform its legal reasoning function. The main goal of our application of LLMs is to enhance and simplify user input and output for the theorem-proving system. Using logical model theory, we show how there can exist an equivalence between laws represented in logic of the theorem-proving system, fixed in time when the theorem-proving system was set up, and new semantics given by LLMs. These results are based on logical model theory and Löwenheim–Skolem theorems.

The rapid advancement of large language models (LLMs) has intensified the need for effective mechanisms to transform continuous multimodal data into discrete representations suitable for language-based processing. Discrete tokenization, with vector quantization (VQ) as a central approach, offers both computational efficiency and compatibility with LLM architectures. Despite its growing importance, there is a lack of a comprehensive survey that systematically examines VQ techniques in the context of LLM-based systems. This work fills this gap by presenting the first structured taxonomy and analysis of discrete tokenization methods designed for LLMs. We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines. Beyond algorithm-level investigation, we discuss existing research in terms of classical applications without LLMs, LLM-based single-modality systems, and LLM-based multimodal systems, highlighting how quantization strategies influence alignment, reasoning, and generation performance. In addition, we identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints. Finally, we discuss emerging research directions such as dynamic and task-adaptive quantization, unified tokenization frameworks, and biologically inspired codebook learning. This survey bridges the gap between traditional vector quantization and modern LLM applications, serving as a foundational reference for the development of efficient and generalizable multimodal systems. A continuously updated version is available at: https://github.com/jindongli-Ai/LLM-Discrete-Tokenization-Survey.

As large language models (LLMs) achieve remarkable success in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and are being actively explored currently. In this article, we focus on adapting and enhancing large language models for recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation realms, i.e., LLMs fail to effectively extract useful information from a pure textual context of long user behavior sequence, even if the length of context is well below the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely, R etrieval- e nhanced L arge La nguage models Plus (ReLLaX), which provides full-stack optimization from three perspectives, i.e., data, prompt, and parameter. For data-level enhancement, we design semantic user behavior retrieval (SUBR) to reduce the heterogeneity of the behavior sequence, thus lowering the difficulty for LLMs to extract the essential information from user behavior sequences. Although SUBR can improve the data quality, further increase in the sequence length will still raise its heterogeneity to a level where LLMs can no longer comprehend it. Hence, we further propose to perform prompt-level and parameter-level enhancement, with the integration of conventional recommendation models (CRMs). As for prompt-level enhancement, we apply soft prompt augmentation (SPA) to explicitly inject collaborative knowledge from CRMs into the prompt. The item representations of LLMs are thus more aligned with recommendation, helping LLMs better explore the item relationships in the sequence and facilitating comprehension. Finally, for parameter-level enhancement, we propose component fully-interactive LoRA (CFLoRA). By enabling sufficient interaction between the LoRA atom components, the expressive ability of LoRA is extended, making the parameters effectively capture more sequence information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e., from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments are conducted on three real-world public datasets to demonstrate the superiority of ReLLaX compared with existing baseline models, as well as its capability to alleviate lifelong sequential behavior incomprehension. Our code is available. 1

Trained on vast corpora, Large Language Models (LLMs) have the potential to encode visualization design knowledge and best practices. However, if they fail to do so, they might provide unreliable visualization recommendations. What visualization design preferences, then, have LLMs learned? We contribute DracoGPT, a method for extracting, modeling, and assessing visualization design preferences from LLMs. To assess varied tasks, we develop two pipelines-DracoGPT-Rank and DracoGPT-Recommend-to model LLMs prompted to either rank or recommend visual encoding specifications. We use Draco as a shared knowledge base in which to represent LLM design preferences and compare them to best practices from empirical research. We demonstrate that DracoGPT can accurately model the preferences expressed by LLMs, enabling analysis in terms of Draco design constraints. Across a suite of backing LLMs, we find that DracoGPT-Rank and DracoGPT-Recommend moderately agree with each other, but both substantially diverge from guidelines drawn from human subjects experiments. Future work can build on our approach to expand Draco's knowledge base to model a richer set of preferences and to provide a robust and cost-effective stand-in for LLMs.

Serious Bacterial Infection (SBI) in neonates and young infants often exhibit nonspecific symptoms and clinical signs in the early stages of illness, making early diagnosis challenging. Timely recognition and appropriate treatment are essential to prevent adverse outcomes. While several clinical algorithms are widely used for SBI risk stratification, these tools have limitations, particularly low positive predictive value. This study evaluates the diagnostic accuracy of general-purpose large language models (LLMs) in detecting SBI in neonates and infants under 90 days of age admitted to the emergency department. Our objective is to improve diagnostic precision, reduce unnecessary interventions, and enhance patient outcomes. LLM performance was compared against traditional machine learning models, state-of-the-art rule-based methods, and an ensemble of physicians to assess their potential as clinical decision-support tools in scenarios of diagnostic uncertainty. On a dataset of 742 patients, LLMs demonstrated diagnostic accuracy comparable to traditional machine learning models and state-of-the-art rule-based methods. The optimized CatBoost (class-weighted) model achieved the best overall performance, with a PPV of 0.70, NPV of 0.90, sensitivity of 0.54, specificity of 0.95, F1-score of 0.60, and MCC of 0.54, outperforming the baseline CatBoost model and achieving results on par with large language models (LLMs) and physicians. When optimally prompted, LLMs performed on par with ensembles of experienced clinicians. Additionally, LLMs exhibited effective medical reasoning and provided credible diagnostic predictions, particularly valuable in cases of clinician uncertainty. The models achieved balanced performance across multiple evaluation metrics, including PPV, NPV, sensitivity, specificity, F1-score, and Matthew’s correlation coefficient (MCC). ChatGPT-4o achieved a sensitivity of 0.65 and specificity of 0.83, with an MCC of 0.41. Claude Sonnet 3.5 reached a sensitivity of 0.60 and specificity of 0.86, MCC 0.42 and Google Gemini 2.0 Flash had lower sensitivity (0.43) but the highest specificity (0.94), with an MCC of 0.43. In comparison, the best-performing individual pediatrician achieved a higher sensitivity (0.74) but lower specificity (0.68), with an MCC of 0.33, while the pediatricians’ majority vote yielded sensitivity of 0.69, specificity of 0.81, and MCC of 0.43 — comparable to the top-performing LLMs. These Artificial intelligence tools offer a promising direction for SBI risk prediction, achieving performance comparable to that of experienced pediatric specialists, while maintaining simplicity of use/data-preprocessing for potential real-world applications.

The application of deep learning to tabular data remains an ongoing challenge, with tree-based models such as XGBoost consistently outperforming neural network methods in most real-world scenarios. Recent advances in Large Language Models (LLMs) highlight their ability to generalize to new tasks via few-shot or one-shot prompting, yet questions remain about their utility in structured, tabular domains—particularly for resource-efficient Small Language Models (SLMs). In this work, we systematically evaluate the effectiveness of both large and small transformer-based language models for tabular classification tasks using few-shot strategies. Our approach investigates input serialization schemes and prompt engineering to maximize performance in low-data regimes. Experiments on benchmark datasets—including Diabetes, Heart Failure, and German Credit Risk—demonstrate that well-designed SLMs can approach, and occasionally match, the performance of much larger models, substantially reducing the need for extensive labeled data and retraining. However, further advances are required for language models to consistently rival the strongest tree-ensemble baselines. Our findings support the idea that SLMs, when properly prompted, offer a promising, flexible, and label-efficient alternative for automating and dynamizing machine learning pipelines on tabular data.

Inspired by the success of Large Language Models (LLMs) in language and vision tasks, there has been growing interest in applying LLMs to graph tasks, particularly on Text-Attributed Graphs (TAGs). However, most prior work tackles the node classification task. In this work, we evaluate an LLM's ability to reason over structured data and infer new facts based on learned patterns by focusing on link prediction (LP)-the task of predicting missing links between nodes-that is understudied in the literature. This task poses two key challenges: (1) How to effectively integrate pairwise structural information, which is crucial for LP performance, into LLMs, and (2) how to address the computational bottleneck during inference. To tackle these challenges, we propose LinkGPT, the first LLM-based training and inference framework specifically designed for LP on homogeneous TAGs. To enhance the LLM's ability to understand the underlying structure, we carefully design a node encoder and pairwise encoder, and leverage a two-stage instruction tuning to effectively incorporate the nodewise and pairwise information into LLMs. For inference efficiency, we introduce a retrieval-reranking scheme. Extensive experiments show that LinkGPT achieves state-of-the-art performance on real-world graphs and demonstrates superior zero-shot and few-shot generalization. At inference time, it achieves a 10× speedup while maintaining high LP accuracy.

Artificial intelligence (AI) and large language models (LLMs), are potential tools for enhancing healthcare delivery and clinical research. Presently, there is a scarcity of research regarding the viewpoints of medical students toward LLMs. This study aims to explore the perceptions of LLMs and their applications among this demographic. A cross-sectional study was done using an online survey. It was designed using Google Forms and circulated from July 2023 to August 2023. The target population included medical students from Ecuador, Ethiopia, India, Mauritius, Pakistan, United Arab Emirates (UAE), and the United States (USA). A total of 1180 responses were collected from 10 medical colleges across 7 countries. The UAE had the largest number of responses (31.7%), followed by Ecuador (17.6%), and Mauritius (13.7%). And 77.4% respondents were already aware about LLMs before attempting the survey, most popularly, ChatGPT. Participants were most familiar with the specific use of LLMs in research (46%), and the lowest familiarity was seen in the use of LLMs in healthcare (37%). More than half of the participants (52%) support the use of LLMs healthcare, yet a significant number of them remained neutral (37%) or disagreed (31%) on LLMs being safe in this context. While most of the medical students are aware of LLMs and their applications, the multinational survey demonstrated hesitancy among medical students in fully adopting LLMs into healthcare practice and clinical research. This study prompts further exploration of medical students' attitudes and acceptance concerning the integration of LLMs for research methodologies, ethical considerations, and healthcare practice.

Psychometric and moral benchmarks are increasingly used to evaluate large language models (LLMs), aiming to measure their capabilities, surface implicit biases, and assess alignment with human values. However, interpreting LLM responses to these benchmarks is methodologically challenging, a nuance often overlooked in existing literature. We empirically demonstrate that LLM responses to a standard psychometric benchmark (generalized trust from the World Values Survey) correlate strongly with known survey results across language communities. Critically, we observe LLMs achieve this while explicitly referencing known survey results and the broader literature, even without direct prompting. We further show these correlations can be amplified or effectively eliminated by subtle changes in evaluation task design, revealing that replicating known results does not validate LLMs as naive subjects. Given LLMs' access to relevant literature, their ability to replicate known human behavior constitutes an invalid evaluation for assessing the suitability of large language models as naive subjects. Fascinating though it may be, this ability provides no evidence of generalizability to novel or out-of-sample behaviors. We discuss implications for alignment research and benchmarking practices.

The object of this study is the process that generates machine-readable Country-by-Country reports in XML format using large language models. This paper addresses the task related to the current dependence of the process that generates these reports on specialized software, which leads to additional financial costs. The research and analysis of the effectiveness of publicly available large language models for generating Country-by-Country reports with new data showed high results, provided that an example model of such generation was prompted. Three large language models out of nine studied yielded results close to ideal (obtained by manual preparation or specialized systems), namely 96 points out of 100 according to the devised evaluation methodology. Four other studied models demonstrated slightly lower efficiency, but their level is also sufficient for practical use. At the same time, the resulting average cost of generating one report (US cents 4.2) is significantly lower than in the case of using specialized systems. Regarding the effectiveness of general-purpose large language models for generating Country-by-Country reports in the absence of a generation example, it is currently insufficient for practical use. In this case, all of the models studied showed results close to 0 points, i.e., completely incorrect reports were obtained. Such results are attributed to the insufficient amount of sample data during training of publicly available models. Thus, publicly available large language models could in practice replace specialized software systems designed to generate Country-by-Country reports in XML format, at least in the case of generating new reports

In recent years, Transformer-based large language models (LLMs) have significantly improved upon their text generation capability. Mental health is a serious concern that can be addressed using LLM-based automated mental health counselors. These systems can provide empathetic responses to individuals in need while considering the negative beliefs, stigma, and taboos associated with mental health issues. Considering the large size of these LLMs makes it difficult to deploy these automated counselors on low cost/resource devices such as edge devices. Therefore, the motivation of the present study to analyze the effectiveness of lightweight LLMs in the development of automated mental health counseling systems. In this study, lightweight open source LLMs such as Google’s T5s (small variant), BARTB (base variant), FLAN-T5s (small variant), and Microsoft’s GODELB (base variant) have been fine-tuned for automated mental health counseling task utilizing a diverse set of datasets publicly available online. The experimental results reveal that BART’s base variant outperformed the other models across all key metrics such as ROUGE-1, ROUGE-2, ROUGE-L, and BLEU with scores of 0.4727, 0.2665, 0.3554, and 25.3993 respectively. In comparison to other models, BART-base model generated empathetic, and emotionally supportive responses. These findings highlight the potential of lightweight LLMs (small size LLMs), in advancing the field of LLM-based mental health counseling solutions and underscore the need for exploration of lightweight LLMs for this mental health counseling use case. The code for this work is available at the following link: https://github.com/diviitmg03/Comparative-analysis-of-LLMs-.git.