A Study on ChatGPT-Based Code Translation from Python to Java

Purpose:To evaluate the performance of Large Language Models (LLMs) on optometry examination questions and compare their accuracy and readability with Bachelor of Optometry students.Methods:A cross-sectional comparative study was conducted using the publicly available, free versions of five LLM models from four platforms (ChatGPT 3.5, ChatGPT 4o, Gemini, CoPilot, and DeepSeek) and a group of 15 third- and fourth-year optometry students. Two sets of multiple-choice questions (20 theoretical and 20 clinical) were administered to both the students and the LLMs. Theoretical questions covered core optometric knowledge, while clinical questions simulated real-life patient scenarios. Responses were graded by senior ophthalmologists for accuracy, and readability was assessed via readable.com using four indices, including Flesch–Kincaid Grade Level, Flesch Reading Ease Score, Coleman Liau Score, and Simple Measure of Gobbledygook (SMOG) Index.Results:The overall scores of the optometry students (28.13 ± 3.33) were comparable to those of the LLMs (29 ± 4.41). In theoretical questions, LLMs (15.40 ± 1.82) performed at par with the students (14.07 ± 2.21), with DeepSeek and CoPilot outperforming students (scoring 17 each). However, in clinical questions, the students performed better, highlighting the limitations of LLMs in context-specific reasoning. Pairwise comparisons of the readability analysis revealed that Gemini and DeepSeek provided significantly most readable explanations, while ChatGPT 3.5 produced the most complex responses. Across models, readability varied for Flesch–Kincaid grade level (P = 0.0213), Flesch Reading Ease Score (P = 0.0014), and SMOG (P = 0.0412), with a nonsignificant trend for Coleman Liau Score (P = 0.0529).Conclusion:LLMs show reasonable accuracy, matching students in theoretical performance but underperforming in clinical reasoning. Gemini and DeepSeek offer superior readability, highlighting their promise as educational tools. Future research should focus on integrating LLMs into curricula while balancing them with hands-on clinical education.

This study aimed to evaluate the performance of large language models (LLMs) and multimodal LLMs in interpreting the Breast Imaging Reporting and Data System (BI-RADS) categories and providing clinical management recommendations for breast radiology in text-based and visual questions. This cross-sectional observational study involved two steps. In the first step, we compared ten LLMs (namely ChatGPT 4o, ChatGPT 4, ChatGPT 3.5, Google Gemini 1.5 Pro, Google Gemini 1.0, Microsoft Copilot, Perplexity, Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Opus 200K), general radiologists, and a breast radiologist using 100 text-based multiple-choice questions (MCQs) related to the BI-RADS Atlas 5th edition. In the second step, we assessed the performance of five multimodal LLMs (ChatGPT 4o, ChatGPT 4V, Claude 3.5 Sonnet, Claude 3 Opus, and Google Gemini 1.5 Pro) in assigning BI-RADS categories and providing clinical management recommendations on 100 breast ultrasound images. The comparison of correct answers and accuracy by question types was analyzed using McNemar's and chi-squared tests. Management scores were analyzed using the Kruskal- Wallis and Wilcoxon tests. Claude 3.5 Sonnet achieved the highest accuracy in text-based MCQs (90%), followed by ChatGPT 4o (89%), outperforming all other LLMs and general radiologists (78% and 76%) (P < 0.05), except for the Claude 3 Opus models and the breast radiologist (82%) (P > 0.05). Lower-performing LLMs included Google Gemini 1.0 (61%) and ChatGPT 3.5 (60%). Performance across different categories of showed no significant variation among LLMs or radiologists (P > 0.05). For breast ultrasound images, Claude 3.5 Sonnet achieved 59% accuracy, significantly higher than other multimodal LLMs (P < 0.05). Management recommendations were evaluated using a 3-point Likert scale, with Claude 3.5 Sonnet scoring the highest (mean: 2.12 ± 0.97) (P < 0.05). Accuracy varied significantly across BI-RADS categories, except Claude 3 Opus (P < 0.05). Gemini 1.5 Pro failed to answer any BI-RADS 5 questions correctly. Similarly, ChatGPT 4V failed to answer any BI-RADS 1 questions correctly, making them the least accurate in these categories (P < 0.05). Although LLMs such as Claude 3.5 Sonnet and ChatGPT 4o show promise in text-based BI-RADS assessments, their limitations in visual diagnostics suggest they should be used cautiously and under radiologists' supervision to avoid misdiagnoses. This study demonstrates that while LLMs exhibit strong capabilities in text-based BI-RADS assessments, their visual diagnostic abilities are currently limited, necessitating further development and cautious application in clinical practice.

The high performance of generative artificial intelligence (AI) and large language models (LLM) in examination contexts has triggered an intense debate about their applications, effects and risks. What legal aspects need to be considered when using LLM in teaching and assessment? What possibilities do language models offer? Statutes and laws are used to assess the use of LLM: - University statutes, state higher education laws, licensing regulations for doctors - Copyright Act (UrhG) - General Data Protection Regulation (DGPR) - AI Regulation (EU AI Act) LLM and AI offer opportunities but require clear university frameworks. These should define legitimate uses and areas where use is prohibited. Cheating and plagiarism violate good scientific practice and copyright laws. Cheating is difficult to detect. Plagiarism by AI is possible. Users of the products are responsible. LLM are effective tools for generating exam questions. Nevertheless, careful review is necessary as even apparently high-quality products may contain errors. However, the risk of copyright infringement with AI-generated exam questions is low, as copyright law allows up to 15% of protected works to be used for teaching and exams. The grading of exam content is subject to higher education laws and regulations and the GDPR. Exclusively computer-based assessment without human review is not permitted. For high-risk applications in education, the EU's AI Regulation will apply in the future. When dealing with LLM in assessments, evaluation criteria for existing assessments can be adapted, as can assessment programmes, e.g. to reduce the motivation to cheat. LLM can also become the subject of the examination themselves. Teachers should undergo further training in AI and consider LLM as an addition.

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

To assess the performance of state-of-the-art large language models in classifying vertebral metastasis stability using the Spinal Instability Neoplastic Score (SINS) compared to human experts, and to evaluate the impact of task-specific refinement including in-context learning on their performance. This retrospective study analyzed 100 synthetic CT and MRI reports encompassing a broad range of SINS scores. Four human experts (two radiologists and two neurosurgeons) and four large language models (Mistral, Claude, GPT-4 turbo, and GPT-4o) evaluated the reports. Large language models were tested in both generic form and with task-specific refinement. Performance was assessed based on correct SINS category assignment and attributed SINS points. Human experts demonstrated high median performance in SINS classification (98.5% correct) and points calculation (92% correct), with a median point offset of 0 [0-0]. Generic large language models performed poorly with 26-63% correct category and 4-15% correct SINS points allocation. In-context learning significantly improved chatbot performance to near-human levels (96-98/100 correct for classification, 86-95/100 for scoring, no significant difference to human experts). Refined large language models performed 71-85% better in SINS points allocation. In-context learning enables state-of-the-art large language models to perform at near-human expert levels in SINS classification, offering potential for automating vertebral metastasis stability assessment. The poor performance of generic large language models highlights the importance of task-specific refinement in medical applications of artificial intelligence.

Textual Proficiency and Visual Deficiency: A Comparative Study of Large Language Models and Radiologists in MRI Artifact Detection and Correction.

In this study, we employed a large language model to evaluate the diagnostic efficacy of radiology reports of bone scintigraphy in the context of identifying SAPHO syndrome, and further examined the potential of such a model to augment the diagnostic procedure. Imaging data and clinical information of 151 patients (105/46 women/men, mean age: 53.5 years) who underwent bone scintigraphy for suspected Synovitis, Acne, Pustulosis, Hyperostosis, and Osteitis (SAPHO) syndrome between January 2007 and December 2022 were retrospectively reviewed. ChatGPT-4.0 was used as the large language model. The diagnostic performance of the large language model was verified by comparing the cases judged to have SAPHO syndrome that fulfilled Kahn's classification criteria based on a combination of concise radiology reports and skin lesions such as palmoplantar pustulosis, with cases diagnosed with SAPHO syndrome by rheumatologists based on all clinical information. The diagnostic accuracy of a large language model for analysing bone scintigraphy radiology reports in conjunction with information about skin symptoms, such as palmoplantar pustulosis, achieved a sensitivity of 83.5%, specificity of 69.4%, and an overall accuracy of 76.8%. This research indicates the prospective value of extensive language models in scrutinizing radiology accounts from bone scintigraphy for the diagnosis of SAPHO syndrome.

Performance of Popular Large Language Models in Glaucoma Patient Education: a randomized controlled study

BackgroundThe accumulation of vast electronic medical records (EMRs) through medical informatization creates significant research value, particularly in obstetrics. Diagnostic standardization across different health care institutions and regions is vital for medical data analysis. Large language models (LLMs) have been extensively used for various medical tasks. Prompt engineering is key to use LLMs effectively.ObjectiveThis study aims to evaluate and compare the performance of LLMs with various prompt engineering techniques on the task of standardizing obstetric diagnostic terminology using real-world obstetric data.MethodsThe paper describes a 4-step approach used for mapping diagnoses in electronic medical records to the International Classification of Diseases, 10th revision, observation domain. First, similarity measures were used for mapping the diagnoses. Second, candidate mapping terms were collected based on similarity scores above a threshold, to be used as the training data set. For generating optimal mapping terms, we used two LLMs (ChatGLM2 and Qwen-14B-Chat [QWEN]) for zero-shot learning in step 3. Finally, a performance comparison was conducted by using 3 pretrained bidirectional encoder representations from transformers (BERTs), including BERT, whole word masking BERT, and momentum contrastive learning with BERT (MC-BERT), for unsupervised optimal mapping term generation in the fourth step.ResultsLLMs and BERT demonstrated comparable performance at their respective optimal levels. LLMs showed clear advantages in terms of performance and efficiency in unsupervised settings. Interestingly, the performance of the LLMs varied significantly across different prompt engineering setups. For instance, when applying the self-consistency approach in QWEN, the F1-score improved by 5%, with precision increasing by 7.9%, outperforming the zero-shot method. Likewise, ChatGLM2 delivered similar rates of accurately generated responses. During the analysis, the BERT series served as a comparative model with comparable results. Among the 3 models, MC-BERT demonstrated the highest level of performance. However, the differences among the versions of BERT in this study were relatively insignificant.ConclusionsAfter applying LLMs to standardize diagnoses and designing 4 different prompts, we compared the results to those generated by the BERT model. Our findings indicate that QWEN prompts largely outperformed the other prompts, with precision comparable to that of the BERT model. These results demonstrate the potential of unsupervised approaches in improving the efficiency of aligning diagnostic terms in daily research and uncovering hidden information values in patient data.

IntroductionRecent artificial intelligence (AI) advances can generate human-like responses to a wide range of queries, making them a useful tool for healthcare applications. Therefore, the potential use of large language models (LLMs) in controlled environments regarding efficacy, reproducibility, and operability will be of paramount interest.ObjectiveWe investigated if and how GPT 3.5 and GPT 4 models can be directly used as a part of a GxP validated system and compared the performance of externally hosted GPT 3.5 and GPT 4 against LLMs, which can be hosted internally. We explored zero-shot LLM performance for named entity recognition (NER) and relation extraction tasks, investigated which LLM has the best zero-shot performance to be used potentially for generating training data proposals, evaluated the LLM performance of seven entities for medical NER in zero-shot experiments, selected one model for further performance improvement (few-shot and fine-tuning: Zephyr-7b-beta), and investigated how smaller open-source LLMs perform in contrast to GPT models and to a small fine-tuned T5 Base.MethodsWe performed reproducibility experiments to evaluate if LLMs can be used in controlled environments and utilized guided generation to use the same prompt across multiple models. Few-shot learning and quantized low rank adapter (QLoRA) fine-tuning were applied to further improve LLM performance.Results and ConclusionWe demonstrated that zero-shot GPT 4 performance is comparable with a fine-tuned T5, and Zephyr performed better than zero-shot GPT 3.5, but the recognition of product combinations such as product event combination was significantly better by using a fine-tuned T5. Although Open AI launched recently GPT versions to improve the generation of consistent output, both GPT variants failed to demonstrate reproducible results. The lack of reproducibility together with limitations of external hosted systems to keep validated systems in a state of control may affect the use of closed and proprietary models in regulated environments. However, due to the good NER performance, we recommend using GPT for creating annotation proposals for training data as a basis for fine-tuning.

With the rapid advancement of large language models (LLMs), code translation has become a critical yet challenging task in software engineering. Existing evaluations of this task remain unreliable due to two key limitations: the absence of specialized function-level code translation benchmarks and flawed evaluation pipelines. Specifically, current pipelines lack post-processing mechanisms to extract target code from LLMs’ inherently unstable output formats, directly resulting in non-executable translations and false negatives in evaluation. To address these issues, we introduce FuncTransEval, an enhanced multilingual benchmark. It comprises 400×4 functions across four programming languages (Python, C++, Java, and JavaScript) and 12 bidirectional translation pairs, each paired with executable unit tests for fine-grained functional validation. Using FuncTransEval, we systematically assess LLMs’ true capabilities in code translation and identify their core limitations: inadequate understanding of the implicit information such as function logic, type specifications, and cross-language differences. To mitigate these shortcomings, we propose ICAM-Trans (Implicit-Context-Aware Multi-Agent Framework for function-level code translation). ICAM-Trans is a hierarchical, autonomous multi-agents architecture that explicitly uncovers and leverages these implicit contextual information during translation. It employs a Translation Orchestrate Agent (TOA) to autonomously coordinate four specialized Context Analysis Agents (CAAs), which conduct pre-translation analyses on different aspects. Insights from these analyses guide a Context-Aware Translation Agent (CTA) to generate semantically faithful target code. Experiments on FuncTransEval demonstrate that ICAM-Trans consistently outperforms strong baselines, validating its effectiveness in achieving high-fidelity and interpretable function-level code translation.

The Entity Relationship Extraction task involves accurately extracting target entities and uncovering their complex relational interdependencies within unstructured text, thereby establishing foundational structural knowledge for downstream applications including knowledge graph construction and semantic network population. While conventional approaches predominantly focus on sentence-level entity relationship extraction in general-domain texts. However, with the continuous expansion of practical needs, the task of entity relationship extraction no longer merely satisfies the extraction of relationships between entities from a single sentence. Instead, it requires accurately identifying the complex relationships between entities from multiple sentences.meanwhile, in the context of the Electric-power industry, in the task of entity relationship extraction in the electric professional field, due to the presence of a large number of professional terms and specific expressions in the texts of the power professional field, the relationships between entities are more complex and difficult to define. Using previous methods cannot accurately identify the entities in the power professional field and their relationships from the texts. To address these challenges, we propose a new framework for entity-relationship extraction in the electric domain based on large language models and modular fine-tuning, which we have named the R-E-M(Relationship-Entity Matching) framework. This framework comprises four components. First, it enhances the large language model's capability to recognize domainspecific entities in the power industry through targeted modular fine-tuning. Next, an entity-relationship extraction module identifies relevant entities and their corresponding relationships within the text. Subsequently, the entity-relationship matching module establishes predefined matching patterns between entities and relationships for each identified relation type. Finally, the prompt selection module dynamically chooses appropriate prompts based on different relationship types while specifying standardized output formats for the model. From the final experimental results, our proposed new framework has significantly improved the large language model's performance in the entity-relationship extraction task for power domain texts, elevating the Micro-F1 score from 0.185 to 0.575 -a substantial gain of 0.39, demonstrating remarkable enhancement.

To evaluate the performance of three large language models in answering questions regarding pediatric developmental dysplasia of the hip. We formulated 18 open-ended clinical questions in both Chinese and English and established a gold standard set of answers to benchmark the responses of the large language models. These questions were presented to ChatGPT-4o, Gemini, and Claude 3.5 Sonnet. The responses were evaluated by two independent reviewers using a 5-point scale. The average score, rounded to the nearest whole number, was taken as the final score. A final score of 4 or 5 indicated an accurate response, whereas a final score of 1, 2, or 3 indicated an inaccurate response. The raters demonstrated a high level of agreement in scoring the answers, with weighted Kappa coefficients of 0.865 for Chinese responses (p < 0.001) and 0.875 for English responses (p < 0.001). No significant differences were observed among the three large language models in terms of accuracy when answering questions, with rates of 83.3%, 77.8%, and 77.8% for Claude 3.5 Sonnet, ChatGPT-4o, and Gemini in the Chinese responses (p = 1), and 83.3%, 83.3%, and 72.2% for ChatGPT-4o, Claude 3.5 Sonnet, and Gemini in the English responses (p = 0.761). In addition, there was no significant difference in the performance of the same large language model between the Chinese and English settings. Large language models demonstrate high accuracy in delivering information on dysplasia of the hip, maintaining consistent performance across both Chinese and English, which suggests their potential utility as medical support tools. Level II.

This study aimed to evaluate the diagnostic accuracy of multimodal large language models in classifying superior labial frenulum attachments from intraoral photographs using expert consensus as the reference standard. Five experts (two periodontists and three orthodontists) established the consensus standard by classifying frenulum attachments in 117 intraoral images as mucosal, gingival, papillary, and papilla penetrating. The same photographs were then presented to three multimodal large language models (ChatGPT 4o, Gemini 2.5 Pro, and Microsoft Copilot GPT-4), and their diagnostic performance was evaluated using accuracy, sensitivity, specificity, and F1 score. Reliability was assessed using Fleiss' and Cohen's Kappa, and diagnostic performances were compared using Cochran's Q test. Human raters demonstrated almost perfect agreement (κ = 0.838, p < 0.001), whereas large language models showed poor inter-model agreement (κ = -0.124, p < 0.001). ChatGPT achieved slight significant agreement with the consensus (κ = 0.114, p = 0.019), although its clinical relevance was negligible. Gemini (κ = 0.099) and Copilot (κ = 0.027) showed no significant agreement (p > 0.05). Copilot yielded the highest overall accuracy (46.2%), followed by Gemini (44.5%) and ChatGPT (35.0%). The performance of the large language models varied across frenulum types. Current multimodal large language models demonstrate inconsistent and clinically insufficient accuracy in the classification of superior labial frenulum attachments from photographs. Domain-specific training is essential before large language models can be considered reliable diagnostic tools in dentistry.

ObjectiveThis study aimed to evaluate the performance of large language models (LLMs) in answering questions from the American Board of Surgery In-Training Examination (ABSITE).MethodsMultiple choice ABSITE Quiz was entered into the most popular LLMs as prompts. ChatGPT-4 (OpenAI), Copilot (Microsoft), and Gemini (Google) were used in the study. The research comprised 170 questions from 2017 to 2022, which were divided into four subgroups: Definitions, Biochemistry/Pharmaceutical, Case Scenario, and Treatment & Surgical Procedures. All questions were queried in LLMs, between October 1, 2024, and October 5, 2024. Correct answer rates of LLMs were evaluated.ResultsThe correct response rates for all questions were 79.4% for ChatGPT, 77.6% for Copilot, and 52.9% for Gemini, with Gemini significantly lower than both LLMs (P < 0.001). In the definition category, the correct response rates were 93.5% for ChatGPT, 90.3% for Copilot, and 64.5% for Gemini, with Gemini significantly lower (P = 0.005 and P = 0.015, respectively). In the Biochemistry/Pharmaceutical question category, the correct response rates were equal in all three groups (83.3%). In the Case Scenario category, the correct response rates were 76.3% in ChatGPT, 72.8% for Copilot, and 46.5% for Gemini, with Gemini significantly lower (P < 0.001). In the Treatment & Surgical Procedures category, the correct response rates were 69.2% for ChatGPT, 84.6% for Copilot, and 53.8% for Gemini. Although Gemini had the lowest accuracy, there was no statistically significant difference (P = 0.236).ConclusionIn the ABSITE Quiz, ChatGPT and Copilot had similar success, whereas Gemini was significantly behind.