A structured evaluation of LLM-generated step-by-step instructions in cadaveric brachial plexus dissection

Background Autoimmune hepatitis (AIH) is a chronic immune-mediated liver disease that requires long-term management, in which effective patient education plays a critical role. With the rapid development of large language models (LLMs), AI-generated health information is increasingly accessed by patients; however, the readability, quality, and educational suitability of LLM-generated AIH-related content remain insufficiently evaluated. Methods Five widely used LLMs—ChatGPT, Doubao, DeepSeek, Wenxin Yiyan, and Tongyi Qianwen—were assessed based on their responses to 20 frequently asked AIH patient education questions covering five thematic categories. Text readability was evaluated using multiple indices, including the Automated Readability Index, Flesch Reading Ease Score, Gunning Fog Index, Flesch–Kincaid Grade Level, Coleman–Liau Index, SMOG, and Linsear Write formula. Information quality and educational suitability were assessed using the Global Quality Score (GQS) and the Chinese version of the Patient Education Materials Assessment Tool (C-PEMAT). Clinical Intent Alignment (CIA) was used to evaluate the coverage of guideline-defined medical key points based on the 2025 EASL Clinical Practice Guidelines. Inter-rater reliability was analyzed using Cohen’s kappa, and comparative and correlation analyses were performed. Results Significant differences were observed among the LLMs in readability, information quality, and educational suitability (all p < 0.05). ChatGPT achieved the highest GQS and C-PEMAT scores, followed by Doubao and DeepSeek, whereas Wenxin Yiyan and Tongyi Qianwen showed lower performance and greater variability. CIA analysis indicated comparable coverage of guideline-defined clinical intent across models. Readability varied significantly across content themes, with texts related to disease mechanisms and diagnostic processes exhibiting higher linguistic complexity. Correlation analysis demonstrated moderate associations between GQS and grade-level readability indices, whereas C-PEMAT and CIA showed weak correlations with traditional readability metrics. Conclusion Substantial variability exists among LLMs in generating AIH patient education materials. Model selection critically influences information quality and educational suitability, whereas content theme primarily affects linguistic complexity. Although most models produced moderate-to-good quality information, relatively high readability levels suggest that further simplification may be needed for general patient populations. A multidimensional evaluation framework integrating readability, quality, educational suitability, and clinical intent alignment is essential for the responsible use of LLMs in AIH patient education.

Aim: The aim of this study is to compare three large language models (LLMs) (ChatGPT-5, Gemini 2.5 Flash, and DeepSeek-V3.1) in terms of accuracy, understandability, and readability, based on the answers provided to frequently asked endodontic questions. Methodology: Thirty open-ended frequently asked questions were generated using the AlsoAsked and AnswerThePublic websites. Two experienced endodontists scored the accuracy of the responses using a 5-point Likert scale. The understandability of the responses was analyzed using the Patient Education Materials Assessment Tool for Printed Materials (PEMAT-P) tool. Readability was assessed using the Flesch-Kincaid Readability Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Simple Measure of Gobbledygook (SMOG), and Coleman-Liau Index (CLI). Group comparisons were performed using the ANOVA/Kruskal-Wallis test, followed by post-hoc Dunn-Bonferroni tests. Results: Inter-rater agreement was excellent (accuracy ICC: 0.908–0.917; reliability ICC: 0.992–0.995; all p < 0.001). A significant difference was found between the models in terms of accuracy (p < 0.001): DeepSeek-V3.1 (4.63 ± 0.81) scored the highest and performed significantly better than ChatGPT-5 (3.93 ± 0.79) and Gemini 2.5 Flash (3.67 ± 0.76). No significant difference between ChatGPT-5 and Gemini 2.5 Flash (p > 0.05). The understandability (PEMAT-P) scores were similar (p = 0.683), and all models scored above 70% (ChatGPT-5, 77.46%; Gemini, 76.04%; DeepSeek-V3.1, 77.57%). Differences were found in readability metrics: DeepSeek-V3.1 scored higher than ChatGPT-5 in FRES (p = 0.044); Gemini scored higher than DeepSeek-V3.1 in FKGL (p = 0.001); in GFI, Gemini 2.5 Flash scored higher than both ChatGPT-5 (p = 0.036) and DeepSeek-V3.1 (p < 0.001); in SMOG, Gemini outperformed DeepSeek-V3.1 (p = 0.003); and in CLI, ChatGPT-5 was higher than DeepSeek-V3.1 (p = 0.004). No significant correlation was found between readability and understandability (p > 0.05). Conclusion: DeepSeek-V3.1 outperformed ChatGPT-5 and Gemini 2.5 Flash in terms of accuracy. While all models produced similar scores above the PEMAT-P understandability threshold (70%), there were significant differences in readability metrics; furthermore, no model consistently reached the recommended 6th–8th grade level.

Aim: This study aimed to evaluate and compare the readability and quality of information in responses generated by artificial intelligence (AI) models to patients' frequently asked questions about systemic isotretinoin, a medication commonly prescribed in dermatology. Materials and Methods:Thirty-four frequently asked questions from patients using isotretinoin were prepared by a team of dermatology specialists.These questions were posed to three AI-based text-generation tools (ChatGPT, Gemini 2.0, and Copilot), and the responses were analyzed.The resulting texts were compared in terms of readability levels [Flesch Reading Ease score (FRES), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog index (GFOG), Coleman-Liau index (CLI), and Automated Readability index (ARI)], sentence lengths, and content quality, which was evaluated by dermatologists.Results: None of the AI models achieved the optimal readability threshold (FRES 60).Readability metrics differed significantly among models.Gemini produced responses that were significantly less readable and more complex than those produced by ChatGPT and Copilot across all readability indices, including FRES, FKGL, SMOG, GFOG, CLI, and ARI; post-hoc analyses confirmed differences between Gemini and the other models.Sentence counts also differed significantly, with Gemini generating longer responses than Copilot.In contrast, Likert-based quality scores and response appropriateness were comparable across models, with no statistically significant differences observed. Conclusion:This study demonstrates that AI models produce academic responses that are difficult for those unfamiliar with medical terminology to understand, and can generate outputs with variable readability in health-related content.These findings highlight the need for careful evaluation of AI-based content for use in healthcare.

Aim: The aim of this study was to compare three large language models (LLMs); ChatGPT-5 (OpenAI), Gemini 2.5 Pro (Google), and DeepSeek-V3.1 (Hangzhou DeepSeek AI), in terms of their ability to provide information on questions asked about avulsion, a type of traumatic dental injury. Methods: In the study, 25 open-ended questions prepared based on the guidelines of the International Association of Dental Traumatology (IADT) were directed to large language models in their "Think / DeepThink / Reasoning" modes. The responses were evaluated by two experienced experts using a five-point Likert scale for accuracy and a three-point Likert scale for completeness (exhaustiveness). Readability was analyzed using the Flesch Reading Ease Score (FRES), Flesch–Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Simplified Measure of Gobbledygook (SMOG), and Coleman–Liau Index (CLI). Inter-rater reliability was assessed, and ANOVA/Kruskal-Wallis and post-hoc Dunn-Bonferroni tests were used for group comparisons, based on the data's normality. Results: The results showed that all three models demonstrated high evaluator agreement in terms of both accuracy and completeness. Significant differences were observed between the models in terms of accuracy scores (p=0.016). In subgroup analyses, the ChatGPT-5 model's accuracy score was significantly higher than that of the DeepSeek-V3.1 model (p=0.020). Significant differences were observed between the models in terms of completeness scores (p=0.006). In the subgroup analyses, the completeness scores of the ChatGPT-5 and Gemini 2.5 Pro models were significantly higher than those of the DeepSeek-V3.1 model (p=0.007 and p=0.046, respectively). In terms of readability, the DeepSeek-V3.1 model achieved higher FRES scores and lower FKGL, GFI, and SMOG scores. Overall, ChatGPT-5 demonstrated the highest performance in terms of accuracy and completeness, while DeepSeek-V3.1 achieved the highest readability values. Conclusion: The results show that ChatGPT-5 is the most reliable model in terms of clinical accuracy and content integrity, whereas DeepSeek-V3.1 produces more linguistically understandable responses. However, no model can completely replace human expertise. These systems should be considered tools to support clinical decision-making processes and should be used under physician supervision, especially in pediatric emergencies.

Purpose:To evaluate the performance of Large Language Models (LLMs) on optometry examination questions and compare their accuracy and readability with Bachelor of Optometry students.Methods:A cross-sectional comparative study was conducted using the publicly available, free versions of five LLM models from four platforms (ChatGPT 3.5, ChatGPT 4o, Gemini, CoPilot, and DeepSeek) and a group of 15 third- and fourth-year optometry students. Two sets of multiple-choice questions (20 theoretical and 20 clinical) were administered to both the students and the LLMs. Theoretical questions covered core optometric knowledge, while clinical questions simulated real-life patient scenarios. Responses were graded by senior ophthalmologists for accuracy, and readability was assessed via readable.com using four indices, including Flesch–Kincaid Grade Level, Flesch Reading Ease Score, Coleman Liau Score, and Simple Measure of Gobbledygook (SMOG) Index.Results:The overall scores of the optometry students (28.13 ± 3.33) were comparable to those of the LLMs (29 ± 4.41). In theoretical questions, LLMs (15.40 ± 1.82) performed at par with the students (14.07 ± 2.21), with DeepSeek and CoPilot outperforming students (scoring 17 each). However, in clinical questions, the students performed better, highlighting the limitations of LLMs in context-specific reasoning. Pairwise comparisons of the readability analysis revealed that Gemini and DeepSeek provided significantly most readable explanations, while ChatGPT 3.5 produced the most complex responses. Across models, readability varied for Flesch–Kincaid grade level (P = 0.0213), Flesch Reading Ease Score (P = 0.0014), and SMOG (P = 0.0412), with a nonsignificant trend for Coleman Liau Score (P = 0.0529).Conclusion:LLMs show reasonable accuracy, matching students in theoretical performance but underperforming in clinical reasoning. Gemini and DeepSeek offer superior readability, highlighting their promise as educational tools. Future research should focus on integrating LLMs into curricula while balancing them with hands-on clinical education.

This study aimed to evaluate the performance of four large language models (LLMs)-ChatGPT, Gemini, Copilot, and Claude-in responding to upper eyelid blepharoplasty-related questions, focusing on medical accuracy, clinical relevance, response length, and readability. A set of queries regarding upper eyelid blepharoplasty, covering six categories (anatomy, surgical procedure, additional intraoperative procedures, postoperative monitoring, follow-up, and postoperative complications) were posed to each LLM. An identical prompt establishing clinical context was provided before each question. Responses were evaluated by three ophthalmologists using a 5-point Likert scale for medical accuracy and a 3-point Likert scale for clinical relevance. The length of the responses was assessed. Readability was also evaluated using the Flesch Reading Ease Score, Flesch-Kincaid Grade Level, Coleman-Liau Index, Gunning Fog Index, and Simple Measure of Gobbledygook grade. A total of 30 standardized questions were presented to each LLM. None of the responses from any LLM received a score of 1 regarding medical accuracy for any question. ChatGPT achieved an 80% 'highly accurate' response rate, followed by Claude (60%), Gemini (40%), and Copilot (20%). None of the responses from ChatGPT and Claude received a score of1 regarding clinical relevance, whereas 10% of Gemini's responses and 26.7% of Copilot's responses received a score of 1. ChatGPT also provided the most clinically 'relevant' responses (86.7%), outperforming the other LLMs. Copilot generated the shortest responses, while ChatGPT generated the longest. Readability analyses revealed that all responses required advanced reading skills at a 'college graduate' level or higher, with Copilot's responses being the most complex. ChatGPT demonstrated superior performance in both medical accuracy and clinical relevance among evaluated LLMs regarding upper eyelid blepharoplasty, particularly excelling in postoperative monitoring and follow-up categories. While all models generated complex texts requiring advanced literacy, ChatGPT's detailed responses offer valuable guidance for ophthalmologists managing upper eyelid blepharoplasty cases. This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

Patients are turning to the internet to access educational materials to help them make healthcare decisions, making readability an important factor. This cross-sectional study assessed the readability of online patient education materials for myopia management treatments that have regulatory approval. The top 10 Google search results from May 2024 for freely available online patient information on myopia management modalities and regulatory-approved products in Canada and Australia were analysed for readability. The modalities included orthokeratology, myopia control spectacle lenses, myopia control soft contact lenses and atropine. The products included MiYOSMART® [HOYA®], Stellest® [Essilor®], MyoCare® [ZEISS], MiSight® 1 day [CooperVision®], ACUVUE® Abiliti® 1-Day [Johnson & Johnson], NaturalVue® Multifocal 1 Day [VTI], ACUVUE® Abiliti® Overnight [Johnson & Johnson] and Eikance [Aspen Pharmacare Australia]. These searches gave 120 results. Readability was assessed with Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Simple Measure of Gobbledygook (SMOG) Index and Coleman Liau Index (CLI). Additionally, websites were scored on Journal of the American Medical Association (JAMA) benchmark criteria. Statistical analysis was performed with two-tailed tests. Of 120 websites, none met the recommended sixth-grade reading level across all readability indices. There were 13 websites meeting at least one readability index, 10 being product-related. There were seven websites satisfying all four JAMA benchmarks, while the majority met one. There was a weak positive relationship between product search rank and readability (SMOG p = 0.02, GFI p = 0.02) and a weak negative relationship between JAMA benchmarks and readability for both modality (CLI p = 0.045) and product (CLI p = 0.049). Online information about myopia management is generally written above the recommended sixth-grade reading level and does not meet all JAMA benchmarks. Websites that appear as top search results do not necessarily have easier readability. The readability of online patient education materials may influence access to treatment and outcomes.

Readability & quality of large language model responses in CAR-T patient education

Large Language Models (LLMs) are increasingly used by caregivers to obtain pediatric health information. However, concerns persist regarding the accuracy, reliability, and readability of AI-generated content, especially in pediatric dentistry, where caregiver comprehension is crucial. To conduct an exploratory feasibility assessment of evaluating accuracy, quality, reliability, and readability of responses generated by ChatGPT-4, Google Gemini, and DeepSeek to common pediatric dentistry queries. This exploratory comparative cross-sectional feasibility study utilized 15 patient-oriented pediatric dentistry questions identified through structured searches and expert screening. Each question was submitted verbatim to ChatGPT-4, Gemini, and DeepSeek under standardized conditions. Responses were independently evaluated by three calibrated pediatric dentistry experts using the Global Quality Scale (GQS), a modified DISCERN tool, and the Accuracy of Information Index (AOI). Readability was assessed using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level (FKGL). Inter-examiner reliability was assessed using intraclass correlation coefficients (ICC). Statistical comparisons between LLMs were performed using a fixed-effects model with post-hoc pairwise analysis. Inter-examiner agreement was further evaluated using Bland-Altman analysis. A p-value of <0.05 was considered statistically significant. Overall scoring was consistent across examiners, with minor variability observed across domains. A linear mixed-effects model conducted separately for each domain demonstrated that LLM type significantly influenced GQS scores (F = 7.90, p = 0.00), with Gemini and DeepSeek outperforming ChatGPT. No significant differences were observed for AOI (p = 0.44) and DISCERN (p = 0.06). Bland-Altman analysis indicated minimal inter-examiner bias; however, the limits of agreement were relatively wide considering the scale range, reflecting variability between individual ratings. Single-measure ICC demonstrated poor agreement (ICC = 0.26), while higher reliability observed when scores were averaged (ICC = 0.90). This study offers an exploratory feasibility assessment of LLM evaluation in pediatric dentistry. While the models generally produced high-quality outputs, variations in accuracy, readability, and significant inter-examiner variability highlight important methodological challenges. These findings represent preliminary groundwork and require validation in larger, clinically diverse, real-world settings. LLMs may serve as supportive informational tools; however, their outputs should be interpreted cautiously and used to complement, not replace professional clinical judgment.

Large language models (LLMs) have demonstrated transformative potential in health care. They can enhance clinical and academic medicine by facilitating accurate diagnoses, interpreting laboratory results, and automating documentation processes. This study evaluates the efficacy of LLMs in generating surgical operation reports and discharge summaries, focusing on accuracy, efficiency, and quality. This study assessed the effectiveness of three leading LLMs-ChatGPT-4.0, ChatGPT-4o, and Claude-using six prompts and analyzing their responses for readability and output quality, validated by plastic surgeons. Readability was measured with the Flesch-Kincaid, Flesch reading ease scores, and Coleman-Liau Index, while reliability was evaluated using the DISCERN score. A paired two-tailed t-test (p<0.05) compared the statistical significance of these metrics and the time taken to generate operation reports and discharge summaries against the authors' results. Table 3 shows statistically significant differences in readability between ChatGPT-4o and Claude across all metrics, while ChatGPT-4 and Claude differ significantly in the Flesch reading ease and Coleman-Liau indices. Table 6 reveals extremely low p-values across BL, IS, and MM for all models, with Claude consistently outperforming both ChatGPT-4 and ChatGPT-4o. Additionally, Claude generated documents the fastest, completing tasks in approximately 10 to 14 s. These results suggest that Claude not only excels in readability but also demonstrates superior reliability and speed, making it an efficient choice for practical applications. The study highlights the importance of selecting appropriate LLMs for clinical use. Integrating these LLMs can streamline healthcare documentation, improve efficiency, and enhance patient outcomes through clearer communication and more accurate medical reports. This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

BackgroundInformation leaflets in research studies should be age-appropriate to be understood, however the formal readability of children’s participant information leaflets (PILs) for research studies has not been assessed.MethodsA single-centre cross-sectional study assessing paediatric PILs. Six readability tests were applied (Gunning Fog Index (GFI), Simple Measure of Gobbledygook (SMOG), Flesch Kincaid Grade Level (FKGL), Coleman–Liau Index (CLI), Automated Readability Index (ARI) and Flesch Reading Ease score (FRE). Results were compared between age groups, and whether the PIL was from either a commercially sponsored or investigator led study.Results191 paediatric PILs were included. Age categories; <10 years (n = 65), ≤12 (n = 73), ≤15 (n = 73) and ≥16 (n = 61); were used for analysis. There were 39 commercial PILs and 226 non-commercial PILs. For the ≤10 and ≤12 age bands, all 6 median readability scores exceeded the target age group (thus hard to read, p < 0.005), and there was no difference in readability scores between these two age bands. Four scores from the readability tests were considered age-appropriate in the ≤15 year category, and all median scores were age-appropriate in the ≥16 years age groups. Readability scores for children’s PILs were significantly higher in commercially sponsored versus non-commercial studies (P < 0.005).ConclusionImprovements are required to make children’s PILs readable for the target audience, particularly in commercially sponsored research studies.ImpactPaediatric participant information leaflets may not be readable in research studies, especially in younger age groups.PILs for children participating in commercially sponsored studies were less readable than non-commercial studies.Research teams writing PILs for a paediatric study need to consider the use of readability tools to ensure that the information they are providing is readable by the target audience.

Diabetic retinopathy is a global leading cause of blindness. Patients increasingly use the internet to search for health-related information that may affect medical decision-making, but to date, no standard exists across published websites. To assess the quality, content, and readability of information found online for diabetic retinopathy. This cross-sectional study analyzed 11 medical sites with information on diabetic retinopathy. Twenty-six questions were composed to include information most relevant to patients, and each website was independently evaluated by 1 vitreoretinal surgeon and 2 vitreoretinal fellows. Readability was analyzed using an online readability tool. The JAMA benchmarks were used to evaluate the quality of each site. Data were collected from December 2018 to January 2019 and analyzed in February 2019. A 26-question survey, JAMA benchmarks, Flesch reading ease score, Flesch-Kincaid grade level, Gunning Fog Index, Coleman Liau Index, and Simple Measure of Gobbledygook Index. The mean (SD) questionnaire score for all websites was 55.76 (13.38) (95% CI, 47.85-63.67) of 104 possible points. There was a difference between the content quality of the websites (H = 25.811, P = .004). The mean (SD) reading grade for all websites was 11.30 (1.79; 95% CI, 10.24-12.36). No correlation was found between content accuracy and the mean reading grade (r = 0.445, P = .17) or Google rank (r = -0.260, P = .43). No website achieved the full 4 JAMA benchmarks, and only 1 website achieved 3 of the 4 JAMA benchmarks. No correlation was found between the accuracy of the content of the website and JAMA benchmarks (r = 0.422, P = .20). The interobserver reproducibility was similar among the 3 observers (r = 0.87 between observers 1 and 2, r = 0.83 between observers 1 and 3, and r = 0.84 between observers 2 and 3, P < .001). These findings suggest that freely available information online about diabetic retinopathy varies by source but is generally of low quality. The material presented seems difficult to interpret and exceeds the recommended reading level for health information. Most websites reviewed did not provide sufficient information using the grading scheme used to support the patient in making medical decisions.

Evaluating the Readability of Online Patient Education Materials for Trigeminal Neuralgia

Large Language Models (LLM) like ChatGPT and Gemini have potential in nutrition applications, but recent studies suggest they provide inaccurate dietary advice. The aim of this study was to evaluate the most commonly used LLMs, ChatGPT and Gemini, for dietary recommendations for patients with irritable bowel syndrome (IBS). Various tools were used to assess the responses of LLMs in this study. The Guideline Compliance Score was created using IBS guidelines. The quality of the responses provided by LLMs was assessed using The Global Quality Score (GQS) and Completeness, Lack of Misinformation, Evidence, Appropriateness, Relevance (CLEAR) tool. Understandability and actionability were assessed using the Patient Education Materials Assessment Tool (PEMAT). The readability of ChatGPT and Gemini's responses was evaluated using Flesch Reading Ease (FRE) and Flesch Kincaid Grade Level (FKGL). This study found that most responses from ChatGPT (70%) and Gemini (57.5%) were compliant with the guidelines, but there was no significant difference in guideline compliance, quality, understandability, actionability, or readability scores (p > 0.05). The CLEAR tool showed a moderate positive correlation with PEMAT actionability (r = 0.467, p = 0.038) and understandability (r = 0.568, p = 0.009), a strong positive correlation with GQS (r = 0.611, p = 0.004). In addition, FRE and FKGL had a strong negative correlation (r = -0.784, p < 0.001), while the Guideline Compliance Score showed a moderate negative correlation with FRE (r = -0.537, p = 0.015). The study emphasizes the need for further model improvements before relying solely on LLMs in clinical nutrition practice, emphasizing the importance of dietitians' recommendations and the collaboration between AI models and healthcare teams.

Online educational materials for appendectomy patients have good quality but poor readability