Quantitative analysis of the text and graphic content in ophthalmic slide presentations

Research Type: Level 5 - Case report, Expert opinion, Personal observation Introduction/Purpose: The internet has become a primary resource for patients seeking information on health-related topics, including the diagnosis and management of musculoskeletal injuries. However, the quality and readability of this information vary widely. Ankle sprains, a common injury, are frequently researched and discussed online. This study aimed to assess the quality, reliability, and readability of online resources related to ankle sprains. By evaluating the trustworthiness of these resources, this study aims to provide insights into how online content on ankle sprains can be improved, ensuring it serves as a reliable and accessible source of information for patients. Methods: Searches were conducted across three search engines (Google, Yahoo, and Bing) using the terms “sprained ankle,” “twisted ankle,” and “torn ankle.” The first 20 websites from each search were selected, excluding those intended for healthcare professionals. Quality assessment was conducted using the DISCERN instrument, JAMA benchmark, and AAOS quality rating guidelines. Readability was evaluated with the Flesch-Kincaid Reading Grade Level (FKGL) and Flesch Reading Ease Score (FRE). Statistical analysis was performed using one-way analysis of variance (ANOVA). Results: A total of 49 websites were reviewed for quality and readability. The quality ratings were poor across all tools: the DISCERN instrument yielded a mean score of 40.08 ± 9.0 (range, 25.67-59.67 out of 80), the JAMA benchmark had a mean score of 1.65 ± 1.19 (range, 0-4 out of 4), and the AAOS quality rating resulted in a mean score of 11.4 ± 4.61 (range, 3-22 out of 24). Readability assessments revealed a mean Flesch Reading Ease (FRE) score of 58.8 ± 11.94 (range, 26.2-80.2) and a mean Flesch-Kincaid Reading Grade Level (FKGL) of 9.49 ± 2.97 (range, 5.3-21.5), indicating moderate readability at a high school to early college level. Conclusion: The online sources on ankle sprains had poor quality and readability, which may contribute to the spread of misinformation. To address this issue, efforts should be made to improve the quality of content through evidence-based information and enhance readability by simplifying language to better match the general public’s understanding. Future initiatives may include collaboration with healthcare organizations to improve online resources and ensure they are accessible and reliable for patients seeking accurate information.

6544 Background: Informed consent forms (ICF) play an integral role in the consent process for clinical trials (CT) by serving as a guide for discussion between the research team and study subjects. Our aims were to compare the quality of ICF among different 1) trial phases, 2) funding sources, 3) oncology subspecialties, 4) disease settings and 5) intervention modalities. Methods: ICF for prospectively conducted CT approved at the Princess Margaret Hospital from 2005–2007 were examined for their content on benefits and risks, alternatives, voluntary participation and confidentiality. Readability was assessed with the Flesch Reading Ease (FRE) score (range 0 to 100 with higher scores denoting greater reading ease) and Reading Grade Level (RGL). Results: Of the 262 evaluable CT, ICF contained an average of 3,982 words, 379 sentences and 10.5 pages. The mean FRE score and RGL were 61.2 and 7.4, respectively. All ICF explicitly stated that the CT was investigational. Only 2 (1%) promised direct personal benefits, 16 (6%) suggested the chance of cure or prolonged survival and 89 (34%) indicated a potential for tumor response. Conversely, 239 (91%) mentioned the risk of serious harms, 217 (83%) admitted that some side effects could be unknown or unpredictable and 126 (48%) reported hospitalization or death as a possibility. Alternatives to participation, right to withdraw from study and data confidentiality were addressed in 242 (92%), 254 (97%) and 260 (99%) ICF, respectively. Multivariate comparisons of benefit and risk descriptions and readability were tabulated (see table). ICF for hematology CT were most likely to highlight major risks (p=0.02), while readability was better in phase I CT and in those which were conducted by medical oncology, sponsored by governmental agencies and involved systemic therapy (all p<0.05). Conclusions: ICF provided a realistic overview of the benefits and risks of CT, although the risk of hospitalization or fatality was underreported. While readability was acceptable in most ICF, these documents remained lengthy. Parameter N Description of Benefits and Risks Readability % of CFs Stating CA May Improve p-value % of CFs Stating Risk of Death p-value Mean FRE p-value Mean RGL p-value Phase I I/II II III 57 25 90 90 26.3 24.0 33.3 42.2 0.12 43.9 44.0 48.9 51.1 0.28 62.9 58.5 61.2 60.8 0.002 7.1 7.8 7.3 7.5 0.001 Funding Academic Gov't Industry Unknown 36 39 158 29 38.9 41.0 29.1 42.9 0.27 27.8 51.3 56.3 25.0 0.16 60.4 64.1 60.8 60.8 0.003 7.5 6.9 7.5 7.4 <0.001 Specialty MedOnc Heme RadOnc Surgery 101 60 52 49 38.6 26.7 42.3 24.5 0.14 51.5 70.0 28.9 34.7 0.02 62.4 61.2 60.2 59.8 0.003 7.2 7.4 7.5 7.6 0.002 Setting Diagnostic Curative Metastatic 28 73 161 32.1 34.3 34.2 0.95 17.9 38.4 57.8 0.23 59.8 60.3 61.8 0.47 7.6 7.5 7.3 0.56 Modality SystemicTx Other 193 69 34.2 33.3 0.20 87.3 12.7 0.06 61.8 59.6 0.02 7.3 7.7 0.001 %: percentage; CFs: consent forms; CA: cancer; FRE: Flesch Reading Ease score; RGL: Reading Grade Level; Gov't: government; MedOnc: medical oncology; Heme: hematology; RadOnc: radiation oncology; SystemicTx: systemic therapy; p-values listed are based on multivariate analyses. No significant financial relationships to disclose.

Importance: In the modern-day health care environment, it is expected that a large proportion of patients will utilize the Internet to gather health care-related information. However, online sources are often not subjected to adequate quality control measures. Objective: To evaluate the quality of printed online education materials (POEMs) pertaining to elective facial cosmetic surgery. Design, Setting, and Participants: This is a descriptive correlational study. Independent Internet searches for the terms "rhinoplasty," "rhytidectomy," "blepharoplasty," and "browlift" were carried out using the Google search engine. The top 20 web resources for each term were analyzed. Main Outcomes and Measures: All web resources were analyzed for HONcode certification. The DISCERN instrument was then used to determine reliability and specificity of information, whereas readability was assessed using the Flesch Reading Ease (FRE) score and the Flesch-Kincaid Grade Level (FKGL). Results: Of the resulting 80 web resources, 30% (n = 24) were HONcode certified. None of the surgical society web resources (n = 12) were HONcode certified, and only 42% (n = 8) of web resources associated with academic institutions were HONcode certified. Out of a maximum DISCERN score of 80, the mean (standard deviation [SD]) scores were 48 (11) for rhinoplasty, 50 (11) for rhytidectomy, 47 (11) for blepharoplasty, and 45 (10) for browlift. The mean DISCERN scores of the HONcode-certified web resources did not differ significantly from scores of uncertified web resources (p = 0.069). There was no significant difference in overall mean DISCERN scores for the different web resource categories. The mean (SD) FRE score was 45.7 (11.9), whereas the mean FKGL score was 10.7 (1.9). HONcode-certified web resources had significantly higher FRE scores (p = 0.028) and significantly lower FKGL scores (p = 0.0003). Conclusions and Relevance: A significant proportion of POEMs regarding elective cosmetic facial plastic surgery remains uncertified and of inadequate quality. This can negatively impact the informed decision-making process and increase the risk of poor outcomes and lower satisfaction rates.

Quality and readability of online information on dental treatment for snoring and obstructive sleep apnea

Quality and readability of internet-based information on halitosis

This study aims to evaluate and compare the performance of artificial intelligence chatbots by assessing the reliability and quality of the information they provide regarding penis enhancement (PE). Search trends for keywords related to PE were determined using Google Trends ( https://trends.google.com ) and Semrush ( https://www.semrush.com ). Data covering a ten-year period was analyzed, taking into account regional trends and changes in search volume. Based on these trends, 25 questions were selected and categorized into three groups: general information (GI), surgical treatment (ST) and myths/misconceptions (MM). These questions were posed to three advanced chatbots: ChatGPT-4, Gemini Pro and Llama 3.1. Responses from each model were analyzed for readability using the Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease Score (FRES), while the quality of the responses was evaluated using the Ensuring Quality Information for Patients (EQIP) tool and the Modified DISCERN Score. All chatbot responses exhibited difficulty in readability and understanding according to FKGL and FRES, with no statistically significant differences among them (FKGL: p = 0.167; FRES: p = 0.366). Llama achieved the highest median Modified DISCERN score (4 [IQR:1]), significantly outperforming ChatGPT (3 [IQR:0]) and Gemini (3 [IQR:2]) (p < 0.001). Pairwise comparisons showed no significant difference between ChatGPT and Gemini (p = 0.070), but Llama was superior to both (p < 0.001). In EQIP scores, Llama also scored highest (73.8 ± 2.2), significantly surpassing ChatGPT (68.7 ± 2.1) and Gemini (54.2 ± 1.3) (p < 0.001). Across categories, Llama consistently achieved higher EQIP scores (GI:71.1 ± 1.6; ST: 73.6 ± 4.1; MM: 76.3 ± 2.1) and Modified DISCERN scores (GI:4 [IQR:0]; ST:4 [IQR:1]; MM:3 [IQR:1]) compared to ChatGPT (EQIP: GI:68.4 ± 1.1; ST: 65.7 ± 2.2; MM:71.1 ± 1.7; Modified DISCERN: GI:3 [IQR:1]; ST:3 [IQR:1]; MM:3 [IQR:0]) and Gemini (EQIP: GI:55.2 ± 1.4; ST:55.2 ± 1.6; MM:2.6 ± 2.5; Modified DISCERN: GI:1 [IQR:2]; ST:1 [IQR:2]; MM:3 [IQR:0]) (p < 0.001). This study highlights Llama's superior reliability in providing PE-related health information, though all chatbots struggled with readability.

Patients have immediate access to their diagnostic reports but these reports exceed the recommended reading level for patient-facing materials. Generative artificial intelligence may be a tool for improving patient comprehension of health information. This study assessed the readability and accuracy of ChatGPT-simplified breast pathology reports. Ten de-identified patient breast pathology reports were simplified by ChatGPT-4.0 using three different prompts. Prompt 1 requested simplification, Prompt 2 added a 6th-grade-level specification, and Prompt 3 requested essential information. The Flesch-Kincaid Reading Level (FKRL) and Flesch Reading Ease Score (FRES) were utilized to quantify readability and ease of reading, respectively. Five physicians used a four-point scale to assess factual correctness, relevancy, and fabrications to determine overall accuracy. Mean scores and standard deviations for FKRL, FRES, and accuracy scores were compared using analysis of variance (ANOVA) and t-tests. Prompt 2 demonstrated a reduction in FKRL (p<0.001) and an increase in FRES (p<0.001), demonstrating improved readability and ease of reading. ChatGPT-simplified reports received an overall accuracy score of 3.59/4 (standard deviation [SD] ±0.17). The scores by rubric category were 3.62 (SD ± 0.31) for factual correctness (4 = completely correct), 3.27 (SD ± 0.44) for relevancy (4 = completely relevant), and 3.89 (SD ± 0.11) for fabricated information (4 = no fabricated information). ChatGPT simplified breast pathology reports to the reading level recommended for patient-facing materials when given a grade-level specification while mostly maintaining accuracy. To minimize the risk of medically inaccurate and/or misleading information, ChatGPT-simplified reports should be reviewed before dissemination.

Numerous professional societies, clinical practices, and hospitals provide Internet-based patient education materials (PEMs) to the general public, but not all of this information is written at a reading level appropriate for the average patient. The National Institutes of Health and the US Department of Health and Human Services recommend that PEMs be written at or below the sixth-grade level. Our purpose was to assess the readability of endoscopic sinus surgery (ESS)-related PEMs available on the Internet and compare readability levels of PEMs provided by three sources: professional societies, clinical practices, and hospitals. A descriptive and correlational design was used for this study. The readability of 31 ESS-related PEMs was assessed with four different readability indices: Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES), Simple Measure of Gobbledygook (SMOG), and Gunning Frequency of Gobbledygook (Gunning FOG). Averages were evaluated against national recommendations and between each source using analysis of variance and t tests. The majority of PEMs (96.8%) were written above the recommended sixth-grade reading level, based on FKGL (P < .001). Only one article (3.2%) had an FKGL at or below the sixth-grade level. The mean readability values were: FRES 47.1 ± 13.4, FKGL 10.7 ± 2.4, SMOG 13.7 ± 1.6, and Gunning FOG 12.4 ± 2.7. Current Internet-based PEMs related to ESS, regardless of source type, were written well above the recommended sixth-grade level. Materials from the hospitals/university-affiliated websites had lower readability scores, but were still above recommended levels. Web-based PEMs pertaining to ESS should be written with the average patient in mind.

Analysis of Internet-Based Patient Education Materials Related to Pituitary Tumors

Retrospective Observational Study. The objective of this study was to assess the utility of ChatGPT, an artificial intelligence chatbot, in providing patient information for lumbar spinal fusion and lumbar laminectomy in comparison with the Google search engine. ChatGPT, an artificial intelligence chatbot with seemingly unlimited functionality, may present an alternative to a Google web search for patients seeking information about medical questions. With widespread misinformation and suboptimal quality of online health information, it is imperative to assess ChatGPT as a resource for this purpose. The first 10 frequently asked questions (FAQs) related to the search terms "lumbar spinal fusion" and "lumbar laminectomy" were extracted from Google and ChatGPT. Responses to shared questions were compared regarding length and readability, using the Flesch Reading Ease score and Flesch-Kincaid Grade Level. Numerical FAQs from Google were replicated in ChatGPT. Two of 10 (20%) questions for both lumbar spinal fusion and lumbar laminectomy were asked similarly between ChatGPT and Google. Compared with Google, ChatGPT's responses were lengthier (340.0 vs. 159.3 words) and of lower readability (Flesch Reading Ease score: 34.0 vs. 58.2; Flesch-Kincaid grade level: 11.6 vs. 8.8). Subjectively, we evaluated these responses to be accurate and adequately nonspecific. Each response concluded with a recommendation to discuss further with a health care provider. Over half of the numerical questions from Google produced a varying or nonnumerical response in ChatGPT. FAQs and responses regarding lumbar spinal fusion and lumbar laminectomy were highly variable between Google and ChatGPT. While ChatGPT may be able to produce relatively accurate responses in select questions, its role remains as a supplement or starting point to a consultation with a physician, not as a replacement, and should be taken with caution until its functionality can be validated.

Aim: This study aimed to evaluate and compare the readability and quality of information in responses generated by artificial intelligence (AI) models to patients' frequently asked questions about systemic isotretinoin, a medication commonly prescribed in dermatology. Materials and Methods:Thirty-four frequently asked questions from patients using isotretinoin were prepared by a team of dermatology specialists.These questions were posed to three AI-based text-generation tools (ChatGPT, Gemini 2.0, and Copilot), and the responses were analyzed.The resulting texts were compared in terms of readability levels [Flesch Reading Ease score (FRES), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog index (GFOG), Coleman-Liau index (CLI), and Automated Readability index (ARI)], sentence lengths, and content quality, which was evaluated by dermatologists.Results: None of the AI models achieved the optimal readability threshold (FRES 60).Readability metrics differed significantly among models.Gemini produced responses that were significantly less readable and more complex than those produced by ChatGPT and Copilot across all readability indices, including FRES, FKGL, SMOG, GFOG, CLI, and ARI; post-hoc analyses confirmed differences between Gemini and the other models.Sentence counts also differed significantly, with Gemini generating longer responses than Copilot.In contrast, Likert-based quality scores and response appropriateness were comparable across models, with no statistically significant differences observed. Conclusion:This study demonstrates that AI models produce academic responses that are difficult for those unfamiliar with medical terminology to understand, and can generate outputs with variable readability in health-related content.These findings highlight the need for careful evaluation of AI-based content for use in healthcare.

ObjectiveIdiopathic subglottic stenosis (ISS) is a chronic condition characterized by disease recurrence and multiple surgeries. These frustrated patients may utilize the internet to research their condition. The aim of this study was to determine the quality and readability of online ISS information.Methods“Idiopathic subglottic stenosis” was entered into Google. The first 50 websites that met inclusion criteria were extracted. The DISCERN instrument, Flesch Reading Ease Score (FRES), and Flesch‐Kincaid Grade Level (FKGL) assessed the quality and readability, respectively. Means, SDs, Pearson correlation coefficients, and two‐tailed Student's t‐test were calculated.ResultsThe 50 websites consisted of 17 patient‐targeted and 33 professional‐targeted websites, plus 30 major and 20 minor websites. The overall DISCERN, FRES, and FKGL scores were 2.81 ± 0.99, 27.75 ± 15.27, and 13.65 ± 2.79, respectively (mean ± SD). Patient‐targeted websites had significantly lower quality (DISCERN [P < .00]) but were easier to read (lower FKGL [P < .00], higher FRES [P < .00]) than professional‐targeted websites. Minor websites had a significantly lower quality (DISCERN [P < 0.00]) but were easier to read (lower FKGL [P < .00], higher FRES [P < .00]) than major websites. There was a positive correlation between overall quality and difficulty in readability.ConclusionThe quality of online ISS information was suboptimal. Resources were too difficult to comprehend and readability scores were above AMA and NIH recommendations. Improved online information is required to properly educate this patient population.Level of EvidenceLevel 4.

Plain-language prompting improves readability of ChatGPT-generated patient education for meniscal surgery without loss of accuracy.

Purpose:To evaluate the performance of Large Language Models (LLMs) on optometry examination questions and compare their accuracy and readability with Bachelor of Optometry students.Methods:A cross-sectional comparative study was conducted using the publicly available, free versions of five LLM models from four platforms (ChatGPT 3.5, ChatGPT 4o, Gemini, CoPilot, and DeepSeek) and a group of 15 third- and fourth-year optometry students. Two sets of multiple-choice questions (20 theoretical and 20 clinical) were administered to both the students and the LLMs. Theoretical questions covered core optometric knowledge, while clinical questions simulated real-life patient scenarios. Responses were graded by senior ophthalmologists for accuracy, and readability was assessed via readable.com using four indices, including Flesch–Kincaid Grade Level, Flesch Reading Ease Score, Coleman Liau Score, and Simple Measure of Gobbledygook (SMOG) Index.Results:The overall scores of the optometry students (28.13 ± 3.33) were comparable to those of the LLMs (29 ± 4.41). In theoretical questions, LLMs (15.40 ± 1.82) performed at par with the students (14.07 ± 2.21), with DeepSeek and CoPilot outperforming students (scoring 17 each). However, in clinical questions, the students performed better, highlighting the limitations of LLMs in context-specific reasoning. Pairwise comparisons of the readability analysis revealed that Gemini and DeepSeek provided significantly most readable explanations, while ChatGPT 3.5 produced the most complex responses. Across models, readability varied for Flesch–Kincaid grade level (P = 0.0213), Flesch Reading Ease Score (P = 0.0014), and SMOG (P = 0.0412), with a nonsignificant trend for Coleman Liau Score (P = 0.0529).Conclusion:LLMs show reasonable accuracy, matching students in theoretical performance but underperforming in clinical reasoning. Gemini and DeepSeek offer superior readability, highlighting their promise as educational tools. Future research should focus on integrating LLMs into curricula while balancing them with hands-on clinical education.

Observational study. To evaluate the readability of commonly utilized patient-reported outcome measures (PROMs) in spine surgery. Although studies have evaluated patient education materials, discharge instructions, and informed consent forms in spine surgery, there is a dearth of literature on the readability of PROMs despite widespread health illiteracy. Without knowledge of PROM readability, it is unclear whether these measures are able to be understood by the average spine patient. We analyzed all commonly utilized nonvisual PROMs within the spinal literature and uploaded PROMs into an online readability calculator. The Flesch Reading Ease Score (FRES) and Simple Measure of Gobbledygook (SMOG) Index were collected. A FRES>79 or SMOG<7 was considered readable by the general population per American Medical Association and Centers for Disease Control guidelines. A stricter threshold recommended in health care (SMOG <6 or FRES>89) was then used to further review readability. Seventy-seven PROMs were included. Based on FRES, the mean readability of all PROMs was 69.2 ± 17.2 (range, 10-96.4), indicating an average eighth to ninth-grade reading level. The mean readability score categorized by the SMOG Index was 8.12 ± 2.65 (range, 3.1-25.6), representing an eighth-grade reading level. Compared with the reading level of the general population, 49 (63.6%) PROMs are written above the United States literacy level, according to FRES. Using stricter definitions of readability, 8 PROMs were considered readable, including PROM Information System pain behavior (FRES: 96.4 and SMOG: 5.2), PROM Information System sleep disturbance (SMOG: 5.6), Neck Pain and Disability Scale (SMOG: 4.3), and Zung Depression Scale (SMOG: 3.1). Most PROMs utilized in spine surgery require an average reading competency far above the average patient's comprehension. This may have a meaningful impact on understanding PROM instruments and may affect the accuracy of complete surveys and the rates of incompletion.