• Home
  • Search
  • Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models
  • Cite Icon19
  • https://doi.org/10.1001/jamaophthalmol.2025.2413Copy DOI Icon

Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

IMPORTANCEOpenAI’s recent large language model (LLM) o1 has dedicated reasoning capabilities, but it remains untested in specialized medical fields like ophthalmology. Evaluating o1 in ophthalmology is crucial to determine whether its general reasoning can meet specialized needs or if domain-specific LLMs are warranted.OBJECTIVETo assess the performance and reasoning ability of OpenAI’s o1 compared with other LLMs on ophthalmological questions.DESIGN, SETTING, AND PARTICIPANTSIn September through October 2024, the LLMs o1, GPT-4o (OpenAI), GPT-4 (OpenAI), GPT-3.5 (OpenAI), Llama 3–8B (Meta), and Gemini 1.5 Pro (Google) were evaluated on 6990 standardized ophthalmology questions from the Medical Multiple-Choice Question Answering (MedMCQA) dataset. The study did not analyze human participants.MAIN OUTCOMES AND MEASURESModels were evaluated on performance (accuracy and macro F1 score) and reasoning abilities (text-generation metrics: Recall-Oriented Understudy for Gisting Evaluation [ROUGE-L], BERTScore, BARTScore, AlignScore, and Metric for Evaluation of Translation With Explicit Ordering [METEOR]). Mean scores are reported for o1, while mean differences (Δ) from o1’s scores are reported for other models. Expert qualitative evaluation of o1 and GPT-4o responses assessed usefulness, organization, and comprehensibility using 5-point Likert scales.RESULTSThe LLM o1 achieved the highest accuracy (mean, 0.877; 95% CI, 0.870 to 0.885) and macro F1 score (mean, 0.877; 95% CI, 0.869 to 0.884) (P < .001). In BERTScore, GPT-4o (Δ = 0.012; 95% CI, 0.012 to 0.013) and GPT-4 (Δ = 0.014; 95% CI, 0.014 to 0.015) outperformed o1 (P < .001). Similarly, in AlignScore, GPT-4o (Δ = 0.019; 95% CI, 0.016 to 0.021) and GPT-4 (Δ = 0.024; 95% CI, 0.021 to 0.026) again performed better (P < .001). In ROUGE-L, GPT-4o (Δ = 0.018; 95% CI, 0.017 to 0.019), GPT-4 (Δ = 0.026; 95% CI, 0.025 to 0.027), and GPT-3.5 (Δ = 0.008; 95% CI, 0.007 to 0.009) all outperformed o1 (P < .001). Conversely, o1 led in BARTScore (mean, −4.787; 95% CI, −4.813 to −4.762; P < .001) and METEOR (mean, 0.221; 95% CI, 0.218 to 0.223; P < .001 except GPT-4o). Also, o1 outperformed GPT-4o in usefulness (o1: mean, 4.81; 95% CI, 4.73 to 4.89; GPT-4o: mean, 4.53; 95% CI, 4.40 to 4.65; P < .001) and organization (o1: mean, 4.83; 95% CI, 4.75 to 4.90; GPT-4o: mean, 4.63; 95% CI, 4.51 to 4.74; P = .003).CONCLUSIONS AND RELEVANCEThis study found that o1 excelled in accuracy but showed inconsistencies in text-generation metrics, trailing GPT-4o and GPT-4; expert reviews found o1’s responses to be more clinically useful and better organized than GPT-4o. While o1 demonstrated promise, its performance in addressing ophthalmology-specific challenges is not fully optimal, underscoring the potential need for domain-specialized LLMs and targeted evaluations.

Similar Papers
  • Research Article

BEnchmarking Large Language Models for Ophthalmology (BELO): An Expert-Curated Data Set and Evaluation Framework for Knowledge and Reasoning

  • Dec 26, 2025
  • Ophthalmology Science
  • Sahana Srinivasan +31
  • PDF
  • Preprint Article

Knowledge Enhancement of Small-Scale Models in Medical Question Answering (Preprint)

  • Nov 03, 2024
  • Xinbai Li +3
  • Research Article
  • Citations58

Quality of Answers of Generative Large Language Models Versus Peer Users for Interpreting Laboratory Test Results for Lay Patients: Evaluation Study

  • Apr 17, 2024
  • Journal of Medical Internet Research
  • Zhe He +8
  • Research Article
  • Citations1

Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: Quantitative Pilot Feasibility Study.

  • Apr 13, 2026
  • JMIR formative research
  • Andy Li +4
  • Research Article
  • Citations52

Custom Large Language Models Improve Accuracy: Comparing Retrieval Augmented Generation and Artificial Intelligence Agents to Non-Custom Models for Evidence-Based Medicine

  • Nov 07, 2024
  • Arthroscopy: The Journal of Arthroscopic and Related Surgery
  • Joshua J Woo +7
  • Research Article

Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework

  • May 07, 2025
  • Mathematics
  • Yusong Ke +4
  • Research Article

Evaluating large language models for clinical note processing: local fine-tuning and internal-external validation using electronic health records from South Asia.

  • Feb 25, 2026
  • BMC medical informatics and decision making
  • Seyed Alireza Hasheminasab +18
  • Research Article
  • Citations6

Large Language Models in Randomized Controlled Trials Design: Observational Study

  • Sep 03, 2025
  • Journal of Medical Internet Research
  • Liyuan Jin +6
  • Research Article
  • Citations21

Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study

  • Apr 07, 2025
  • JMIR Formative Research
  • Denise Lee +6
  • Research Article
  • Citations7

Evaluating the performance of large language & visual-language models in cervical cytology screening

  • May 23, 2025
  • npj Precision Oncology
  • Qi Hong +15
  • Research Article
  • Citations39

One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering

  • Dec 24, 2023
  • medRxiv
  • Han Yang +5
  • Research Article
  • Citations9

Comparative Evaluation of Large Language Models for Translating Radiology Reports into Hindi

  • Sep 04, 2024
  • Indian Journal of Radiology and Imaging
  • Amit Gupta +3
  • Research Article
  • Citations2

Large Language Models in Crisis Informatics for Zero and Few-Shot Classification

  • May 22, 2025
  • ACM Transactions on the Web
  • Cinthia Sánchez +2
  • Research Article

Detecting \u201clarge language models fingerprint\u201d for Japanese texts generated by six LLMs

  • Jan 01, 2026
  • Frontiers in Artificial Intelligence
  • Wataru Zaitsu +4
  • Research Article
  • Citations4

Evaluating Retrieval Augmented Generation-enhanced Large Language Models for Question Answering On German Neurovascular Guidelines

  • Sep 02, 2025
  • Clinical Neuroradiology
  • Marius Vach +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.