Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy

Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study

Purpose : Diabetic retinopathy (DR) is a leading cause of blindness in low- and middle-income countries. This study aims to assess the agreement and diagnostic test accuracy of DR grading using fundus photographs by the allied medical personnel (AMP) as compared to ophthalmologist at DR screening program. Methods : Two 45 degree disc-centred and macula centred fundus photographs captured using digital fundus camera from 1344 eyes were graded by six AMP twice at three months and six months of training. DR grading was done using Early Treatment Diabetic Retinopathy Study criteria. Retina specialist grading was taken as the reference standard. Percentage agreement using kappa coefficient (k) and diagnostic test accuracy were assessed. Results : Intra-rater agreement of most AMP was almost perfect for detection of any DR and vision-threatening diabetic retinopathy (VTDR) at three and six months. Inter-rater agreement at 3 months at second grading was substantial for any DR (k = 0.62; 95% confidence interval (CI): 0.67–0.68) and VTDR (k = 0.63; 95% CI: 0.47–0.75). Inter-rater agreement in 6 months at second grading was also substantial for any DR (k = 0.66; 95% CI: 0.61–0.71) and VTDR (k = 0.64; 95% CI: 0.52–0.75). The sensitivity and specificity for any DR at second grading in 3 months was 94.99% and 63.97% and at 6 months was 95.55% and 66.91%, respectively. Conclusion : The inter-rater agreement of AMP as compared to ophthalmologists for any DR or VTDR was substantial at both 3 and 6 months. AMP can support as first-level DR graders for timely referral and treatment of VTDR.

Purpose. To automate the retinal screening process and to improve chances of early detection/diagnosis of diabetic retinopathy which may reduce vision impairment cases due to the diseaseMethods. Experiments were conducted on EYEPACS dataset that had retinal fundus images of patients with diabetic retinopathy in all severity levels. Based on the learning done by the Inception networks during the training phase, the images were classified to 5 classes based on the severity as: Normal, Mild, Moderate, severe and proliferative diabetic retinopathy. The images were augmented to increase the number of images with the disease and were randomly chosen for the training, validation and test datasets.Results: The training was conducted using the Inception network for 100 epochs and validation accuracy was tested every 1000 iterations on a total of 109,540 images. The major misclassifications that occurred were between the adjacent classes of severity. It achieved a sensitivity of 94.6%, specificity of 81.4%, accuracy of 92% and precision of 93.2%. Conclusion: The retinal fundus images were examined and automated using Inception network by training the images from scratch. This model can be applied for detection of other retinal diseases, as the model weights are learnt to recognize subtle features present in the retinal images

Background/AimThe relationship between diabetic retinopathy (DR) and cognitive impairment (CI) is unclear due to equivocal findings from cross-sectional studies and a lack of long-term data. In this population-based cohort study,...

Diabetic retinopathy (DR) is one of the most common microvascular complications associated with diabetes mellitus. However, its correlation with another diabetes-related disorder, cognitive impairment, has not been well studied. This systematic review and meta-analysis aimed to explore the association between DR and cognitive impairment. MEDLINE (PubMed), the Cochrane Library, and EMBASE databases were searched for observational studies that reported an association between DR and cognitive impairment. Data from selected studies were extracted, and a meta-analysis was conducted using fixed-effects modeling. Fifteen observational studies were included in the systematic review, and 10 studies were included in the meta-analysis. The odds ratio of the association between DR and cognitive impairment was 2.24 (95% confidence interval [CI], 1.89–2.66; I2 = 0.8%). The hazard ratio of the association between DR and cognitive impairment was significant in four studies, ranging from 1.09–1.32. Minimal or mild DR was not significantly associated with cognitive impairment (odds ratio [OR], 2.04; 95% CI, 0.87–4.77). However, the association between proliferative DR and cognitive impairment (OR, 3.57; 95% CI, 1.79–7.12; I2 = 16.6%) was not stronger than the association between moderate or worse DR and cognitive impairment (OR, 4.26; 95% CI, 2.01–9.07; I2 = 0.0%). DR is associated with cognitive impairment, and screening for DR will be helpful for the early identification of individuals with cognitive impairment. Further studies are needed to confirm the association between proliferative DR and cognitive impairment.

Diabetic retinopathy (DR) is the most common complication of diabetes mellitus and one of the leading causes of preventable blindness in the adult working population. Visual loss can be prevented from the early stages of DR, when the treatments are effective. Therefore, early diagnosis is paramount. However, DR may be clinically asymptomatic until the advanced stage, when vision is already affected and treatment may become difficult. For this reason, diabetic patients should undergo regular eye examinations through screening programs. Traditionally, DR screening programs are run by trained specialists through visual inspection of the retinal images. However, this manual analysis is time consuming and expensive. With the increasing incidence of diabetes and the limited number of clinicians and sanitary resources, the early detection of DR becomes non-viable. For this reason, computed-aided diagnosis (CAD) systems are required to assist specialists for a fast, reliable diagnosis, allowing to reduce the workload and the associated costs. We hypothesize that the application of novel, automatic algorithms for fundus image analysis could contribute to the early diagnosis of DR. Consequently, the main objective of the present Doctoral Thesis is to study, design and develop novel methods based on the automatic analysis of fundus images to aid in the screening, diagnosis, and treatment of DR. In order to achieve the main goal, we built a private database and used five retinal public databases: DRIMDB, DIARETDB1, DRIVE, Messidor and Kaggle. The stages of fundus image processing covered in this Thesis are: retinal image quality assessment (RIQA), the location of the optic disc (OD) and the fovea, the segmentation of RLs and EXs, and the DR severity grading. RIQA was studied with two different approaches. The first approach was based on the combination of novel, global features. Results achieved 91.46% accuracy, 92.04% sensitivity, and 87.92% specificity using the private database. We developed a second approach aimed at RIQA based on deep learning. We achieved 95.29% accuracy with the private database and 99.48% accuracy with the DRIMDB database. The location of the OD and the fovea was performed using a combination of saliency maps. The proposed methods were evaluated over the private database and the public databases DRIVE, DIARETDB1 and Messidor. For the OD, we achieved 100% accuracy for all databases except Messidor (99.50%). As for the fovea location, we also reached 100% accuracy for all databases except Messidor (99.67%). The joint segmentation of RLs and EXs was accomplished by decomposing the fundus image into layers. Results were computed per pixel and per image. Using the private database, 88.34% per-image accuracy (ACCi) was reached for the RL detection and 95.41% ACCi for EX detection. An additional method was proposed for the segmentation of RLs based on superpixels. Evaluating this method with the private database, we obtained 84.45% ACCi. Results were validated using the DIARETDB1 database. Finally, we proposed a deep learning framework for the automatic DR severity grading. The method was based on a novel attention mechanism which performs a separate attention of the dark and the bright structures of the retina. The Kaggle DR detection dataset was used for development and validation. The International Clinical DR Scale was considered, which is made up of 5 DR severity levels. Classification results for all classes achieved 83.70% accuracy and a Quadratic Weighted Kappa of 0.78. The methods proposed in this Doctoral Thesis form a complete, automatic DR screening system, contributing to aid in the early detection of DR. In this way, diabetic patients could receive better attention for their ocular health avoiding vision loss. In addition, the workload of specialists could be relieved while healthcare costs are reduced.

BackgroundWith the diabetes mellitus (DM) prevalence increasing annually, the human grading of retinal images to evaluate DR has posed a substantial burden worldwide. SmartEye is a recently developed fundus image processing and analysis system with lesion quantification function for DR screening. It is sensitive to the lesion area and can automatically identify the lesion position and size. We reported the diabetic retinopathy (DR) grading results of SmartEye versus ophthalmologists in analyzing images captured with non-mydriatic fundus cameras in community healthcare centers, as well as DR lesion quantitative analysis results on different disease stages.MethodsThis is a cross-sectional study. All the fundus images were collected from the Shanghai Diabetic Eye Study in Diabetics (SDES) program from Apr 2016 to Aug 2017. 19,904 fundus images were acquired from 6013 diabetic patients. The grading results of ophthalmologists and SmartEye are compared. Lesion quantification of several images at different DR stages is also presented.ResultsThe sensitivity for diagnosing no DR, mild NPDR (non-proliferative diabetic retinopathy), moderate NPDR, severe NPDR, PDR (proliferative diabetic retinopathy) are 86.19, 83.18, 88.64, 89.59, and 85.02%. The specificity are 63.07, 70.96, 64.16, 70.38, and 74.79%, respectively. The AUC are PDR, 0.80 (0.79, 0.81); severe NPDR, 0.80 (0.79, 0.80); moderate NPDR, 0.77 (0.76, 0.77); and mild NPDR, 0.78 (0.77, 0.79). Lesion quantification results showed that the total hemorrhage area, maximum hemorrhage area, total exudation area, and maximum exudation area increase with DR severity.ConclusionsSmartEye has a high diagnostic accuracy in DR screening program using non-mydriatic fundus cameras. SmartEye quantitative analysis may be an innovative and promising method of DR diagnosis and grading.

Diabetic Retinopathy caused by Diabetes Mellitus is a major vision threatening condition across the global population. Early detection and grading of diabetic retinopathy are pivotal to avoiding the associated vision impairment, and retinal image analysis serves as an effective method for the screening process. Analysing retinal images manually to detect diabetic retinopathy and grade them is a time-consuming pro cess that necessitates the involvement of experts to perform the classification. Computer-aided diagnosis of diabetic retinopathy through retinal image analysis is an effective tool to reduce the time involved in the screening process. This study proposed a convolutional neural network based Selective Feature Map Fusion architecture that leverages the fused feature maps of ResNet50 and EfficientNetV2L networks for the detection and grading of diabetic retinopathy. Feature set of ResNet50 and EfficientNetV2L captures different aspects of the underlying lesion distribution, resulting in a more comprehensive representation of the features. This approach helps the model generalize better to unseen data by incorporating diverse data aspects, thereby preventing overfitting to a specific feature set and developing a model that is more robust to noise. Selective feature fusion helps to reduce the computational overhead of the the feature fusion computation. Comparative analysis against the ResNet50 and EfficientNetV2L models individually revealed the superior performance of the fused model in both detection and grading of diabetic retinopathy. The proposed model attained an excellent level of accuracy of 95.2% in diabetic retinopathy detection with a sensitivity of 95.2% and a specificity of 95.2%, when tested on the IDRiD dataset. The model achieved a classification accuracy of 92% for diabetic retinopathy grading with a sensitivity of 88%, a specificity of 98% and an F1-score of 0.89. Moreover, the fused framework attained superior performance when compared against existing methodologies on the IDRiD dataset. The developed model exhibited robust performance also in the DeepDRiD dataset, and it can be employed successfully in the screening processes for the diagnosis and grading of diabetic retinopathy.

PurposeIntroducing a new technique to improve deep learning (DL) models designed for automatic grading of diabetic retinopathy (DR) from retinal fundus images by enhancing predictions’ consistency.MethodsA convolutional neural network (CNN) was optimized in three different manners to predict DR grade from eye fundus images. The optimization criteria were (1) the standard cross-entropy (CE) loss; (2) CE supplemented with label smoothing (LS), a regularization approach widely employed in computer vision tasks; and (3) our proposed non-uniform label smoothing (N-ULS), a modification of LS that models the underlying structure of expert annotations.ResultsPerformance was measured in terms of quadratic-weighted κ score (quad-κ) and average area under the receiver operating curve (AUROC), as well as with suitable metrics for analyzing diagnostic consistency, like weighted precision, recall, and F1 score, or Matthews correlation coefficient. While LS generally harmed the performance of the CNN, N-ULS statistically significantly improved performance with respect to CE in terms quad-κ score (73.17 vs. 77.69, P < 0.025), without any performance decrease in average AUROC. N-ULS achieved this while simultaneously increasing performance for all other analyzed metrics.ConclusionsFor extending standard modeling approaches from DR detection to the more complex task of DR grading, it is essential to consider the underlying structure of expert annotations. The approach introduced in this article can be easily implemented in conjunction with deep neural networks to increase their consistency without sacrificing per-class performance.Translational RelevanceA straightforward modification of current standard training practices of CNNs can substantially improve consistency in DR grading, better modeling expert annotations and human variability.

Rapid, bloody, and blinding diabetic retinopathy

BackgroundIn diabetic retinopathy (DR) screening programmes feature-based grading guidelines are used by human graders. However, recent deep learning approaches have focused on end to end learning, based on labelled data at the whole image level. Most predictions from such software offer a direct grading output without information about the retinal features responsible for the grade. In this work, we demonstrate a feature based retinal image analysis system, which aims to support flexible grading and monitor progression.MethodsThe system was evaluated against images that had been graded according to two different grading systems; The International Clinical Diabetic Retinopathy and Diabetic Macular Oedema Severity Scale and the UK’s National Screening Committee guidelines.ResultsExternal evaluation on large datasets collected from three nations (Kenya, Saudi Arabia and China) was carried out. On a DR referable level, sensitivity did not vary significantly between different DR grading schemes (91.2–94.2.0%) and there were excellent specificity values above 93% in all image sets. More importantly, no cases of severe non-proliferative DR, proliferative DR or DMO were missed.ConclusionsWe demonstrate the potential of an AI feature-based DR grading system that is not constrained to any specific grading scheme.

ABSTRACTThe early detection of diabetic retinopathy (DR) illnesses improves diagnosis and lowers the risk of permanent blindness. As a result, screening for DR in fundus images is an important method used to diagnose diabetes and other eye diseases. However, detecting diseases manually requires a significant amount of time and work. Deep learning (DL) techniques have produced encouraging results in categorizing fundus images. Still, the multi‐class DR disease remains a difficult task. Thus, the proposed framework adopted a novel Criss‐Cross Attention‐Based Squeeze‐and‐Excitation Assisted Vision Transformer (CCA‐SE‐ViT) model to classify DR from fundus images. Initially, the defective region of the retina is segmented using a novel dilated depth‐wise separable convolutional U‐Net model (dDSC‐UNet). Then, using the segmented regions, the fundus images are classified into multiple classes as age‐related macular degeneration (AMD), DR, glaucoma, cataracts, myopia, hypertension, normal, other abnormalities, and DR cases are classified into no DR, mild, moderate, severe, and proliferative DR, respectively. The retinal fundus images are obtained from publicly available datasets like OIA‐ODIR and APTOS 2019. The proposed methodology for multi‐class categorization of retinal illnesses in the OIA‐ODIR dataset yielded 97.2% accuracy, 96.7% precision, 96.1% recall, 95.9% F1‐score, and 96.4% specificity. The APTOS dataset was used for multi‐class classification of DR illnesses, and the results were 99.68% accuracy, 99.08% precision, 99.31% recall, 99.19% F1‐score, and 99.26% specificity. The results demonstrated that the proposed method accurately identifies DR using retinal fundus images.

To compare general ophthalmologists, retina specialists, and the EyeArt Artificial Intelligence (AI) system to the clinical reference standard for detecting more than mild diabetic retinopathy (mtmDR). Prospective, pivotal, multicenter trial conducted from April 2017 to May2018. Participants were aged ≥ 18 years who had diabetes mellitus and underwent dilated ophthalmoscopy. A total of 521 of 893 participants met these criteria and completed the study protocol. Participants underwent 2-field fundus photography (macula centered, disc centered) for the EyeArt system, dilated ophthalmoscopy, and 4-widefield stereoscopic dilated fundus photography for reference standard grading. For mtmDR detection, sensitivity and specificity of EyeArt gradings of 2-field, fundus photographs and ophthalmoscopy grading versus a rigorous clinical reference standard comprising Reading Center grading of 4-widefield stereoscopic dilated fundus photographs using the ETDRS severity scale. The AI system provided automatic eye-level results regarding mtmDR. Overall, 521 participants (999 eyes) at 10 centers underwent dilated ophthalmoscopy: 406 by nonretina and 115 by retina specialists. Reading Center graded 207 positive and 792 eyes negative for mtmDR. Of these 999 eyes, 26 eyes were ungradable by the EyeArt system, leaving 973 eyes with both EyeArt and Reading Center gradings. Retina specialists correctly identified 22 of 37 eyes as positive (sensitivity 59.5%) and 182 of 184 eyes as negative (specificity 98.9%) for mtmDR versus the EyeArt AI system that identified 36 of 37 as positive (sensitivity 97%) and 162 of 184 eyes as negative (specificity of 88%) for mtmDR. General ophthalmologists correctly identified 35 of 170 eyes as positive (sensitivity 20.6%) and 607 of 608 eyes as negative (specificity 99.8%) for mtmDR compared with the EyeArt AI system that identified 164 of 170 as positive (sensitivity 96.5%) and 525 of 608 eyes as negative (specificity 86%) for mtmDR. The AI system had a higher sensitivity for detecting mtmDR than either general ophthalmologists or retina specialists compared with the clinical reference standard. It can potentially serve as a low-cost point-of-care diabetic retinopathy detection tool and help address the diabetic eye screening burden.

Artificial intelligence for diabetic retinopathy screening in Africa

Background: Diabetic retinopathy (DR) is a long-term complication of diabetes characterized by progressive damage to the small blood vessels of the retina, which can ultimately threaten vision. Glycated hemoglobin (HbA1c) is widely recognized as a reliable marker for monitoring long-term glycemic control and may serve as an early indicator for identifying individuals at higher risk of developing DR. Aim: To evaluate the correlation between HbA1c level and grade of diabetic retinopathy, to study the awareness of diabetic retinopathy in diabetic patients and to identify the systemic risk factors associated with diabetic retinopathy. Materials and methods: This study was designed as a cross-sectional, hospital-based study. A total of 206 individuals attending the non-communicable disease (NCD) clinic at a tertiary care hospital in Kalaburagi. Data collection was carried out over a 12-month period, from November 2023 to November 2024. Results: In our study, Out of the 206 participants, diabetic retinopathy was detected in 133 patients, giving a prevalence of 64.5%. The condition was observed more frequently in males, with the highest proportion of cases falling in the 50–60 year age group. Distribution by severity revealed that 53 patients (25.75%) had mild non-proliferative DR (NPDR), 32 (16.47%) had moderate NPDR, 4 (1.94%) showed severe NPDR, and 6 (2.91%) had very severe NPDR. Proliferative DR (PDR) was present in 8 patients (3.88%), another 8 (3.88%) showed advanced diabetic eye disease, and 22 (10.7%) exhibited clinically significant macular edema (CSME). A progressive increase in HbA1c levels was noted with higher grades of retinopathy, and correlation analysis confirmed a significant positive relationship (r = 0.523, 0.687, 0.872; p &lt; 0.05). Awareness about DR was generally poor, with nearly two-thirds of patients (67.48%). Conclusion: This study demonstrates a clear positive association between elevated HbA1c levels and the severity of diabetic retinopathy. Limited awareness about the condition among patients appears to contribute to delayed detection, resulting in advanced retinal changes and substantial visual impairment. These findings highlight the importance of regular eye screening, better patient education, and effective control of both blood glucose and blood pressure in reducing the burden of diabetic retinopathy