Distributed Deep Learning with Data Parallelism for Diabetic Retinopathy Classification

Distributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters.

Deep Learning (DL) techniques have been widely deployed in many application domains. The growth of DL models’ size and complexity demands distributed training of DL models. Since DL training is complex, software implementing distributed DL training is error-prone. Thus, it is crucial to test distributed deep learning software to improve its reliability and quality. To address this issue, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">differential</i> testing technique—D<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup>, which leverages a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distributed</i> equivalence rule that we create to test distributed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">deep</i> learning software. The rationale is that the same model trained with the same model input under different distributed settings should produce equivalent prediction output within certain thresholds. The different output indicates potential bugs in the distributed deep learning software. D<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> automatically generates a diverse set of distributed settings, DL models, and model input to test distributed deep learning software. Our evaluation on two of the most popular DL libraries, i.e., PyTorch and TensorFlow, shows that D<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> detects 21 bugs, including 12 previously unknown bugs.

Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between workers pose serious scaling problems. In this article, we present a quantitative survey of communication optimization techniques for data parallel distributed DL. We first identify the major communication challenges and classify the existing solutions into three levels, namely the learning algorithm, the system architecture, and the network infrastructure. We present the state-of-the-art communication optimization techniques and conduct a comparative study of seven common lossless distributed DL methods on a 32-GPU cluster with 100Gb/s InfiniBand (IB). We show that the DL models with low model intensity (such as BERT and BERT-Large) are difficult to scale out even with the best available lossless algorithm over 100Gb/s IB; and the system architecture and scheduling algorithms have a critical impact on the scaling property. We conclude the article with discussions of open issues for further investigation.

In this paper, we introduce a distributed deep learning platform, BAIPAS, Big Data and AI based Predication and Analysis System. In the case of deep learning using big data, it takes much time to train with data. To reduce training time, there is a method that uses distributed deep learning. When big data exists in external storage, training takes a long time because it takes a lot of network I/O time when data is loaded during deep learning operations. We propose data locality management as a way to reduce training time with big data. BAIPAS is a distributed deep learning platform that aims to provide quick learning from big data, easy installation and monitoring of the platform, and convenience for developers of deep learning models. In order to provide fast training using big data, data is distributed and stored in worker-server storage using a data locality and shuffling, and then training is performed. The data locality manager analyzes the training data and the state information of the worker servers. This distributes the data scheduling according to the available storage space of the worker server and the learning performance of the worker server. However, if each worker server conducts deep learning using the distributed training data, model overfitting may occur as compared with the method of learning with the full training data set. To solve this problem, we applied a shuffling method that moves already learned data to another worker server when training is performed. Thereby, each worker server can contain the full training data set. BAIPAS uses Kubernetes and Docker to provide easy installation and monitoring of the platform. It also provides pre-processing modules, management tools, automation of cluster creation, resource monitoring, and other resources; so developers can easily develop deep learning models.

With the breakthrough of AlphaGo, deep reinforcement learning has become a recognized technique for solving sequential decision-making problems. Despite its reputation, data inefficiency caused by its trial and error learning mechanism makes deep reinforcement learning difficult to apply in a wide range of areas. Many methods have been developed for sample efficient deep reinforcement learning, such as environment modelling, experience transfer, and distributed modifications, among which distributed deep reinforcement learning has shown its potential in various applications, such as human-computer gaming and intelligent transportation. In this paper, we conclude the state of this exciting field, by comparing the classical distributed deep reinforcement learning methods and studying important components to achieve efficient distributed learning, covering single player single agent distributed deep reinforcement learning to the most complex multiple players multiple agents distributed deep reinforcement learning. Furthermore, we review recently released toolboxes that help to realize distributed deep reinforcement learning without many modifications of their non-distributed versions. By analysing their strengths and weaknesses, a multi-player multi-agent distributed deep reinforcement learning toolbox is developed and released, which is further validated on Wargame, a complex environment, showing the usability of the proposed toolbox for multiple players and multiple agents distributed deep reinforcement learning under complex games. Finally, we try to point out challenges and future trends, hoping that this brief review can provide a guide or a spark for researchers who are interested in distributed deep reinforcement learning.

DPro-SM – A distributed framework for proactive straggler mitigation using LSTM

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

ABSTRACTMore vertical service areas than only data processing, storing, and communication are promised by fog‐cloud computing. Due to its great efficiency and scalability, distributed deep learning (DDL) across fog‐cloud computing environments is a widely used application among them. With training limited to sharing parameters, DDL can offer more privacy protection than centralized deep learning. Nevertheless, DDL still faces two significant security obstacles when it comes to fog‐cloud computing are How to ensure that users' identities are not stolen by outside enemies, and How to prevent users' privacy from being disclosed to other internal participants in the process of training. In this manuscript, Interference Tolerant Fast Convergence Zeroing Neural Network for Security and Privacy Preservation with Reptile Search Optimization Algorithm in Fog‐Cloud Computing environment (SPP‐ITFCZNN‐RSOA‐FCC) is proposed. ITFCZNN is proposed for security and privacy preservation, Then Reptile Search Optimization Algorithm (RSOA) is proposed to optimize the ITFCZNN, and Effective Lightweight Homomorphic Cryptographic Algorithm (ELHCA) is used to encrypt and decrypt the local gradients. The proposed SPP‐ITFCZNN‐RSOA‐FCC system attains a better security balance, efficiency, and functionality than existing efforts. The proposed SPP‐ITFCZNN‐RSOA‐FCC is implemented using Python. The performance metrics like accuracy, resource overhead, computation overhead, and communication overhead are considered. The performance of the SPP‐ITFCZNN‐RSOA‐FCC approach attains 29.16%, 20.14%, and 18.93% high accuracy, and 11.03%, 26.04%, and 23.51% lower Resource overhead compared with existing methods including FedSDM: Federated learning dependent smart decision making component for ECG data at internet of things incorporated Edge‐Fog‐Cloud computing (SPP‐FSDM‐FCC), A collaborative computation with offloading in dew‐enabled vehicular fog computing to compute‐intensive with latency‐sensitive dependence‐aware tasks: Federated deep Q‐learning method (SPP‐FDQL‐FCC), and a fog‐edge‐enabled intrusion identification scheme for smart grids (SPP‐FSVM‐FCC) respectively.

Using big marine data to train deep learning models is not efficient, or sometimes even possible, on local computers. In this paper, we show how distributed learning in the cloud can help more efficiently process big data and train more accurate deep learning models. In addition, marine big data is usually communicated over wired networks, which if possible to deploy in the first place, are costly to maintain. Therefore, wireless communications dominantly conducted by acoustic waves in underwater sensor networks, may be considered. However, wireless communication is not feasible for big marine data due to the narrow frequency bandwidth of acoustic waves and the ambient noise. To address this problem, we propose an optimized deep learning design for low-energy and real-time image processing at the underwater edge. This leads to trading the need to transmit the large image data, for transmitting only the low-volume results that can be sent over wireless sensor networks. To demonstrate the benefits of our approaches in a real-world application, we perform fish segmentation in underwater videos and draw comparisons against conventional techniques. We show that, when underwater captured images are processed at the collection edge, 4 times speedup can be achieved compared to using a landside server. Furthermore, we demonstrate that deploying a compressed DNN at the edge can save 60% of power compared to a full DNN model. These results promise improved applications of affordable deep learning in underwater exploration, monitoring, navigation, tracking, disaster prevention, and scientific data collection projects.

Using Deep Learning Models to Characterize Major Retinal Features on Color Fundus Photographs

Accelerating Distributed Deep Learning draws increasing attention recently. Existing solutions like gradient compression, pipelining computation/communication, flow scheduling based on gradient disparities, neglect load balancing for Distributed Deep Learning (DDL) flows in DataCenter Network. In this paper, we review the PS traffic pattern in usual DDL training and propose that minimizing tail Flow Completion Time (FCT) of DDL flows is objective for DDL traffic Load Balancing. Besides, traffics of DDL differ from traditional DataCenter in temporal and spatial distribution. Current DCN Load Balancing become unadaptable to optimization requirements of DDL traffic, so we propose Tailor, a brand new load balancing scheme for DDL traffics training in DCN. Tailor is a solution without complex switch modification that sufficiently leverages the known parameter amount of DDL flows to discover congestion and reroutes by ECMP-like hash function to compensate bandwidth for slower flows. With large-scale simulation, we validate the effectiveness of Tailor. Tailor behaves well in both symmetric and asymmetric topology. It reduces time cost of DDL training by decreasing about 20-30% tail FCT than other load balancing mechanisms.

Diabetic Retinopathy (DR) is one of the severe microvascular conditions that affect the retina of the eye. It can result in permanent partial or total blindness if not discovered and treated early. Across the world, diabetes impacts in excess of 50% of people below 70 years of age. Many diabetic people, however, fail to notice the disease and suffer visual impairments as a result of the time it takes to see an ophthalmologist who screens and analyses the patient's retina. This research study focuses on the automated detection and categorization of DR based on distinct severity levels using fundus images. The method starts with a pre-processing step to get rid of any extraneous noise from the edges and aids in the focus on the area of interest. The DR fundus images were then categorized into various severity levels using DL models like VGG19, ResNet50, and Inception-V3 Deep Learning (DL) models. The logic for the presented DL model has been tested on the EyePACS (kaggle) DR dataset. In terms of classification accuracy, the suggested DL model with ResNet50 configuration surpassed other models such as VGG19 and Inception-V3 as well as other current models, according to the findings of the research.

A robust and lightweight deep attention multiple instance learning algorithm for predicting genetic alterations.

We propose an algorithm for the detection of retinal landmarks (optic nerve head or optic disc, macula, and vasculature) based on optic cup location and anatomical structural details from diabetic retinopathy (DR) images of both left and right eye. Our algorithm uses color fundus images obtained from mydriatic camera. The algorithm proceeds through four main steps 1. Color image pre-processing- to enhance and remove noise from the image. 2. Detection of optic nerve head-The optic nerve head is located using a fact that the optic cup is the brightest region in the optic nerve head. At the same time exudates (DR lesion) which appear in same gray level as optic nerve head are suppressed since we only concentrate on optic cup and optic nerve head. So by calculating the mean value of the intensities of 50 ? 50 subimages (50 ? 50 is approximate area of optic cup) throughout the image and then selecting that 50 ? 50 sub image with the highest mean value, locates optic cup in the image. Using this, optic nerve head is located by increasing the area of interest around the optic cup. Since we detect optic cup first, which is embedded in optic nerve head there will not be any false detection of optic nerve head when size and shape of the exudates (DR lesion) are same as that of optic nerve head 3.Detection of macula-It is located at a distance of approximately twice the diameter of the optic nerve head just below the horizontal axis of the optic nerve head. 4. Detection of vasculature-We have used logical AND operation on two images, one being a thresholded image and another being an edge detected image. The thresholding is done on an adaptive histogram equalized image. Edge detection is done using canny edge detector. Proposed algorithm has been tested on both normal and DR images. Detected optic disc area is validated by comparing it with expert ophthalmologists' hand-drawn ground-truths. The quantitative performance is evaluated by calculating sensitivity, specificity and predictive value. Overall sensitivity (Se), specificity (Sp) and predictive value (PV) obtained in detecting optic nerve head from normal images and from abnormal images are 97.2%, 99.72%, and 88.75% and 93.93%, 99.72%, and 84.18% respectively.

Massive volumes of data are needed for deep learning (DL) models to provide accurate diagnosis results. Numerous studies of fault diagnosis systems have demonstrated the effectiveness of DL models over shallow machine learning (SL) in terms of feature extraction, feature dimensional reduction and diagnosis performance. Occasionally, during data acquisition, a problem with a sensor renders some of the data potentially unsuitable for further analysis, leaving only a small data sample. To compensate for this deficiency, a DL model based on a stacked sparse autoencoder (SSAE) model is designed to deal with limited sample data. In this article, the fault diagnosis system is developed based on time-frequency image pattern recognition. Therefore, two gearbox datasets are used to evaluate the proposed diagnosis system. The results from the experiments prove that the proposed system is capable of achieving high diagnostic accuracy even with limited sample data. The proposed fault diagnosis system achieved 100% and 99% diagnosis performance on experimental gearbox and wind turbine gearbox datasets, respectively. The proposed diagnosis system increased diagnosis performance between 10% and 20% over the standard SSAE model. In addition, the proposed model achieved higher diagnosis performance compared to deep neural network and convolutional neural networks models.