Marble

Deep Learning (DL) is continuously adopted in many industrial applications at a rapidly increasing pace. This includes safety- and security-critical applications where errors in the DL system can lead to massive or even fatal losses. With the rise of DL adoption, trustworthy AI initiatives have been introduced that cover quality assurance principles such as robustness, fairness, and security of a DL system. Robustness aims to address the DL system's ability to predict new unseen inputs that are relevant to the DL application. Fairness aims to enable equal performance for all demographics, such as gender, age or ethnicity. Security aims to address new DL threats and defenses. Combining all three principles enables a more trustworthy DL system, which gives benefits to developers and end users. Developers gain confidence in deployment and end users gain trust in adopting a DL system. So far, the principles of robustness, fairness and security have been outlined in trustworthy initiatives from an expectations point of view under ISO or regulatory acts under the European Commission\cite{ai_act}. However, these initiatives give little attention to methods and quantitative assessment strategies on how those principles can be enabled and validated. Without assessment strategies, forming benchmarks to determine the quality of a DL system becomes a challenging task. This motivates research questions such as: ``\textit{At what point can a DL system be considered sufficient for deployment?}” and ``\textit{Is the data used for testing relevant and complete for the target application?}” In this thesis, we aim to address such questions and present foundational improvements by introducing out-of-distribution (OOD) awareness to robustness, fairness and security. Finally, the research methodologies are integrated into a national standard of Singapore which represents a high impact outcome of this work. First, we conduct a large-scale OOD empirical study to analyse which OOD technique is most suited for real-world DL system testing. Then, we propose an OOD testing criteria to enhance error discovery and propose distribution aware robustness enhancement by showing that filtering errors far from the trained distribution increases robustness by 21.5\%. We then enhance the field of OOD by introducing a fine-grained OOD technique assessing the ground truth of data augmentation and increasing accuracy by 31.9\% by guiding data augmentation selection with distribution awareness. We further apply the same augmentation technique to subsets of the training data to balance accuracy among demographics to enhance fairness. Finally, we introduce OOD defense approaches against two novel DL threats which fully evade traditional defenses. Ultimately, the thesis concludes with increasing quality assurance research impact by presenting the methodology that led to launching a national standard. The standard integrates our security research and evaluates it among four real-world use cases. This represents an important step in creating and validating trustworthy DL systems enabling increased trust in the intelligent solution and thereby further accelerating adoption in a safeguarded manner.

BackgroundMissed fractures are the most common diagnostic errors in musculoskeletal imaging and can result in treatment delays and preventable morbidity. Deep learning, a subfield of artificial intelligence, can be used to accurately detect fractures by training algorithms to emulate the judgments of expert clinicians. Deep learning systems that detect fractures are often limited to specific anatomic regions and require regulatory approval to be used in practice. Once these hurdles are overcome, deep learning systems have the potential to improve clinician diagnostic accuracy and patient care.Questions/purposesThis study aimed to evaluate whether a Food and Drug Administration–cleared deep learning system that identifies fractures in adult musculoskeletal radiographs would improve diagnostic accuracy for fracture detection across different types of clinicians. Specifically, this study asked: (1) What are the trends in musculoskeletal radiograph interpretation by different clinician types in the publicly available Medicare claims data? (2) Does the deep learning system improve clinician accuracy in diagnosing fractures on radiographs and, if so, is there a greater benefit for clinicians with limited training in musculoskeletal imaging?MethodsWe used the publicly available Medicare Part B Physician/Supplier Procedure Summary data provided by the Centers for Medicare & Medicaid Services to determine the trends in musculoskeletal radiograph interpretation by clinician type. In addition, we conducted a multiple-reader, multiple-case study to assess whether clinician accuracy in diagnosing fractures on radiographs was superior when aided by the deep learning system compared with when unaided. Twenty-four clinicians (radiologists, orthopaedic surgeons, physician assistants, primary care physicians, and emergency medicine physicians) with a median (range) of 16 years (2 to 37) of experience postresidency each assessed 175 unique musculoskeletal radiographic cases under aided and unaided conditions (4200 total case-physician pairs per condition). These cases were comprised of radiographs from 12 different anatomic regions (ankle, clavicle, elbow, femur, forearm, hip, humerus, knee, pelvis, shoulder, tibia and fibula, and wrist) and were randomly selected from 12 hospitals and healthcare centers. The gold standard for fracture diagnosis was the majority opinion of three US board-certified orthopaedic surgeons or radiologists who independently interpreted the case. The clinicians’ diagnostic accuracy was determined by the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, sensitivity, and specificity. Secondary analyses evaluated the fracture miss rate (1-sensitivity) by clinicians with and without extensive training in musculoskeletal imaging.ResultsMedicare claims data revealed that physician assistants showed the greatest increase in interpretation of musculoskeletal radiographs within the analyzed time period (2012 to 2018), although clinicians with extensive training in imaging (radiologists and orthopaedic surgeons) still interpreted the majority of the musculoskeletal radiographs. Clinicians aided by the deep learning system had higher accuracy diagnosing fractures in radiographs compared with when unaided (unaided AUC: 0.90 [95% CI 0.89 to 0.92]; aided AUC: 0.94 [95% CI 0.93 to 0.95]; difference in least square mean per the Dorfman, Berbaum, Metz model AUC: 0.04 [95% CI 0.01 to 0.07]; p < 0.01). Clinician sensitivity increased when aided compared with when unaided (aided: 90% [95% CI 88% to 92%]; unaided: 82% [95% CI 79% to 84%]), and specificity increased when aided compared with when unaided (aided: 92% [95% CI 91% to 93%]; unaided: 89% [95% CI 88% to 90%]). Clinicians with limited training in musculoskeletal imaging missed a higher percentage of fractures when unaided compared with radiologists (miss rate for clinicians with limited imaging training: 20% [95% CI 17% to 24%]; miss rate for radiologists: 14% [95% CI 9% to 19%]). However, when assisted by the deep learning system, clinicians with limited training in musculoskeletal imaging reduced their fracture miss rate, resulting in a similar miss rate to radiologists (miss rate for clinicians with limited imaging training: 9% [95% CI 7% to 12%]; miss rate for radiologists: 10% [95% CI 6% to 15%]).ConclusionClinicians were more accurate at diagnosing fractures when aided by the deep learning system, particularly those clinicians with limited training in musculoskeletal image interpretation. Reducing the number of missed fractures may allow for improved patient care and increased patient mobility.Level of EvidenceLevel III, diagnostic study.

Deep learning (DL) neural networks have only recently been employed to interpret chest radiography (CXR) to screen and triage people for pulmonary tuberculosis (TB). No published studies have compared multiple DL systems and populations. We conducted a retrospective evaluation of three DL systems (CAD4TB, Lunit INSIGHT, and qXR) for detecting TB-associated abnormalities in chest radiographs from outpatients in Nepal and Cameroon. All 1196 individuals received a Xpert MTB/RIF assay and a CXR read by two groups of radiologists and the DL systems. Xpert was used as the reference standard. The area under the curve of the three systems was similar: Lunit (0.94, 95% CI: 0.93–0.96), qXR (0.94, 95% CI: 0.92–0.97) and CAD4TB (0.92, 95% CI: 0.90–0.95). When matching the sensitivity of the radiologists, the specificities of the DL systems were significantly higher except for one. Using DL systems to read CXRs could reduce the number of Xpert MTB/RIF tests needed by 66% while maintaining sensitivity at 95% or better. Using a universal cutoff score resulted different performance in each site, highlighting the need to select scores based on the population screened. These DL systems should be considered by TB programs where human resources are constrained, and automated technology is available.

With recent breakthroughs in Deep Learning (DL), DL systems are increasingly deployed in safety-critical fields. Hence, some software testing methods are required to ensure the reliability and safety of DL systems. Since the rules of DL systems are inferred from training data, it is difficult to know the implementation rules about each behavior of DL systems. At the same time, Random Testing (RT) is a popular testing method and the knowledge about software implementation is not needed when we use RT. Therefore, RT is very suitable for the testing of DL systems. And the existing mechanisms for testing DL systems also depend heavily on RT by the labeled test data. In order to increase the effectiveness of RT for DL systems, we design, implement and evaluate the Adaptive Random Testing for DL systems (ARTDL), which is the first Adaptive Random Testing (ART) method to improve the effectiveness of RT for DL systems. ARTDL refers to the idea of ART. That is, fewer test cases are needed to detect failures by selecting the test case with the furthest distance from non-failure-causing test cases. Firstly, we propose the Feature-based Euclidean Distance (FED) as the distance metric that can be used to measure the difference between failure-causing inputs and non-failure-causing inputs. Secondly, we verify the availability of FED by presenting the failure pattern of DL models. Finally, we design ARTDL algorithm to generate the test cases that are more likely to cause failures based on the FED. We implement ARTDL to test top performing DL models in the field of image classification and automatic driving. The results show that, on average, the number of test cases used to find the first bug is reduced by 62.74% through ARTDL, compared with RT.

Deep learning (DL) defines a new data-driven programming paradigm that\nconstructs the internal system logic of a crafted neuron network through a set\nof training data. We have seen wide adoption of DL in many safety-critical\nscenarios. However, a plethora of studies have shown that the state-of-the-art\nDL systems suffer from various vulnerabilities which can lead to severe\nconsequences when applied to real-world applications. Currently, the testing\nadequacy of a DL system is usually measured by the accuracy of test data.\nConsidering the limitation of accessible high quality test data, good accuracy\nperformance on test data can hardly provide confidence to the testing adequacy\nand generality of DL systems. Unlike traditional software systems that have\nclear and controllable logic and functionality, the lack of interpretability in\na DL system makes system analysis and defect detection difficult, which could\npotentially hinder its real-world deployment. In this paper, we propose\nDeepGauge, a set of multi-granularity testing criteria for DL systems, which\naims at rendering a multi-faceted portrayal of the testbed. The in-depth\nevaluation of our proposed testing criteria is demonstrated on two well-known\ndatasets, five DL systems, and with four state-of-the-art adversarial attack\ntechniques against DL. The potential usefulness of DeepGauge sheds light on the\nconstruction of more generic and robust DL systems.\n

Testing deep learning (DL) systems requires extensive and diverse, yet valid, test inputs. While synthetic test input generation methods, such as metamorphic testing, are widely used for DL testing, they risk introducing invalid inputs that do not accurately reflect real-world scenarios. Invalid test inputs can lead to misleading results. Hence, there is a need for automated validation of test inputs to ensure effective assessment of DL systems. In this paper, we propose a test input validation approach for vision-based DL systems. Our approach uses active learning to balance the trade-off between accuracy and the manual effort required for test input validation. Further, by employing multiple image-comparison metrics, it achieves better results in classifying valid and invalid test inputs compared to methods that rely on single metrics. We evaluate our approach using an industrial and a public-domain dataset. Our evaluation shows that our multi-metric, active learning-based approach produces several optimal accuracy-effort trade-offs, including those deemed practical and desirable by our industry partner. Furthermore, provided with the same level of manual effort, our approach is significantly more accurate than two state-of-theart test input validation methods, achieving an average accuracy of 97%. Specifically, the use of multiple metrics, rather than a single metric, results in an average improvement of at least 5.4% in overall accuracy compared to the state-of-the-art baselines. Incorporating an active learning loop for test input validation yields an additional 7.5% improvement in average accuracy, bringing the overall average improvement of our approach to at least 12.9% compared to the baselines.

Deep learning has become an increasingly heated topic as artificial intelligence is on the rise. At the same time, hardware restrictions in real applications have driven the investigation of combining another rising technique, stochastic computing (SC), with deep learning systems to achieve low power costs. By far, operations successfully implemented include addition, multiplication, inner product, and other more complicated nonlinear functions such as hyperbolic tangent (tanh) with linear finite state machines (FSM). The inner product implementation realizes convolution, a core of neural networks, therefore encouraging SC- based deep learning neural network implementations. Meanwhile, extremely long bitstream lengths are needed to achieve satisfying accuracy, especially for large-scale deep learning systems, causing latency issues. The integration of parallelism is thus considered in an attempt to alleviate the latency issue. In this paper, an optimization to stochastic computing based deep learning system is proposed by introducing parallel FSM implementations to replace serial ones generally used in previous works. Substituting serial linear FSMs with several parallel linear FSMs of the same size yet with shorter bitstream length, parallel FSM aims at trading hardware for processing latency. The accuracy performance of a sample parallel FSM unit is evaluated against its counterpart in serial implementation before a case study verifies that the replacement sacrifices little accuracy, while reducing computing time exponentially in actual deep learning system realizations.

Certifiable robustness, the functionality of verifying whether the given region surrounding a data point admits any adversarial example, provides guaranteed security for neural networks deployed in adversarial environments. A plethora of work has been proposed to certify the robustness of feed-forward networks, e.g., FCNs and CNNs. Yet, most existing methods cannot be directly applied to recurrent neural networks (RNNs), due to their sequential inputs and unique operations. In this paper, we present Cert-RNN, a general framework for certifying the robustness of RNNs. Specifically, through detailed analysis for the intrinsic property of the unique function in different ranges, we exhaustively discuss different cases for the exact formula of bounding planes, based on which we design several precise and efficient abstract transformers for the unique calculations in RNNs. Cert-RNN significantly outperforms the state-of-the-art methods (e.g., POPQORN) in terms of (i) effectiveness -- it provides much tighter robustness bounds, and (ii) efficiency -- it scales to much more complex models. Through extensive evaluation, we validate Cert-RNN's superior performance across various network architectures (e.g., vanilla RNN and LSTM) and applications (e.g., image classification, sentiment analysis, toxic comment detection, and malicious URL detection). For instance, for the RNN-2-32 model on the MNIST sequence dataset, the robustness bound certified by Cert-RNN is on average 1.86 times larger than that by POPQORN. Besides certifying the robustness of given RNNs, Cert-RNN also enables a range of practical applications including evaluating the provable effectiveness for various defenses (i.e., the defense with a larger robustness region is considered to be more robust), improving the robustness of RNNs (i.e., incorporating Cert-RNN with verified robust training) and identifying sensitive words (i.e., the word with the smallest certified robustness bound is considered to be the most sensitive word in a sentence), which helps build more robust and interpretable deep learning systems. We will open-source Cert-RNN for facilitating the DNN security research.

Deep learning systems for automatic diagnosis of infant cry signals

Deep learning systems have fundamentally improved genomic analysis by probing genomic design patterns that relate to different diseases. Working with confusing extensive genomic data causes difficulties during traditional genetics analysis. Recurrent Neural Networks (RNNs) applications provide the best benefits for sequential genomic data evaluation within deep learning methods. The networks excel at maintaining sequential order which results in precise prediction of mutations together with rigorous evaluation of gene expression and exact identification of disease categories. RNNs apply their capacity to recognize long patterns to find gene transformation links to illnesses and this enhances precise individual medical diagnosis results. Through their implementation in scientific studies RNNs enable scientists to study transcriptomic proteomic and epigenomic data combinations in order to identify new biological links. RNN development meets ongoing challenges because of its high computational requirements and unexplained model logic and scarce database accessibility. This paper examines RNNs in genomic analysis by reviewing their application procedures and their benefits for genetic medicine diagnosis and precision treatment applications.

Identifying named entities is vital for many Natural Language Processing (NLP) applications. Much of the earlier work for identifying named entities focused on using handcrafted features and knowledge resources (feature engineering). This is a barrier for resource-scarce languages as many resources are not readily available. Recently, Deep Learning techniques have been proposed for various NLP tasks requiring little/no hand-crafted features and knowledge resources, instead the features are learned from the data. Many proposed deep learning solutions for Named Entity Recognition (NER) still rely on feature engineering as opposed to feature learning. However, it is not clear whether the deep learning system or the engineered features are responsible for the positive results reported. This is in contrast with the goal of deep learning systems i.e., to learn the features from the data itself. In this study, we show that a feature learned deep learning system is a viable solution to NER task. We test our deep learning systems on CoNLL English and Spanish NER datasets. Our system is able to give comparable results with the existing state-of-the-art feature engineered systems for English. We report the best performance of 89.27 F-Score for English when comparing with systems which do not use any handcrafted features or knowledge resources. Evaluation of our trained system on out-of-domain data indicate that the results are promising with the reported results. Our system when tested on Spanish NER achieves the best reported F-Score of 82.59 indicating its applicability to other languages.

Laryngeal cancer detection on throat area images is a vital application of medical image diagnosis and computer vision (CV) in the healthcare domain. It contains the analysis and detection of cancerous or abnormal tissues from the larynx, an essential part of the respiratory and vocal systems. Several machine learning (ML) and deep learning (DL) systems are executed for classifying the extraction features as both cancerous and healthy tissue. Convolutional Neural Networks (CNNs) and recurrent neural networks (RNNs) have shown promise in this context. With this motivation, this study designs a new Laryngeal Cancer Detection and Classification using the Aquila Optimization Algorithm with Deep Learning (LCDC-AOADL) technique on neck region images. The purpose of the LCDC-AOADL technique is to examine the histopathological images for the recognition and classification of Laryngeal Cancer. In the presented LCDC-AOADL technique, the Inceptionv3 model is used for the feature extraction process. Besides, the LCDC-AOADL technique employed a deep belief network (DBN) model for the identification and classification of LC. Moreover, the AOA is utilized for the hyperparameter tuning of the DBN model which results in improved detection rate. The simulation analysis of the LCDC-AOADL method is validated on the benchmark Laryngeal dataset. The experimental results pointed out the enhanced detection results of the LCDC-AOADL technique over other recent approaches with a maximum accuracy of 96.02%, precision of 92.10%, recall of 91.87%, and F-score of 91.86%.

Nuclear safeguards inspectors expend significant time and maintain intense focus in reviewing video surveillance for safeguards relevant events. To increase efficiency and reduce the time burden of safeguards inspectors performing surveillance data review, this paper presents a novel deep learning (DL) systems concept to integrate generalized DL models into the safeguards surveillance review workflow. The Agency is investigating several DL algorithms for object recognition, localization, tracking, and flagging relevant activities. The project team is working closely with nuclear safeguards inspectors to identify review use cases (based on specific safeguards objectives) and collect their associated requirements. We focused on CANDU and LWR Nuclear Power Plants (NPPs) and their associated dry storage areas as these present a particularly heavy burden on the inspector surveillance review process due to the number of these facilities under safeguards worldwide. Initial DL algorithm results on safeguards data are promising. Using a convolutional neural network, the team attained a mean average precision (mAP) of 92.9% identifying and localizing spent fuel (SF) casks from a 475 surveillance image dataset. Further, the team had initial success in training a recurrent neural network to identify reactor area activities in video clips, successfully indicating when SF casks enter or exit a pool. We discuss how such DL algorithms would be integrated into the Next Generation Surveillance Review (NGSR) software application. Another issue impacting review productivity is the long time inspectors may have to wait when running these algorithms in NGSR. We present a concept to pre-process remotely collected surveillance data with DL models as the data arrives to IAEA headquarters so that results are already available when starting a new review in NGSR. The proposed DL system concept shows a pathway and workflow for increasing an inspector’s surveillance review productivity by quickly and accurately identifying declared and undeclared safeguards relevant objects and activities in large quantities of surveillance imagery data.

Technological improvements lead big data producing, processing and storing systems. These systems must contain extraordinary capabilities to overcome complexity of the big data. Therefore, the methodologies utilized for data analysis have been evolved due to the increase in importance of extracting information from big data. Healthcare systems are important systems dealing with big data analysis. Deep learning is the most applied data analysis method. It becomes one of the most popular and up-to-date artificial neural network types with deep representation ability. Another powerful ability of deep learning is providing feature learning through convolutional neural networks. Deep learning has wide implementation areas in medical applications from diagnosis to treatment. Various deep learning methods are applied to the biomedical problems. In many applications, deep learning solutions are modified in accordance with the requirements of the problems. Through this chapter the most popular and up-to-date deep learning solutions to biomedical problems are discussed. Studies are analyzed according to problem characteristic, importance of solution, requirements and deep learning approaches to solve them. Since the deep learning systems have very effective image and pattern recognition ability, biomedical imaging becomes one of the most suitable application areas. During the first diagnosis and continuous tracking phase of the patients, deep learning systems offer very effective aids to the medicine. Although organ, disease or data type classifications are possible for biomedical application categorization, organ and disease combination are taken into consideration in the chapter.

Deep learning has evolved as a strong and efficient framework that can be applied to a broad spectrum of complex learning problems which were difficult to solve using the traditional machine learning techniques in the past. The advancement of deep learning has been so radical that today it can surpass human‐level performance. As a consequence, deep learning is being extensively used in most of the recent day‐to‐day applications. However, efficient deep learning systems can be jeopardised by using crafted adversarial samples, which may be imperceptible to the human eye, but can lead the model to misclassify the output. In recent times, different types of adversaries based on their threat model leverage these vulnerabilities to compromise a deep learning system where adversaries have high incentives. Hence, it is extremely important to provide robustness to deep learning algorithms against these adversaries. However, there are only a few strong countermeasures which can be used in all types of attack scenarios to design a robust deep learning system. Herein, the authors attempt to provide a detailed discussion on different types of adversarial attacks with various threat models and also elaborate on the efficiency and challenges of recent countermeasures against them.