A comprehensive study on deep learning bug characteristics

Modern software systems are increasingly including machine learning (ML) as an integral component. However, we do not yet understand the difficulties faced by software developers when learning about ML libraries and using them within their systems. To fill that gap this thesis reports on a detailed (manual) examination of 3,243 highly-rated Q&A posts related to ten ML libraries, namely Tensorflow, Keras, scikitlearn, Weka, Caffe, Theano, MLlib, Torch, Mahout, and H2O, on Stack Overflow, a popular online technical Q&A forum. Our findings reveal the urgent need for software engineering (SE) research in this area. The second part of the thesis particularly focuses on understanding the Deep Neural Network (DNN) bug characteristics. We study 2,716 high-quality posts from Stack Overflow and 500 bug fix commits from Github about five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand the types of bugs, their root causes and impacts, bug-prone stage of deep learning pipeline as well as whether there are some common antipatterns found in this buggy software. While exploring the bug characteristics, our findings imply that repairing software that uses DNNs is one such unmistakable SE need where automated tools could be beneficial; however, we do not fully understand challenges to repairing and patterns that are utilized when manually repairing DNNs. So, the third part of this thesis presents a comprehensive study of bug fix patterns to address these questions. We have studied 415 repairs from Stack Overflow and 555 repairs from Github for five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand challenges in repairs and bug repair patterns. Our key findings reveal that DNN bug fix patterns are distinctive compared to traditional bug fix patterns and the most common bug fix patterns are fixing data dimension and neural network connectivity. Finally, we propose an automatic technique to detect ML Application Programming Interface (API) misuses. We started with an empirical study to understand ML API misuses. Our study shows that ML API misuse is prevalent and distinct compared to non-ML API misuses. Inspired by these findings, we contributed Amimla (Api Misuse In Machine Learning Apis) an approach and a tool for ML API misuse detection. Amimla relies on several technical innovations. First, we proposed an abstract representation of ML pipelines to use in misuse detection. Second, we proposed an abstract representation of neural networks for deep learning related APIs. Third, we have developed a representation strategy for constraints on ML APIs. Finally, we have developed a misuse detection strategy for both single and multi-APIs. Our experimental evaluation shows that Amimla achieves a high average accuracy of ∼80% on two benchmarks of misuses from Stack Overflow and Github.

Differential testing offers a promising strategy to alleviate the test oracle problem by comparing the test results between alternative implementations. However, existing differential testing techniques for deep learning (DL) libraries are limited by the key challenges of finding alternative implementations (called \(counterparts\) ) for a given API and subsequently generating diverse test inputs. To address the two challenges, this paper introduces DLL ens , an LLM-enhanced differential testing technique for DL libraries. The first challenge is addressed by an observation that DL libraries are commonly designed to support the computation of a similar set of DL algorithms. Therefore, the counterpart of a given API’s computation could be successfully synthesized through certain composition and adaptation of the APIs from another DL library. DLL ens incorporates a novel counterpart synthesis workflow, leveraging a large language model (LLM) to search for valid counterparts for differential testing. To address the second challenge, DLL ens incorporates a static analysis technique that extracts the path constraints from the implementations of a given API and its counterpart to guide diverse test input generation. The extraction is facilitated by LLM’s knowledge of the concerned DL library and its upstream libraries. DLL ens incorporates validation mechanisms to manage the LLM’s hallucinations in counterpart synthesis and path constraint extraction. We evaluate DLL ens on two popular DL libraries, TensorFlow and PyTorch. Our evaluation shows that DLL ens synthesizes counterparts for 1.84 times as many APIs as those found by state-of-the-art techniques on these libraries. Moreover, under the same time budget, DLL ens covers 7.23% more branches and detects 1.88 times as many bugs as state-of-the-art techniques on 200 randomly sampled APIs. DLL ens has successfully detected 71 bugs in recent TensorFlow and PyTorch libraries. Among them, 59 are confirmed by developers, including 46 confirmed as previously unknown bugs, and 10 of these previously unknown bugs have been fixed in the latest version of TensorFlow and PyTorch.

Background Until now, no method is available for automatic extraction of indicators of image quality and view standardization from echocardiographic recordings. Thus, description of data quality has been indirect, often by description of operators experience and the methodology used. Peak systolic global longitudinal strain (GLS) is a sensitive measure of left ventricular (LV) function, still hampered by random variability due to suboptimal standardization of recordings and variability in data processing by the operators. Data on how acquisitions and data processing influence GLS and its variability are scarce. Objectives Aim was to study the importance of image acquisition and data processing by experienced operators for GLS in a large healthy population by automatically extracting quality indicators using novel deep learning based image analysis software and vendor specific metadata. Secondly, we aimed to study how these characteristics influenced the reference ranges and GLS variability. Methods Participants from a large echocardiographic study were included. Echocardiography was performed according to current recommendations. Two experienced operators and two expert cardiologists read and re-read all GLS recordings (1,412 paired analyses). Acquisition- and data processing characteristics were extracted using deep learning and GLS software metadata providing specific data on position of landmarks in the view, as well as rotation and tilt of the cut-plane according to the best possible standard. From these measurements several quality indicators were provided (Table 1). Results Mean age for the 1,412 participants was 58±12 years and 56% were women. Averaged apical LV foreshortening in the recordings was ≤2 mm and the view specific recordings were well standardized, with high alignment with the preferred cut-plane for tilt and rotation across the three apical views (Figure 1). Most quality indicators influenced GLS and absolute GLS was lower with longer LVs (-0.2% per 1 cm, p=0.003) and longer distance from the transducer to the LV apex (-0.3% per 1 cm, p=0.02) in the recordings, and also for wider regions of interest during data processing (-0.7% per 1 mm, p <0.001). Absolute GLS was 0.7% higher per mm systolic foreshortening of the apical region of interest, and when more knots were placed under initialization of the ROI (0.1% per knot, p=0.004). In test-retest analyses, the results for agreement between operators followed the above presented data. Conclusions Novel quality indicators were successfully extracted using novel DL software and strain specific metadata. Even though acquisitions and data processing were well standardized, several quality indicators from both steps influenced GLS. Deep learning software and data processing metadata may provide quality indicators of the echocardiographic databases of importance to interpret the study outcomes and motivate researchers to optimize echocardiographic acquisitions and analyses.

PurposeThe purpose of this paper is to examine out‐of‐stock (OOS) challenges in the independent grocery sector with a special emphasis on in‐store root causes. The analysis aims to assess the extent and root causes for OOS in the independently controlled retail sector and provide a comparison of these results with the centrally controlled chain sector in Denmark. The paper also seeks to examine the practices and challenges of store ordering and store replenishment processes in the independent sector and identify practical implications for store management and other members of the grocery channels.Design/methodology/approachThe research design includes two studies. Study‐I surveys and compares the extent and root causes of OOS of 42 stores from eight chains in the two sectors. Study‐II identifies, based on qualitative interviews with 17 store managers/owners of independent stores, specific insights on store operations.FindingsThe quantitative study shows that the OOS rates in the independent sector are significantly higher than in the centrally organized sector. Furthermore, the independent grocery sector faces OOS challenges in more categories than the centrally controlled sector. The study also reveals a very large variation in the performance of independent stores. Contrary to the centrally controlled chain store sector, the major root cause for OOS in the independent sector is found in the store ordering process. The qualitative study shows that the main discriminating issues between stores with a low and a high OOS rate are: store management emphasis and commitment to OOS issues; the resulting priority and managerial guidance in store ordering and store replenishment tasks; the stability of staff and the proper planning for replenishment peaks; the store size and resulting space conditions; and the use of appropriate decision heuristics and use of inventory in store ordering.Practical implicationsThe findings have practical implications for store management as it reveals practices to pursue and to avoid. Also, it is argued that the findings have implications for the other members of the grocery channels.Originality/valueEmpirically, this paper explores two issues not being dealt with in depth in previous research, i.e. the OOS challenge in the independent sector and the emphasis on store operations.

Over the past decade, deep learning (DL) has been successfully applied to many industrial domain-specific tasks. However, the current state-of-the-art DL software still suffers from quality issues, which raises great concern especially in the context of safety- and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently addressed, on which a DL software makes incorrect decisions. Such defects occur through either intentional attack or physical-world noise perceived by input sensors, potentially hindering further industry deployment. The intrinsic uncertainty nature of deep learning decisions can be a fundamental reason for its incorrect behavior. Although some testing, adversarial attack and defense techniques have been recently proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty.

Deep learning has influenced almost all major domains of science, technology and engineering fields. The deep learning revolution started with the ground-breaking accuracy obtained in a computer vision problem. Machine vision-based inspection has been one of the pioneering applications of computer vision for industrial applications. The adoption of deep learning for machine vision applications took some time, and the current adoption rate though is satisfactory, it is observed that there is still a long way to go. The contents of the chapter is intended for beginners and managers who are evaluating application of deep learning techniques for vision-based automated inspection. This chapter presents detailed insights of merits and limitations of deep learning techniques for automated inspection tasks, especially in comparison to non-deep learning route. The various aspects of commissioning such as important pitfalls to be cautious about before choosing deep learning, deep learning software, deep learning hardware, types of deep learning networks and their inferences and potential applications in various types of industries are also discussed.KeywordsAutomated inspectionDeep learningComputer visionMachine visionVision hardwareDeep neural networksVision software

Purpose: The purpose of this study is to fill current information cavities in the present material on the determinants of radicalization. The radicalization has badly affected good governance. Education, health, socio-economic and political system are major components of good governance. This research paper examines the emerging trends of Jihad, Extremism and Radicalization in Pakistan.
 Design/Methodology/Approach: A survey was conducted a total of 4200 respondents across the country. There was no response from 200 persons and received the response of 4000 respondents, with 2800 people (70 percent) urban and 1200 respondents (30 percent) from rural area. The figure of respondents from each of the four provinces of Pakistan (Khyber Pakhtunkhwa, Punjab, Balochistan, and Sindh, ), Gilgit Baltistan, and Azad Jammu and Kashmir (AJK),entities reveals the percentage of population of each area vis-à-vis the total population of the country.
 Findings: The results revealed that the causes of radicalization and extremism are as varied as they are abundant. The study of radicalization, Jihad and extremism in both contexts has sought to find out the causes of radicalization. It is concluded that the root causes of the Islamic militancy and religious radicalization in Pakistan are generally religious, political, ideological, economic, and social in nature.
 Implications/Originality/Value: This research paper gives a comprehensive viewpoint and analysis amongst religion, radicalization, governance, extremism, politics, Islamic militancy, sectarianism, conflict and identity. The major root causes of militancy and radicalization were traced out which will be helpful to overcome Islamic militancy, extremism, radicalization and sectarianism in Pakistan.

Hypertension is the major root cause of blood pressure (BP) which in turn causes different cardiovascular diseases (CVDs). Hence BP need to be regularly monitored for preventing CVDs since it can be diagnosed and controlled through constant observation. Photoplethysmography (PPG) is identified as an important low-cost technology for facilitating a convenient and effective process in the early detection of CVDs. Different cardiovascular parameters such as blood oxygen saturation, heart rate, blood pressure, etc can be determined using the PPG technology. These cardiovascular parameters when given as input to the deep learning model is determined to diagnosis CVDs with maximized accuracy to an expected level. In this paper, Hybrid ResNet and Bidirectional LSTM-based Deep Learning Model (HRBLDLM) is proposed for diagnosing CVDs from PPG signals with due help in supporting the physicians during the process of continuous monitoring. This deep learning model mainly concentrated on the diagnosis of stage 1 hypertension, stage 2 hypertension, prehypertension, and normal CVDs with maximized accuracy using PPG signals. The PPG signals determined from PPG-BP dataset for investigation were recorded using IoT-based wearable patient monitoring (WPM) devices during the physical activity that includes high intensity, medium and low intensity movements involved driving, sitting and walking. The experiments conducted for this proposed deep learning model using PPG-BP dataset confirmed a better classification accuracy of 99.62% on par with the baseline PPG-based deep learning models contributed for detecting CVDs.

In this paper, we propose a platform for analyzing data using open source deep learning engine. As the demand for deep learning technology grows, research is needed on a deep learning platform that can process and analyze various learning data. We use a distributed TensorFlow and a TensorFlow serving to provide a deep learning platform for analyzing various data. In order to process large capacity learning data, it preprocesses data in size and type that can be processed by the platform, and provides the function to distribute learning data appropriately according to the processing capability of each node. In the future, we will implement a deep learning platform based on the platform design method, and then we will develop a deep learning model using actual large capacity learning data, and optimize the platform function.

Background A novel, recently FDA-authorized software uses deep learning (DL) to provide prescriptive transthoracic echocardiography (TTE) guidance, allowing novices to acquire standard TTE views. The DL model was trained by >5,000,000 observations of the impact of probe motion on image orientation/quality. This study evaluated whether novice-acquired TTE images guided by this software were of diagnostic quality in patients with and without implanted electrophysiological (EP) devices, focusing on RV size and function, which were thought to be sensitive to EP devices. Some aspects of the study have previously been presented. Methods 240 patients (61±16 years old, 58% male, 33% BMI >30 kg/m2, 91% with cardiac pathology) were recruited. 8 nurses without echo experience each acquired 10 view TTEs in 30 patients guided by the software. 235 of the patients were also scanned by a trained sonographer without assistance from the software. 5 Level 3 echocardiographers independently assessed the diagnostic quality of the TTEs acquired by the nurses and sonographers to evaluate the effect of EP devices on DL software performance. Results Nurses using the AI-guided acquisition software acquired TTEs of sufficient quality to make qualitative assessments of right ventricular (RV) size and function in greater than 80% of cases for patients with and without implanted EP devices (Table). There was no significant difference between nurse- and sonographer-acquired scans. Conclusion These results indicate that new DL software can guide novices to obtain TTEs that enable qualitative assessment of RV size even in the presence of implanted EP devices. The results of the comparison to sonographer-acquired exams indicate the software performance is robust to presence of pacemaker/ICD leads visible in the images (Figure). Nurse-acquired TTE with visible ICD lead Funding Acknowledgement Type of funding source: Private company. Main funding source(s): Caption Health, Inc.

Recent breakthroughs in deep learning and artificial intelligence technologies have enabled numerous mobile applications. While traditional computation paradigms rely on mobile sensing and cloud computing, deep learning implemented on mobile devices provides several advantages. These advantages include low communication bandwidth, small cloud computing resource cost, quick response time, and improved data privacy. Research and development of deep learning on mobile and embedded devices has recently attracted much attention. This paper provides a timely review of this fast-paced field to give the researcher, engineer, practitioner, and graduate student a quick grasp on the recent advancements of deep learning on mobile devices. In this paper, we discuss hardware architectures for mobile deep learning, including Field Programmable Gate Arrays (FPGA), Application Specific Integrated Circuit (ASIC), and recent mobile Graphic Processing Units (GPUs). We present Size, Weight, Area and Power (SWAP) considerations and their relation to algorithm optimizations, such as quantization, pruning, compression, and approximations that simplify computation while retaining performance accuracy. We cover existing systems and give a state-of-the-industry review of TensorFlow, MXNet, Mobile AI Compute Engine (MACE), and Paddle-mobile deep learning platform. We discuss resources for mobile deep learning practitioners, including tools, libraries, models, and performance benchmarks. We present applications of various mobile sensing modalities to industries, ranging from robotics, healthcare and multimedia, biometrics to autonomous drive and defense. We address the key deep learning challenges to overcome, including low quality data, and small training/adaptation data sets. In addition, the review provides numerous citations and links to existing code bases implementing various technologies. These resources lower the user’s barrier to entry into the field of mobile deep learning.

AI-enabled applications are prolific today. Deep Learning (DL) libraries, such as PyTorch and Tensorflow, provide the building blocks for the AI components of these applications. As any piece of software, these libraries can be buggy. An impressive number of bug-finding techniques to address this problem have been proposed, but the lack of a curated set of reproducible bugs in DL libraries hinders credible evaluation of these techniques. We present BugsInDLLs, a database of curated reproducible bugs to fill that gap. Unique challenges exist in this context, such as installing drivers of specific CUDA versions to reproduce certain GPU-related bugs. Our dataset currently consists of 112 environments to reproduce bugs across three popular DL libraries, namely, JAX, Tensorflow, and PyTorch.

Coverage-enhanced fault diagnosis for Deep Learning programs: A learning-based approach with hybrid metrics

As Deep Learning (DL) is continuously adopted in many safety critical applications, its quality and reliability start to raise concerns. Similar to the traditional software development process, testing the DL software to uncover its defects at an early stage is an effective way to reduce risks after deployment. Although recent progress has been made in designing novel testing techniques for DL software, the distribution of generated test data is not taken into consideration. It is therefore hard to judge whether the identified errors are indeed meaningful errors to the DL application. Therefore, we propose a new distribution aware testing technique which aims to generate new unseen test cases relevant to the underlying DL system task. Our results show that this technique is able to filter up to 55.44% of error test case on CIFAR-10 and is 10.05% more effective in enhancing robustness.

This paper presents an investigation on field returned open and short failures related to printed circuit board (PCB), including via hole crack, prepreg crack and insufficient circuit etching. After an experimental study with cross section, time domain reflectometry (TDR), and finite element (FE) modelling, it was found that weak plating and corrosion induced via hole crack was a major root cause of interconnect open failures. Such a failure was always mistakenly treated as solder joint open. Except for via hole crack, PCB prepreg crack was another contributor for open failures. The prepreg material for some lot code in production was in poor quality in terms of long-term durability. Dye and pry and cross section investigation showed that for some field returned units, the pad-to-prepreg interface was the weakest adhesion part in the entire solder joint interconnects. Crack propagated across the whole support and hence pad cratering occurred upon overstress. For the circuit short problem, insufficient etching of copper traces and pads was found to be a dominant contributor in harsh environment for a long time. Particularly, corrosion induced dendrite growth brought a high risk of circuit short of insufficiently etched conductors for the fine-pitch lead-free application. In summary, it is important to monitor and control the incoming quality of incoming PCBs so that the the risk of high field return rate and high cost for repair can be minimized.