A probabilistic framework for mutation testing in deep neural networks

Testing Deep Learning (DL) systems is a complex task as they do not behave like traditional systems would, notably because of their stochastic nature. Nonetheless, being able to adapt existing testing techniques such as Mutation Testing (MT) to DL settings would greatly improve their potential verifiability. While some efforts have been made to extend MT to the Supervised Learning paradigm, little work has gone into extending it to Reinforcement Learning (RL) which is also an important component of the DL ecosystem but behaves very differently from SL. This paper builds on the existing approach of MT in order to propose a framework, RLMutation, for MT applied to RL. Notably, we use existing taxonomies of faults to build a set of mutation operators relevant to RL and use a simple heuristic to generate test cases for RL. This allows us to compare different mutation killing definitions based on existing approaches, as well as to analyze the behavior of the obtained mutation operators and their potential combinations called Higher Order Mutation(s) (HOM). We show that the design choice of the mutation killing definition can affect whether or not a mutation is killed as well as the generated test cases. Moreover, we found that even with a relatively small number of test cases and operators we manage to generate HOM with interesting properties which can enhance testing capability in RL systems.

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does not detect some (non-equivalent) mutants, T can be improved by adding test cases that detect those mutants. Mutants are created by applying mutation operators. Mutation operators are important because they define the characteristics of the system that are tested as well as the characteristics that are improved in the test suite. While mutation operators are well defined for a number of programming paradigms such as sequential or multi-threaded, to the best of our knowledge, mutation operators have not been defined for the actor programming model. In this paper, we define and classify mutation operators that can be used for mutation testing of actor programs.

Clean room software engineering is a formal software development method that can strictly engineer software development and eliminate defects before they can cause serious harm. The clean room software engineering model has its own advantages and disadvantages in use. To use it in traditional software engineering, it is necessary to clarify the relationship between it and traditional software engineering methods and practices. Based on the technology and principles of the clean room software process, this paper gets the advantages and disadvantages of the clean room. Corresponding the traditional method-based software process to the key technology of the clean room, making the traditional management-based software engineering method and the tailored clean room compatible, and the software engineering practice of the clean room software process is compared with the traditional software engineering practice. To discuss the clean room and traditional software engineering, in order to expand the use of clean room methods, improve the clean room software process, and improve the quality of software.

Investigating the learning approaches of students in nursing education

Achieving energy efficiency (EE) fairness among heterogeneous mobile devices will become a crucial issue in future wireless networks. This paper investigates a deep learning (DL) approach for improving EE fairness performance in interference channels (IFCs) where multiple transmitters simultaneously convey data to their corresponding receivers. To improve the EE fairness, we aim to maximize the minimum EE among multiple transmitter–receiver pairs by optimizing the transmit power levels. Due to fractional and max-min formulation, the problem is shown to be non-convex, and, thus, it is difficult to identify the optimal power control policy. Although the EE fairness maximization problem has been recently addressed by the successive convex approximation framework, it requires intensive computations for iterative optimizations and suffers from the sub-optimality incurred by the non-convexity. To tackle these issues, we propose a deep neural network (DNN) where the procedure of optimal solution calculation, which is unknown in general, is accurately approximated by well-designed DNNs. The target of the DNN is to yield an efficient power control solution for the EE fairness maximization problem by accepting the channel state information as an input feature. An unsupervised training algorithm is presented where the DNN learns an effective mapping from the channel to the EE maximizing power control strategy by itself. Numerical results demonstrate that the proposed DNN-based power control method performs better than a conventional optimization approach with much-reduced execution time. This work opens a new possibility of using DL as an alternative optimization tool for the EE maximizing design of the next-generation wireless networks.

Software testing is the dominant method for quality assurance and control in software engineering [1] , [2] . Test suites serve as quality gates to safeguard against programming faults. But not every test suite is written equally. We usually gauge its quality using metrics such as code coverage. These assess how much of the code base has been covered. However, they do not tell if the tests actually test and verify the intentions. Mutation testing does this by deliberately injecting faults into the system under test and verifying how many of them the test suite can detect. For every injected fault that is not detected by the test suite, an additional test should be written. In the academic community, mutation testing is acknowledged as the most promising technique for automated assessment of the strength of a test suite [3] , [4] .

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

In order to solve the current problems( expensive testing cost and low efficiency of test case generation) in mutation test for Simulink models,a mutation testing process and an optimized set of mutation operators were proposed for Simulink models based on program mutation according to the current mutation operators for the Simulink models. Experiments show that this set of mutation operators can effectively reduce the generation number of mutation models without prejudice to the mutation score of testing case set,thus it will effectively save the testing cost. Then a search-based test case generation method for Simulink models mutation testing was described. The test case generation problem was transformed into the objective function minimization problem,and the test cases which can kill the mutation models were ultimately obtained through the optimization of objective function by algorithm of simulated annealing. Finally,the application of a typical case for the method verified the correctness and effectiveness.

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static analyzer. Since static analyzers, in general, report a substantial number of false positive warnings, the intention of this study is to define a prioritization approach of static warnings based on their correspondence with mutations. On the other hand, knowing that Mutation Test has a high application cost, another possibility is to try to identify mutations of some specific mutation operators, which an automatic static analyzer is not adequate to detect. Therefore, this information can be used to prioritize the order of incrementally applying mutation operators considering, firstly, those with no correspondence with static warnings. In both cases, contributing to the establishment of incremental strategies on using automatic static analysis or mutation testing or even a combination of them. We used mutation operators as a fault model to evaluate the direct correspondence between mutations and static warnings. The main advantage of using mutation operators is that they generate a large number of programs containing faults of different types, which can be used to decide the ones most probable to be detected by static analyzers. We provide evidences on the correspondence between mutations and some types of static warnings. The results obtained for a set of 19 open-source programs indicate that: 1) static warnings may be prioritized based on their correspondence level with mutations; 2) specific set of mutation operators and their mutations may be prioritized based on their correspondence level with warnings. It is possible to provide an incremental testing strategy aiming at reducing the cost of both static analysis and mutation testing using the correspondence information between these activities/artifacts.

Mutation testing is a fault based testing technique which examines the adequacy of test cases by generating faulty programs, called mutants, to be run against them. However, one of the pitfalls of mutation testing is its high computational cost, rendering it impractical for industry use. In this survey, we study mutation generation techniques, cost reduction methods and test-data generation approaches found in the literature, before presenting some tools and applications for mutation testing.

Semantic mutation testing

Mutation testing is a technique in which faults (mutants) are injected into a program or application to assess its test suite effectiveness. It works by inserting mutants and running the application’s test suite to identify if the mutants are detected (killed) or not (survived) by the tests. Although computationally expensive, it has proven to be an effective method to assess application test suites. Several mutation testing frameworks and tools have been built for the various programing languages, however, very few tools have been built for the JavaScript language, more specifically, there is a lack of mutation testing tools for the Node.js runtime and npm based applications. The npm Registry is a public collection of modules of open-source code for Node.js, front-end web applications, mobile applications, robots, routers, and countless other needs of the JavaScript community. The over 700,000 packages hosted in npm are downloaded more than 5 billion times per week. More and more software is published in npm every day, representing a huge opportunity to share code and solutions, but also to share bugs and faulty software. In this paper, we briefly describe prior work for mutation operators in JavaScript and Node.js, and propose Mutode, an open source tool which leverages the npm package ecosystem to perform mutation testing for JavaScript and Node.js applications. We empirically evaluated Mutode effectiveness by running it on 12 of the top 20 npm modules that have automated test suites.

The purpose of this study is to evaluate and compare the performance of modern deep learning techniques for automatically recognizing and segmenting multiple organ regions on 3D CT images. CT image segmentation is one of the important task in medical image analysis and is still very challenging. Deep learning approaches have demonstrated the capability of scene recognition and semantic segmentation on nature images and have been used to address segmentation problems of medical images. Although several works showed promising results of CT image segmentation by using deep learning approaches, there is no comprehensive evaluation of segmentation performance of the deep learning on segmenting multiple organs on different portions of CT scans. In this paper, we evaluated and compared the segmentation performance of two different deep learning approaches that used 2D- and 3D deep convolutional neural networks (CNN) without- and with a pre-processing step. A conventional approach that presents the state-of-the-art performance of CT image segmentation without deep learning was also used for comparison. A dataset that includes 240 CT images scanned on different portions of human bodies was used for performance evaluation. The maximum number of 17 types of organ regions in each CT scan were segmented automatically and compared to the human annotations by using ratio of intersection over union (IU) as the criterion. The experimental results demonstrated the IUs of the segmentation results had a mean value of 79% and 67% by averaging 17 types of organs that segmented by a 3D- and 2D deep CNN, respectively. All the results of the deep learning approaches showed a better accuracy and robustness than the conventional segmentation method that used probabilistic atlas and graph-cut methods. The effectiveness and the usefulness of deep learning approaches were demonstrated for solving multiple organs segmentation problem on 3D CT images.

SummaryFor mutation testing, the huge cost of running test suites on a large number of mutants has been a serious obstacle. To resolve this problem, we propose a learning‐based mutant reduction techniqueMuTrain.MuTrainuses cost‐considerate linear regression (i.e., CLARS) to learn amutation model, which predicts the mutation score of a test suite based on the mutation testing results of a previous version of a target program. Then,MuTrainapplies the mutation model for subsequent versions to predict mutation scores with significantly fewer mutants. For effective mutant reduction and accurate mutation score prediction,MuTrainuses fine‐grained mutation operators refined from the existing coarse‐grained mutation operators. The experiment results show thatMuTrainreduces the number of mutants effectively (i.e., selecting only 1.6% of mutants). Moreover,MuTrainpredicts mutation score far more accurately than the existing mutant reduction techniques and random mutant selection. We also found thatMuTrainachieves much greater mutant reduction when it uses the fine‐grained mutation operators than the traditional coarse‐grained mutation operators (i.e., 1.6% vs. 14.6%).