Comparison and Validation of Mutation Testing Tools Based on Java Language

Mutation testing is a technique in which faults (mutants) are injected into a program or application to assess its test suite effectiveness. It works by inserting mutants and running the application’s test suite to identify if the mutants are detected (killed) or not (survived) by the tests. Although computationally expensive, it has proven to be an effective method to assess application test suites. Several mutation testing frameworks and tools have been built for the various programing languages, however, very few tools have been built for the JavaScript language, more specifically, there is a lack of mutation testing tools for the Node.js runtime and npm based applications. The npm Registry is a public collection of modules of open-source code for Node.js, front-end web applications, mobile applications, robots, routers, and countless other needs of the JavaScript community. The over 700,000 packages hosted in npm are downloaded more than 5 billion times per week. More and more software is published in npm every day, representing a huge opportunity to share code and solutions, but also to share bugs and faulty software. In this paper, we briefly describe prior work for mutation operators in JavaScript and Node.js, and propose Mutode, an open source tool which leverages the npm package ecosystem to perform mutation testing for JavaScript and Node.js applications. We empirically evaluated Mutode effectiveness by running it on 12 of the top 20 npm modules that have automated test suites.

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.Â

Mutation testing has been used to evaluate the quality of the test set and provides the confidence in testing activity. Generation as well as running of mutants needs practice. A huge amount of mutants are generated and therefore, it is a very costly activity in terms of time and effort. Automatic mutant generation and execution is essential to support testing. Automation achieves more attention, saves time and effort as developers and testers use automated tools. In the recent years, researchers have found that practitioners have diminutive knowledge about tools and their effectiveness. This paper compares five well-known publicly accessible mutation testing tools: ‘MuClipse’, ‘Judy’, ‘Jumble’, ‘Jester’ and PIT. This comparison uses a collection of Java classes taken from various easily accessible sources. Tests were designed and generated with the help of test generation techniques. The performance of each mutation tool was analyzed and was compared based on their mutation operators, mutation score and execution time.

SummaryFor mutation testing, the huge cost of running test suites on a large number of mutants has been a serious obstacle. To resolve this problem, we propose a learning‐based mutant reduction techniqueMuTrain.MuTrainuses cost‐considerate linear regression (i.e., CLARS) to learn amutation model, which predicts the mutation score of a test suite based on the mutation testing results of a previous version of a target program. Then,MuTrainapplies the mutation model for subsequent versions to predict mutation scores with significantly fewer mutants. For effective mutant reduction and accurate mutation score prediction,MuTrainuses fine‐grained mutation operators refined from the existing coarse‐grained mutation operators. The experiment results show thatMuTrainreduces the number of mutants effectively (i.e., selecting only 1.6% of mutants). Moreover,MuTrainpredicts mutation score far more accurately than the existing mutant reduction techniques and random mutant selection. We also found thatMuTrainachieves much greater mutant reduction when it uses the fine‐grained mutation operators than the traditional coarse‐grained mutation operators (i.e., 1.6% vs. 14.6%).

Semantic mutation testing

A test suite plays a key role in software testing. Mutation testing is a powerful approach to measure the fault-detection ability of a test suite. The mutation testing process requires a large number of mutants to be generated and executed. Hence, mutation testing is also computationally expensive. To solve this problem, predictive mutation testing builds a classification model to predict the test result of each mutant. However, the existing predictive mutation testing methods only can be used to estimate the overall mutation scores of object-oriented programs. To overcome the shortcomings of the existing methods, we propose a new method to directly predict the mutation score for each statement in process-oriented programs. Compared with the existing predictive mutation testing methods, our method uses more dynamic program execution features, which more adequately reflect dynamic dependency relationships among the statements and more accurately reflects information propagation during the execution of test cases. By comparing the prediction effects of logistic regression, artificial neural network, random forest, support vector machine, and symbolic regression, we finally decide to use a single hidden layer feedforward neural network as the predictive model to predict the statement mutation scores. In our two experiments, the mean absolute errors between the statement mutation scores predicted by the neural network and the real statement mutation scores both approximately reach 0.12.

Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does not detect some (non-equivalent) mutants, T can be improved by adding test cases that detect those mutants. Mutants are created by applying mutation operators. Mutation operators are important because they define the characteristics of the system that are tested as well as the characteristics that are improved in the test suite. While mutation operators are well defined for a number of programming paradigms such as sequential or multi-threaded, to the best of our knowledge, mutation operators have not been defined for the actor programming model. In this paper, we define and classify mutation operators that can be used for mutation testing of actor programs.

We present a new method for model-based mutation-driven test case generation. Mutants are generated by making small syntactical modifications to the model or source code of the system under test. A test case kills a mutant if the behavior of the mutant deviates from the original system when running the test. In this work, we use hyperproperties—which allow to express relations between multiple executions—to formalize different notions of killing for both deterministic as well as non-deterministic models. The resulting hyperproperties are universal in the sense that they apply to arbitrary reactive models and mutants. Moreover, an off-the-shelf model checking tool for hyperproperties can be used to generate test cases. We evaluate our approach on a number of models expressed in two different modeling languages by generating tests using a state-of-the-art mutation testing tool.

A probabilistic framework for mutation testing in deep neural networks

Android applications have experienced an outstanding growth in recent years with a projected figure of 378 billion in the number of application downloads by 2020. As a result, verification and validation of Android apps and their graphical user interface through testing is crucial for quality assurance. One way to assess quality assurance is through the use of mutation testing. Mutation testing modifies the code of the System Under Test in small ways in order to produce faulty versions of the program called mutants and compare them to the original version. However, the cost of mutation testing can be expensive and time consuming. We propose a graphical user-friendly Android mutation tool called Edroid whose primary purpose is to mutate Android's main components such as activities, services, content providers and broadcast receivers using the source code of XML files. We introduce fourteen Android mutation operators, where ten of them are new, that are derived from operators used in different fields such as mutation testing for Graphical User Interface and XML schema. We evaluate Edroid on five real world Android applications using testing methodologies such as random testing and black box testing. Results from these experiments demonstrate that Edroid facilitates the generation of mutants. In addition and equally important, the set of proposed mutation operators are effective in revealing errors that improve the quality of a given test suite.

Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite’s fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies’ results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.

This paper describes an effort that has been made to make muJava more accessible. muJava is a Mutation testing tool, built on Java platform for Java programs. In order to make muJava more accessible, a Java stand alone application has been made, to act as a main menu for muJava tool. muJava Menu requires muJava in order to work properly and developed menu represents a third party nonprofit add-on for muJava that should simplify the usage of muJava Mutation system. muJava is not embeded into muJava Menu. Developed application is platform independent, and works on any operating system. Prime goal was to make an application that would lessen configuration efforts regarding muJava tool from the aspects of muJava path location, JDK location, and constant CLASSPATH changes that are needed to generate, and then run muJava mutants. Also, for Windows users, who's operating system version is greater or equal to 5 (Microsoft Windows XP and above), a custom BATCH menu script has been made that is much faster then the previously mentioned muJava wrapper menu.

Jeff Magee and Jeff Kramer John Wiley and Sons, New York, NY, 1999, 374 pp. ISBN 0471987107, $64.99 The book is an introductory text to the ideas, concepts, and topics of concurrency. The text focuses on understanding concepts and techniques, and problems of concurrency, not on the implementation or language. The author's overall goal is that by a combination of learning and doing using Java should make the process of acquiring the skills relating to concurrency interesting, enjoyable, and challenging. The Java language is thus used only for illustration and programming experience of the reader. The book is intended for a computer science student or software developer. A background in programming is expected, as well as some familiarity with the essential concepts of object-oriented programming. Knowledge of the Java programming language and operating systems concepts is beneficial but not required. After reading the text, a reader can expect to have a broad understanding of the concepts of concurrency, the problems that arise, methods to emphasis desirable properties in a concurrent system, and ways to avoid the undesirable ones. The text provides a systematic treatment of concepts, as means to rigorously specify and model concurrent systems, with Java examples to animate and illustrate the concepts discussed. The book is organized into twelve chapters, with three appendices. The first eight chapters provide a concise and comprehensive foundation of concurrency, with the remaining four chapters focusing on more advanced concepts. The latter four chapters are somewhat supplemental and can be read at the discretion of the reader. The content of the text is self-contained, with no reliance upon external references. The authors provide a concluding set of notes and further topics which reference external works, but these are not used in the chapter contents. The introduction provides the foundation for the book, and an idea of what will follow in the rest of the text. The basics of modeling and Java are covered. Authors follow with the concepts of processes, threads and with the idea of concurrency. The emphasis is on using Java, and on the notation used to define a concurrent process or thread. The discussion continues covering interleaving and asynchronous concurrency. Various details about composition of parallel processes and finite state processing modeling are presented. The programming of concurrent execution using Java is then examined via the concept of multiple threads. Here, sharing objects among Java threads is also discussed. The concept of conditional synchronization is covered next, and is used to construct a monitor. The methods of the base object in Java is used, and a waiting queue associated with a Java object is discussed and implemented. This discussion is followed by introduction of deadlock. Authors show the conditions necessary for a deadlock to occur and present an analysis of a concurrent system establishing that such a system is deadlock-free. The classic Dining Philosopher's problem is used to illustrate the distinction between finding deadlocks in models and in programs. The presentation continues with the safety property (that asserts that nothing bad happens), and the liveliness property (that asserts that nothing good happens). Here, authors use the material on model-based design to consolidate many of the concepts previously discussed. No one modeling method or design methodology is emphasized. Then creation and elimination of threads in a running program is covered next. The dynamic starting and stopping of Java threads is presented. The discussion moves then to messaging passing. Asynchronous and synchronous messaging is presented with Java programming examples. The concept of a rendezvous, allowing for interaction in a messaging system, is also presented. Based on the concepts developed so far filter-pipeline, supervisor-worker, and announcer-listener concurrent architectures are examined. Java programs demonstrating how the elements of each model interact at runtime are developed. The presentation completes with modeling and implementing of timed systems. Time is signaled by the passage of successive time ticks. The authors then elaborate on a system that is event-based rather than thread-based. Java programs are used to illustrate in implementation the concepts discussed. The three appendices provide a reference, specification, and listing of semantics for the finite-state modeling algebra used by the authors throughout the book. The text is ideal for teaching in a classroom environment, or for a motivated reader to use for self-paced learning. Its organization follows a logical train of discussion and thought, with each chapter building successively on the other, with the more difficult concepts and topics withheld until later. This gradual approach makes the material flow smoothly and cohesively from the beginning to the end. The book is intended as a textbook, so does organize the material and discussion as one, but does not become too dry or bland to stifle the reader's interest in the material. The examples are inter-mixed with the discussion to highlight and emphasize essential points the authors make in each chapter. The text illustrates concepts well under discussion, and does not become a long-winded recital of theory. The topics examined are fundamental to concurrency, and are well illustrated and discussed, but no one topic is examined extensively. The code examples are well written concerning the programming style, indentation and comments. A CD is provided, containing the codes from the book. In addition supplementary material consisting of presentation slides and an errata is provided on the author's WWW site, as well as on the CD. The book is up to date both on the topics of concurrency, and with the Java applets, code, and programs used to illustrate them. Later, as the Java language evolves, the examples may need to be updated, but for the time being the text well represents that state of the Java language. William F. Gilreath, Micron Technology, Inc., Boise Idaho

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static analyzer. Since static analyzers, in general, report a substantial number of false positive warnings, the intention of this study is to define a prioritization approach of static warnings based on their correspondence with mutations. On the other hand, knowing that Mutation Test has a high application cost, another possibility is to try to identify mutations of some specific mutation operators, which an automatic static analyzer is not adequate to detect. Therefore, this information can be used to prioritize the order of incrementally applying mutation operators considering, firstly, those with no correspondence with static warnings. In both cases, contributing to the establishment of incremental strategies on using automatic static analysis or mutation testing or even a combination of them. We used mutation operators as a fault model to evaluate the direct correspondence between mutations and static warnings. The main advantage of using mutation operators is that they generate a large number of programs containing faults of different types, which can be used to decide the ones most probable to be detected by static analyzers. We provide evidences on the correspondence between mutations and some types of static warnings. The results obtained for a set of 19 open-source programs indicate that: 1) static warnings may be prioritized based on their correspondence level with mutations; 2) specific set of mutation operators and their mutations may be prioritized based on their correspondence level with warnings. It is possible to provide an incremental testing strategy aiming at reducing the cost of both static analysis and mutation testing using the correspondence information between these activities/artifacts.

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.