Mutation Testing with Hyperproperties

Mutation testing is the fault-based software testing approach that is widely applicable for assessing the effectiveness of a test suite. The test suite effectiveness is measured through artificial seeding of faults into the programs under test. Mutation testing tools (MTTs) are significant enablers of the conversion of mutation testing from the research perspective into the real life and mostly applicable testing process. Without using the automatic MTT, mutation testing can’t be really connected in this present reality and is unrealistic to be acknowledged by the industry. Authors analyze six open-source JAVA-based MTT (Jester, JavaMut, MuJava, Jumble, Judy, and Javalanche). The results are based on the performance of various JAVA programs and two real-life applications. Benchmark comparison among the MTT is presented in terms of mutants, mutation operator, mutation score, and quality output. On the basis of comparative analysis, the performance of each tool is explained with the protocol for finding the appropriate tool among the six MTTs. The results show that the MuJava performs best compared to the others.KeywordsMutation testingFault-based testingAdequacy criterionMutation testing tools

SOA (Service Oriented Architecture)를 기반으로 하는 소프트웨어는 비즈니스 요구사항에 기민하게 대응할 수 있어 사용이 확대되고 있다. SOA는 느슨한 서비스 결합과 메시지 교환 중심 아키텍처라는 특징 때문에 기존의 테스팅 방법 적용이 제한된다. 본 논문에서는 BPMN (Business Process Model and Notation)을 이용해 테스트 케이스를 생성하는 기법을 제안한다. 먼저 프로세스를 모델링하고, 이를 통해 비즈니스 흐름 그래프(BFG)를 생성한다. BFG의 입출력 차수를 대칭하고, 오일러 경로를 탐색하여 테스트 케이스를 생성한다. BPMN을 이용한 테스트 케이스 생성 기법은 SOA 기반 시스템에 적용 가능하며, 테스트 케이스 수를 감소시킬 수 있다. The SOA (Service Oriented Architecture) based softwares are escalated because of quickly coping with business requirement. SOA can not apply to existing test method because of loosely coupled service and massage exchange architecture. In this paper, we suggest a method of test case generation using BPMN (Business Process Model and Notation). First we model processes, then make Business Flow Graph (BFG). After searching the euler path through symmetrized BFG about input and output degrees, we generate test cases. A method of test case generation using BPMN can apply at SOA-based system, and reduce the number of test cases.

Automatic train protection (ATP) is an important equipment with a safety requirement of SIL 4 for Chinese Train Control System, whose safety is usually guaranteed by testing. Test cases generation method is an important basis and research focus of ATP. Taking Petri Net as the modelling tool, a test cases generation method of ATP is proposed. First, according to the technical specification, performing a fault mode analysis, ATP Petri Net testing model is established. Then, the test cases are generated automatically by searching the reachability tree of the testing model. Finally, taking the SOM function of ATP as an example, testing model is established and test cases are generated. By comparing the test cases generated by this method and the existing test cases, the correctness and effectiveness of this method is verified.

Context: Consumer tests which assess safety features of modern vehicles have a tradition in Europe. Recently, such test protocols have been substantially extended to also cover active safety systems like Volkswagen's Front Assist. Objective: Simulations for passive safety systems are already a widely adopted approach during vehicle development and internal assessments. As active safety systems are becoming an increasingly important element in a vehicle's safety concept and a differentiating feature, a systematic validation and assessment of such systems is necessary to successfully pass consumer tests and complimentarily identified, relevant traffic scenarios. Method: With this work, we extend our previous conference publication about EuroNCAP CCRs tests by additionally investigating US NCAP scenarios for an AEB system. Therefore, we systematically modeled the allowed variations with a graph where the paths represent concrete test scenarios. These paths are used in a virtual test environment to assess the AEB system. In our previous publication, we illustrated our method of test case generation and simulating consumer test scenarios by showing results of 27 specific test cases. In this work, we focused on integrating a test automatization routine as well as evaluating a set of test cases with a factor of 100 compared to our previous paper. Results: We demonstrate the approach for both EuroNCAP's and US NCAP's CCRs scenarios with a total quantity of more than 2,700 test cases including re-runs to systematically evaluate an AEB algorithm. Our results unveiled varying action points in time for the same initial values for a given consumer test scenario while applying different allowed variations. Conclusion: We foresee the importance of complementary virtual testing for real-world tests on proving grounds especially during the design phase. Our study shows that already small variations that yet accord with the test procedure specification influence the behavior of an active safety system and need to be investigated during development and vehicle testing.

The amount of test cases in real life software testing is huge. There are many studies, such as classification tree and class vector, on generating all the test cases at a time. The practicability of these studies is low as the testers are difficult to manage that huge amount of test cases. When testing a subset of functions of a system or testing the part of systems that have been modified, it would be more practical and convenience for testers if only all the concerned test cases are generated on demand. In order to generate test cases on demand, first we need a modeling language which is capable to model the input. The modeling language should be elegant and convenient such that testers can focus in the testing. This modeling language should support generating test cases according to some criteria as well. Furthermore, the model language should be implementable for computer aided test cases generation. We found hypergraph is a good candidate as the modeling language. In this paper, we report our study on using hypergraph as the modeling language for generating test cases on demand.

Mutation testing is a technique in which faults (mutants) are injected into a program or application to assess its test suite effectiveness. It works by inserting mutants and running the application’s test suite to identify if the mutants are detected (killed) or not (survived) by the tests. Although computationally expensive, it has proven to be an effective method to assess application test suites. Several mutation testing frameworks and tools have been built for the various programing languages, however, very few tools have been built for the JavaScript language, more specifically, there is a lack of mutation testing tools for the Node.js runtime and npm based applications. The npm Registry is a public collection of modules of open-source code for Node.js, front-end web applications, mobile applications, robots, routers, and countless other needs of the JavaScript community. The over 700,000 packages hosted in npm are downloaded more than 5 billion times per week. More and more software is published in npm every day, representing a huge opportunity to share code and solutions, but also to share bugs and faulty software. In this paper, we briefly describe prior work for mutation operators in JavaScript and Node.js, and propose Mutode, an open source tool which leverages the npm package ecosystem to perform mutation testing for JavaScript and Node.js applications. We empirically evaluated Mutode effectiveness by running it on 12 of the top 20 npm modules that have automated test suites.

Extended finite state machine (EFSM) is among the most popular models for model-based testing. However, automated test case generation on EFSM models is still a challenging task since an EFSM model may contains infeasible paths. This paper proposed a novel approach (ATGEM) to generate test case and construct oracle information from EFSM automatically. To address the infeasible problem, a metric based on data flow analysis is presented to predict the infeasible probability so as to bypass the infeasible paths as far as possible and improve the test case generation efficiency. Afterwards, an executable EFSM model is developed to obtain runtime feedback information as a fitness function in order to generate test data and construct oracle information automatically. This approach, which can generate various types data and has a wide range of applications, combines static analysis and dynamic analysis aims to find a preferable feasible path subset to generate test cases and meet adequacy coverage criteria. The experimental results on several EFSM models show that test case generation method and path feasibility metric have good effectiveness. Utilizing path feasibility metric can speed up the process of test case generation greatly, and ATGEM is more efficient than existing method.

In order to solve the current problems( expensive testing cost and low efficiency of test case generation) in mutation test for Simulink models,a mutation testing process and an optimized set of mutation operators were proposed for Simulink models based on program mutation according to the current mutation operators for the Simulink models. Experiments show that this set of mutation operators can effectively reduce the generation number of mutation models without prejudice to the mutation score of testing case set,thus it will effectively save the testing cost. Then a search-based test case generation method for Simulink models mutation testing was described. The test case generation problem was transformed into the objective function minimization problem,and the test cases which can kill the mutation models were ultimately obtained through the optimization of objective function by algorithm of simulated annealing. Finally,the application of a typical case for the method verified the correctness and effectiveness.

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.Â

A specification-based testing method aimed primarily at detecting errors of distributed systems is presented. This specification technique emphasizes the communication behaviors and the synchronization properties of distributed processes. The functions of behavior specification for the testing method serve two purposes. First, it provides the communication-oriented high-level view of the system, which dictates the method of test case generation. Five test criteria peculiar to distributed software are considered. Test data can be derived from the source code based on the symbolic execution technique applied to each test case. Second, the specification serves as a correct reference model, which helps analyze the communication events of the system execution. Currently, this testing method, as an important support to the INTEGRAL verification and validation system, has been largely realized in a SUN workstation environment.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This paper presents a theory of testing that integrates into Hoare and He’s Unifying Theory of Programming (UTP). We give test cases a denotational semantics by viewing them as specification predicates. This reformulation of test cases allows for relating test cases via refinement to specifications and programs. Having such a refinement order that integrates test cases, we develop a testing theory for fault-based testing. Fault-based testing uses test data designed to demonstrate the absence of a set of pre-specified faults. A well-known fault-based technique is mutation testing. In mutation testing, first, faults are injected into a program by altering (mutating) its source code. Then, test cases that can detect these errors are designed. The assumption is that other faults will be caught, too. In this paper, we apply the mutation technique to both, specifications and programs. Using our theory of testing, two new test case generation laws for detecting injected (anticipated) faults are presented: one is based on the semantic level of UTP design predicates, the other on the algebraic properties of a small programming language.

In this article, we propose a novel method for black-box test case generation for business process execution language (BPEL) processes. We also propose a method to compute hit ratio percentage metric. We also compute the total time taken for test case generation and the speed of test case generation. We design the service-oriented architecture (SOA)-based applications using OpenESB tool. We have developed a code converter to generate Tcases compatible .xml code because the OpenESB generated .xml code is incompatible with Tcases input framework. After that, we compute the hit ratio percentage with the help of Hit Ratio Calculator. We have experimented with 12 SOA-based applications, on an average, we achieve 62.78% of hit ratio with an average time of 873.08 ms and speed of 32.41 ≈ 32 test cases per second.

In this article, we propose a novel method for black-box test case generation for business process execution language (BPEL) processes. We also propose a method to compute hit ratio percentage metric. We also compute the total time taken for test case generation and the speed of test case generation. We design the service-oriented architecture (SOA)-based applications using OpenESB tool. We have developed a code converter to generate Tcases compatible .xml code because the OpenESB generated .xml code is incompatible with Tcases input framework. After that, we compute the hit ratio percentage with the help of Hit Ratio Calculator. We have experimented with 12 SOA-based applications, on an average, we achieve 62.78% of hit ratio with an average time of 873.08 ms and speed of 32.41 ≈ 32 test cases per second.

"Within the automotive product development cycle virtual and heterogeneous testing is becoming increasingly established through component, module and vehicle-level simulation. Though a number of standards in this field have been established, models are still mostly created in a fragmented manner: using domain-specific tools to create, manage and execute simulations without standardization of the content of the functional interfaces (FMI does only standardize the format) and limited scalability. This fragmentation leads to a lot of redundant effort as models of the same component or system are re-created several times. HIFI-ELEMENTS project addressed this fragmentation through two main mechanisms: Firstly, developing, validating and publishing a recommendation for standardization of model interfaces for common e-drive components (e-machine, inverter, battery, DC/DC converter, thermal management) and implementation of compliant versions of existing models. Secondly, implementing a seamless workflow linking extended versions of existing tools with effort-saving automated methods for model parameterization and test case generation. This seamless integration will substantially increase the number of integrations and test cases that can be early validated through simulation, leading to optimized efficiency designs and development effort reduction. The standardization also guarantees scalability among fidelity levels, from concept design to XiL through detailed modelling. In this paper we present the results of the Use Case C: Component co-optimization. The purpose of use cases is the demonstration of the advantages of the standardized models and workflow industry relevant scenarios. The work content performed in the use case is very extensive and multidisciplinary. In the first step, the high fidelity models from the expert components developers were validated independently with automated testing tools and later integrated to create a complete vehicle architecture integration. The standardization permitted to seamlessly test several component variants developed within the project for the same architecture, including tens of motor models with different technologies, inverters and high voltage converters with different IGBT technology and various battery packs. This possibility was exploited through co-optimization with multi objective Genetic Algorithm, permitting to select the optimal component combination, powertrain architecture (with and without high voltage DCDC converter) and components parametrization considering the trade-off of consumption and performances. The optimized and baseline variants were used to demonstrate the scalability of the models to different simulation objectives. The model was co-simulated with a traffic simulation environment in order to evaluate the impact of eco-driving recommendation algorithms in a realistic driving situation. The optimized solution was also validated against a wide database of driving conditions including real driving cycles, performance and vehicle dynamics. Finally, the integrated models were seamlessly transferred to real time simulation platform for Model-in-the-Loop testing with a simulated 3D environment aimed at ADAS testing. Real-time capability demonstrates that next steps such as Driver-in-the-Loop and Hardware-in-the loop can be achieved smoothly. The extensive simulation activities performed in this use case demonstrate the benefits of the standard in models exchangeability and effort reduction in model based development. This project received funding from the European Union’s (EU) Horizon 2020 Research and innovation program under grant agreement N 769935."

Fault in a software program can be detected by mutation testing. However, mutation testing is an expensive process in a software testing domain. In this paper, we have introduced a method based on genetic algorithm and mutation analysis for unit testing process. Software industry produces high quality software in which software testing has an important role. First, we make a program/software and intent some mutant in this program/software, find most critical path and optimise test cases using genetic algorithm for the unit testing. Initially generated test cases are refined using genetic algorithm. We use a mutant function for measuring the adequacy of the test case set. The given mutant function is used to calculate a mutant score. We have achieved 100% path coverage and boundary coverage using mutation testing. The objective is to produce a set of good test cases for killing one or more undesired mutants and produces different mutant from original software/program. Unlike simple algorithms, genetic algorithms provide suitability for reducing the data generation at a comparable cost. An optimised test case has been generated by proposed approach for cost reduction and revealing or killing undesired test cases.