A Case Study in Matching Test and Proof Coverage

SummaryIn order to claim conformance with a Java Specification Request, a Java implementation has to pass all tests in an associated Technology Compatibility Kit. This paper presents a model‐based development of a Technology Compatibility Kit test suite and a test execution tool for the draft safety‐critical Java profile specification. The Java Modeling Language is used to model conformance constraints for the profile. Java Modeling Language annotations define contracts for classes and interfaces. The annotations are translated by a tool into runtime assertion checks. Hereby, the design and elaboration of the concrete test cases are simplified, because the expected results are derived from contracts and thus do not need to be provided explicitly. Bottom‐up testing is applied for testing methods of the safety‐critical Java classes, whereas top‐down testing is applied for testing global properties, such as protocols, memory management, and real‐time properties, including scheduling. The tests are executed using a simplified version of JUnit, which makes the test suite executable on resource‐constrained platforms. Copyright © 2017 John Wiley & Sons, Ltd.

Software Testing is an important aspect of the real time software development process. Software testing always assures the quality of software product. As associated with software testing, there are few very important issues where there is a need to pay attention on it in the process of software development test. These issues are generation of effective test case and test suite as well as optimization of test case and suite while doing testing of software product. The important issue is that testing time of the test case and test suite. It is very much important that after development of software product effective testing should be performed. So to overcome these issues of optimization, we have proposed new approach for test suite optimization using genetic algorithm (GA). Genetic algorithm is evolutionary in nature so it is often used for optimization of problem by researcher. In this paper, our aim is to study various selections methods like tournament selection, rank selection and roulette wheel selection and then we apply this genetic algorithm (GA) on various programs which will generate optimized test suite with parameters like fitness value of test case, test suite and take minimum amount of time for execution after certain preset generation. In this paper our main objectives as per the experimental investigation, we show that tournament selection works very fine as compared to other methods with respect fitness selection of test case and test suites, testing time of test case and test suites as well as number of requirements.

The Object Constraint Language (OCL) is a formal specification language that could be used for constraining the modelling elements that occur in UML diagrams. The Java Modeling Language (JML) is a behavioural interface specification language designed for specifying Java classes and interfaces. This paper investigates the mapping of OCL constraints to JML constraints together with its properties. In particular the paper shows how those constructs such as iterators that are not directly supported by JML are transformed to an equivalent form that can be mapped directly to JML. The paper proposes a special library for OCL types implemented in Java that facilitates the mapping. Extracts from a case study was used to illustrate the mapping

Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system. This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Furthermore, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes. The method compares test suites in a platform independent manner. The test suites, which are compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. We exercise our method over three case studies to demonstrate its viability and effectiveness. The first case study illustrates the application of the method and evaluates its effectiveness using a mutation analysis. The second case study evaluates its effectiveness using mutation and coverage analyses. The final case study evaluates it on a real case study, which is Lucene search engine.

Numerous combinatorial testing tools are available for generating test cases. However, many of them are never used in practice. One of the reasons is the lack of empirical studies that involve human subjects applying testing techniques. This paper aims to investigate the applicability of a combinatorial testing tool in the company SOFTEAM. A case study is designed and conducted within the development team responsible for a new product. The participants consist of 3 practitioners from the company. The applicability of the tool has been examined in terms of efficiency, effectiveness and learning effort.

This editorial describes a special issue of papers from the 2014 workshop on Java Technologies for Real-Time and Embedded Systems.1 There are four papers in this special issue. The first paper 2 presents HVMTP, a time predictable and portable Java virtual machine (JVM) implementation with applications in resource-constrained, hard real-time embedded systems, which implements all levels of the safety-critical Java (SCJ) specification. Time predictability is achieved by a combination of time-predictable algorithms, exploiting the programming model of the SCJ profile and harnessing the static knowledge of the hosted SCJ system. This paper presents HVMTP in terms of its design and capabilities and demonstrates how a complete timing model of the JVM represented as a network of timed automata can be obtained using the tool TETASARTSJVM. The timing model readily integrates with the rest of the TETASARTS toolset for temporal verification of SCJ systems. A complete timing scheme in terms of safe worst-case execution times and best-case execution times of the Java bytecodes is derived from the model. The paper takes a first look at how to support the new Java 8 language feature of Lambda expressions in an SCJ context – looking in particular at how the bytecode, used for invoking Java 8 closures, can be implemented in a time-predictable way and integrated into HVMTP. The second paper 3 presents the motivation for and outcomes of an engineering research project on certifiable Java for embedded systems. The project supports the upcoming SCJ standard for safety-critical Java, which defines a subset of Java and libraries aiming for the development of high criticality systems. The outcome of this project includes prototype safety-critical Java implementations, a time-predictable Java processor, analysis tools for memory safety, and example applications to explore the usability of safety-critical Java for this application area. The text summarizes developments and key contributions and concludes with the lessons learned. The third paper 4 presents a model-based development of a Technology Compatibility Kit test suite and a test execution tool for the draft safety-critical Java profile specification. Passing all tests in an associated Technology Compatibility Kit is needed in order for a Java implementation to claim conformance with a Java Specification Request. The Java Modeling Language is used to model conformance constraints for the profile. Java Modeling Language annotations define contracts for classes and interfaces. The annotations are translated by a tool into runtime assertion checks. Hereby, the design and elaboration of the concrete test cases are simplified, because the expected results are derived from contracts and thus do not need to be provided explicitly. Bottom-up testing is applied for testing methods of the safety-critical Java classes, whereas top-down testing is applied for testing global properties, such as protocols, memory management, and real-time properties, including scheduling. The tests are executed using a simplified version of JUnit, which makes the test suite executable on resource-constrained platforms. The last paper 5 shows the weak predictability of Android's internal intra-process and inter-process communication based on Intent messaging and presents a concept to improve its soft real-time capability. The proposed approach introduces a priority-based broadcast handling using explicit priority values, instead of the original first in – first out processing. Furthermore, the size of responsible critical sections is reduced in order to improve the preemptibility and to assure predictable processing times for applications with real-time requirements. Their evaluation highlights the improvements in comparison with the original Android implementation without any loss of the system performance. Additionally, the compatibility with already existing components and applications is preserved. We thank Wolfgang Puffitsch for his work on this special issue.

The CTH multiphysics hydrocode, which is used for a wide range of important calculations, has undertaken in recent years to overhaul its software quality and testing processes. A key part of this effort entailed building a new, robust V&V test suite made up of traditional hydrocode verification problems, such as those listed in the ASC Tri-Lab Test Suite and the Enhanced Tri-Lab Test Suite, as well as validation problems for some of CTHs most frequently used equations of state, materials models, and other key capabilities. Substantial progress towards this goal was made in FY19. In FY20, this test suite has been expanded to include verification and validation tests of the Sesame and JWL equation of state models as well as the Mader verification problem from the Tri-Lab Test Suite and the Blake verification problem - a linear elastic analog to the Hunter problem from the Enhanced Tri-Lab Test Suite. This report documents CTH performance on the new test suite problems. Verification test results are compared to analytic solutions and, for most tests, convergence results are presented. Validation test results are compared to experimental data and mesh refinement studies are included. CTH performs well overall on the new test problems. Convergence rates for the Blake and Mader problems are comparable to those for similar ASC codes. The JWL and Sesame verification tests show good agreement with analytic solutions. Likewise, CTH simulation results show good agreement with experimental validation data for the Sesame and JWL equations of state for the materials tested. Future V&V work will focus on adding tests for other key capabilities like fracture and high explosive models.

Formal methods for smart cards: an experience report

Software testing techniques have advanced significantly over several decades; however, most of current techniques still test a target program as it is, and fail to utilize valuable information of diverse test executions on many variants of the original program in test generation. This paper proposes a new direction for software testing - Invasive Software Testing (IST). IST first generates a set of target program variants m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> from an original target program p by applying mutation operations <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, μ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> . Second, given a test suite T, IST executes m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> with T and records the test runs which increase test coverage compared to p with T. Based on the recorded information, IST generates guideposts for automated test generation on p toward high test coverage. Finally, IST generates test inputs on p with the guideposts and achieves higher test coverage. We developed DEMINER which implements IST for C programs through software mutation and concolic testing. Further, we showed the effectiveness of DEMINER on three real-world target programs Busybox-ls, Busybox-printf, and GNU-find. The experiment results show that the amount of the improved branch coverage by DEMINER is 24.7% relatively larger than those of the conventional concolic testing techniques on average.

To enable scalability and address the needs of real-world software, deductive verification relies on modularization of the target program and decomposition of its requirement specification. In this paper, we present an approach that, given a Java program and a partial requirement specification written using the Java Modeling Language, constructs a semantic slice. In the slice, the parts of the program irrelevant w.r.t. the partial requirements are replaced by an abstraction. The core idea of our approach is to use bounded program verification techniques to guide the construction of these slices. Our approach not only lessens the burden of writing auxiliary specifications (such as loop invariants) but also reduces the number of proof steps needed for verification.

Extended finite state machines (EFSMs) are widely used when deriving tests for checking the functional requirements for software implementations. However, the fault coverage of EFSMbased tests covering appropriate paths, variables, etc., remains rather obscure. Furthermore, these tests are known be incapable of detecting many functional faults frequently occurring in EFSM-based implementations. In this paper, an approach is proposed for deriving complete tests with the help of a proper Java EFSM implementation. Since the software is based on a template, the faults turn directly into EFSM faults. The method proposed here makes it possible to derive test suites that can detect functional faults. In the first step, the EFSM-based test suite derived by a well-known method is checked for completeness with respect to the faults generated by the μJava tool. Then, each undetected fault is easily mapped into an EFSM mutant. In the next step, some FSM abstraction is used to derive a distinguishing sequence for two finite-state machines (if such a sequence exists), which is added to the current test suite. The test derived in this way is complete with respect to the faults generated by μJava. If the corresponding FSM derived by EFSM modeling is too complex or no such FSM can be derived, the resulting test suite can be incomplete. However, the experiments performed by us clearly show that the original test suite extended by distinguishing sequences can detect many functional faults in software implementations when the given EFSM is used as a specification for the system.

Code refactoring is a technique that improves the existing code in order to make software easier to understand and more extensible without changing the external behavior. Software design patterns, programming language independent reusable solutions to comment problems, are well-known in Java communities. On one hand, the refactoring using design patterns brings many benefits such as cost saving, flexibilities, and maintainability. On the other hand, it potentially causes bugs or changes execution behavior of Java programs. This paper proposes a new approach to checking behavior preservation properties of Java programs after applying design patterns. We present new definitions to compute pre/post conditions of program behavior. In the next step, the paper makes use of Java Modeling Language (JML) to represent and check if the refactored program neglects to preserve the external behavior. A motivating example of Adaptive Road Traffic Control (ARTC) is given to illustrate the approach in detail.

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

Regression testing is an essential quality test technique during the maintenance phase of the software. It is executed to ensure the validity of the software after any modification. As software evolves, the test suite expands and may become too large to be executed entirely within a limited testing budget and/or time. So, to reduce the cost of regression testing, it is mandatory to reduce the size of the test suite by discarding the redundant test cases and selecting the most representative ones that do not compromise the effectiveness of the test suite in terms of some predefined criteria such as its fault-detection capability. This problem is known as test suite reduction (TSR); and it is known to be as nondeterministic polynomial-time complete (NP-complete) problem. This paper formulated the TSR problem as a multi-objective optimization problem; and adapted the heuristic binary bat algorithm (BBA) to resolve it. The BBA algorithm was adapted in order to enhance its exploration capabilities during the search for Pareto-optimal solutions. The effectiveness of the proposed multi-objective adapted binary bat algorithm (MO-ABBA) was evaluated using 8 test suites of different sizes, in addition to twelve benchmark functions. Experimental results showed that, for the same fault discovery rate, the MO-ABBA is capable of reducing the test suite size more than each of the multi-objective original binary bat (MO-BBA) and the multi-objective binary particle swarm optimization (MO-BPSO) algorithms. Moreover, MO-ABBA converges to the best solutions faster than each of the MO-BBA and the MO-BPSO.

Regression testing is one of the essential activities during the maintenance phase of software projects. It is executed to ensure the validity of an altered software. However, as the software evolves, regression testing becomes prohibitively expensive. In order to reduce the cost of regression testing, it is mandatory to reduce the size of the test suite by selecting the most representative test cases that do not compromise the effectiveness of the regression testing in terms of fault-detection capability. This problem is known as test suite reduction (TSR) problem, and it is known to be an NP-complete. The paper proposes a multi-objective adapted binary bat algorithm (ABBA) to solve the TSR problem. The original binary bat (OBBA) algorithm was adapted to enhance its exploration capabilities during the search for a Pareto-optimal surface. The effectiveness of the ABBA was evaluated using six Java programs with different sizes. Experimental results showed that for the same fault discovery rate, the ABBA is capable of reducing the test suite size more than the OBBA and the BPSO.