FSM Testing Based on Transition Trees and Complete Round Trip Paths Testing Criteria

Model-based testing creates tests from abstract models of the software. These models are often described as graphs, and test requirements are defined as sub paths in the graphs. As a step toward creating concrete tests, complete (test) paths that include the sub paths through the graph are generated. Each test path is then transformed into a test. If we can generate fewer and shorter test paths, the cost of testing can be reduced. The minimum cost test paths problem is finding the test paths that satisfy all test requirements with the minimum cost. This paper presents new algorithms to solve the problem, and then presents data from an empirical comparison. The algorithms adapt approximation algorithms for the shortest super string problem. The comparison is with an existing tool that uses a brute force approach to extend each sub path to a complete path. One new algorithm is based on the greedy set-covering algorithm and the other is based on finding a matching over a prefix graph. The comparison was performed on open software and showed that both new solutions generate fewer test paths than the brute force approach. The prefix-graph based solution takes much less time than the other two solutions when the number of test requirements is large.

Finite state machines (FSM) are widely used in testing and verification of discrete event and hybrid systems. In order to guarantee fault coverage, various fault models and formal methods are utilized ranging from classical finite state machines (FSM) to logic circuits. The main problem for deriving an FSM-based test suite is the size of an FSM for real digital circuits, and thus, test engineers prefer to randomly derive a prefix of a test sequence and later on extend it with tests for hard detectable faults of a corresponding logic circuit. Usually at the latter step test sequences detecting hard detectable stuck-at faults are considered. In this paper, we are concerned about single functional faults, namely, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{a},\boldsymbol{b}-\mathbf{faults}$</tex> , of a system of Boolean functions of the combinational part of a sequential circuit such as deleting a variable or a cube in a SoP which represents the ON-set or the OFF-set of a corresponding Boolean function and experimentally evaluate fault coverage of a corresponding complete test suite with respect to fault coverage of tests returned by other methods. The experimental results show high fault coverage of test suites complete with respect to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{a},\boldsymbol{b}-\mathbf{faults}$</tex> and thus, it can be recommended to utilize such a test suite instead of a complete FSM-based test suite or to extend randomly generated test sequences for detecting <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{a},\boldsymbol{b}-\mathbf{faults}$</tex> not killed with the prefix.

We explore the problem of finding a minimal complete test suite for a refusal trace (or failure trace) semantics. Since complete test suites are typically infinite, we consider the setting with a bound ℓ on the length of refusal traces of interest. A test suite T is thus complete if it is failed by all processes that contain a disallowed refusal trace of length at most ℓ.The proposed approach is based on generating a minimal complete set of forbidden refusal traces. Our solution utilises several interesting insights into refusal trace semantics. In particular, we identify a key class of refusals called fundamental refusals which essentially determine the refusal trace semantics, and the associated fundamental equivalence relation. We then propose a small but not necessarily minimal test suite based on our theory, which can be constructed with a simple algorithm. Subsequently, we provide an enumerative method to remove all redundant traces from our complete test suite, which comes in two variants, depending on whether we wish to retain the highly desirable uniform completeness (guarantee of shortest counterexamples).A related problem is the construction of a characteristic formula of a process P, that is, a formula ΦP such that every process which satisfies ΦP refines P. Our test generation algorithm can be used to construct such a formula using a variant of Hennessy-Milner logic with recursion.

A large number of research works have addressed the importance of models in software engineering. However, the adoption of model-based techniques in software organizations is limited since these models are perceived to be expensive and not necessarily cost-effective. Focusing on model-based testing, this paper reports on a series of controlled experiments. It investigates the impact of state machine testing on fault detection in class clusters and its cost when compared with structural testing. Based on previous work showing this is a good compromise in terms of cost and effectiveness, this paper focuses on a specific state-based technique: the round-trip paths coverage criterion. Round-trip paths testing is compared to structural testing, and it is investigated whether they are complementary. Results show that even when a state machine models the behavior of the cluster under test as accurately as possible, no significant difference between the fault detection effectiveness of the two test strategies is observed, while the two test strategies are significantly more effective when combined by augmenting state machine testing with structural testing. A qualitative analysis also investigates the reasons why test techniques do not detect certain faults and how the cost of state machine testing can be brought down.

Research demonstrated that faults seeded mutation using operators can be representative of faults in real systems. In this paper, we study the relationship between the different operators used to insert mutants in the fault domain of the system under test and the effectiveness of different state machine test suites at killing those mutants. We are particularly interested in the effectiveness of two interrelated state machine testing strategies at finding different types of faults. Those are the round-trip paths strategy and the transition tree strategy. Using empirical evaluation, we compare the effectiveness of more than two thousand unique test suites at killing mutants seeded using eight different mutation operators. We perform experiments on four experimental objects and provide qualitative analysis of the results. We conclude that neither of the two studied strategies is more effective than the other at killing a certain type of mutants. However, the structure of the finite state machine and the nature of the system under test affect the type of faults detected by the different testing strategies.

Guest Editorial: Special Section from the 11th International Conference on Quality Software (QSIC 2011)

Testing and incremental conformance testing of timed state machines

We explore the problem of finding a minimal complete test suite for refusal trace (or failure trace) semantics. Our approach is based on generating a minimal complete set of forbidden refusal traces and utilises several interesting insights into the semantics. In particular, we identify a key class of refusals called fundamental refusals which essentially determine the refusal trace semantics, and the associated equivalence relation. We then propose a small but not necessarily minimal test suite, which can be constructed with a simple algorithm. Subsequently, we provide an enumerative method to remove all redundant traces from our complete test suite, which comes in two variants, depending on whether we wish to retain the highly desirable uniform completeness.We also address a related problem from modal logic, namely the construction of a characteristic formula of a given process with respect to refusal trace semantics, using a variant of Hennessy-Milner logic with recursion.

SummaryFor partial, nondeterministic, finite state machines, a new conformance relation called strong reduction is presented. It complements other existing conformance relations in the sense that the new relation is well suited for model‐based testing of systems whose inputs are enabled or disabled, depending on the actual system state. Examples of such systems are graphical user interfaces and systems with interfaces that can be enabled or disabled in a mechanical way. We present a new test generation algorithm producing complete test suites for strong reduction. The suites are executed according to the grey‐box testing paradigm: it is assumed that the state‐dependent sets of enabled inputs can be identified during test execution, while the implementation states remain hidden, as in black‐box testing. We show that this grey‐box information is exploited by the generation algorithm in such a way that the resulting best‐case test suite size is only linear in the state space size of the reference model. Moreover, examples show that this may lead to significant reductions of test suite size in comparison to true black‐box testing for strong reduction.

Regression tests ensure that existing functionality is not impaired by changes to an existing software system. However, executing complete test suites often takes much time. Therefore, a subset of tests has to be found that is sufficient to validate whether the system under test is still valid after it has been changed. This test case selection is especially important if regression tests are executed manually, since manual execution is time intensive and costly. To select manual test cases, many regression testing techniques exist that aim on achieving coverage of changed or even new code. Many of these techniques base on coverage data from prior test runs or logical properties such as annotated pre and post conditions in the source code. However, coverage information becomes outdated if a system is changed extensively. Also annotated logical properties are often not available in industrial software systems. We present an approach for test selection that is based on static analyses of the test suite and the system's source code. We combine trace link recovery using latent semantic indexing with the metric change coverage, which assesses the coverage of source code changes. The proposed approach works automatically without the need to execute tests beforehand or annotate source code. Furthermore, we present a first evaluation of the approach.

In this paper, a novel safety-related variant of complete test suites for finite state machines is introduced. Under certain hypotheses which are similar to the ones used in the well-known W-Method and its improved versions, the new method guarantees to uncover every violation of safety properties from a certain well-defined class, while erroneous behaviour without safety relevance may remain undetected. While the method can be based on any of the known complete strategies for FSM testing, its most effective variant is based on the H-method, and this variant is presented in detail, denoted as the Safety-complete H-Method. It is guaranteed that application of the Safety-complete H-Method always results in less or equally many test cases than when applying the original H-Method. In well-defined situations that can be pre-determined from the reference model, the Safety-complete H-Method leads to a substantial reduction of test cases in comparison to the size of the analogous H test suites. We advocate this new test suite for situations, where exhaustive testing of the complete system is too expensive. In these cases, strong guarantees with respect to fault coverage should only be given for the errors representing safety violations, while it may be considered as acceptable if less critical errors remain undetected.

Calculating the code coverage of a test suite depends on the identification of the executed test paths. Available code coverage tools are limited to a few basic tracing strategies and do not provide this information as proper test paths, precluding its use in further analyses. We present ExecutionFlow, a solution to trace executed test paths in Java code. The tool is based on the use of a debugger, aspect-oriented programming, and a testing framework. Through empirical validation we show that it is possible to obtain the test path from several methods and constructors, including high complexity ones. Video demonstration: https://youtu.be/2W5BjVuc6fw

This paper reports a practical experience of software testing accomplished by the Software Quality Laboratory of Tecnopolis Csata Novus Ortus in cooperation with Datamat S.p.A. The objective of the joint project was twofold: the definition and development of a formal test suite, the functional testing of a transaction processing software in the stock exchange application area . With respect to the first goal we had the need of define a method for building a complete test suite in terms of: specification of test environment, test process, test documentation, criteria for test derivation. The main result of such experimentation was to allow the replication of the whole test suite in a different target site. The VALID toolkit was chosen as the kernel for the test environment. This tool allows the validation of systems utilising a GUI based on the X window environment, against its specifications using a black box testing method (Jason, Ritter [4]). For the test process definition and test documentation we have taken as reference the IEEE standard 1012 (Software verification and VALEDation plans), IEEE standard 829 (Software test documentation). In relation to the second goal we have built a library of test procedures formally described with the VALID test language and reaching the full coverage of the functionalities of the application under test. The experience in applying the method developed during this project together with the technology used, has shown that this approach is valuable in reducing testing time and costs especially during regression testing in the software maintenance phase.

Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions are in order. In this paper we investigate how method-level changes in the base-code can serve as a reliable indicator for identifying which tests need to be rerun. We validate the approach on two cases - PMD and Cruise Control - using mutation testing as a means to compare the selected subset against a retest all approach. Our results show that we are able to reach a sizable reduction of the complete test suite, yet with a comparable number of mutants killed by the reduced test suite.

Extended finite state machines (EFSMs) are widely used when deriving tests for checking the functional requirements for software implementations. However, the fault coverage of EFSMbased tests covering appropriate paths, variables, etc., remains rather obscure. Furthermore, these tests are known be incapable of detecting many functional faults frequently occurring in EFSM-based implementations. In this paper, an approach is proposed for deriving complete tests with the help of a proper Java EFSM implementation. Since the software is based on a template, the faults turn directly into EFSM faults. The method proposed here makes it possible to derive test suites that can detect functional faults. In the first step, the EFSM-based test suite derived by a well-known method is checked for completeness with respect to the faults generated by the μJava tool. Then, each undetected fault is easily mapped into an EFSM mutant. In the next step, some FSM abstraction is used to derive a distinguishing sequence for two finite-state machines (if such a sequence exists), which is added to the current test suite. The test derived in this way is complete with respect to the faults generated by μJava. If the corresponding FSM derived by EFSM modeling is too complex or no such FSM can be derived, the resulting test suite can be incomplete. However, the experiments performed by us clearly show that the original test suite extended by distinguishing sequences can detect many functional faults in software implementations when the given EFSM is used as a specification for the system.