An Analysis of FSM Mutation Operators

Guest Editorial: Special Section from the 11th International Conference on Quality Software (QSIC 2011)

Minimizing Characterizing sets

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

A binary-tree-based finite state machine (BT-FSM) is a state machine with a 1-bit input signal whose state transition graph is a binary tree. BT-FSMs are useful in those application areas where searching in a binary tree is required, such as computer networks, compression, automatic control, or cryptography. This paper presents a new architecture for implementing BT-FSMs which is based on the model finite virtual state machine (FVSM). The proposed architecture has been compared with the general FVSM and conventional approaches by using both synthetic test benches and very large BT-FSMs obtained from a real application. In synthetic test benches, the average speed improvement of the proposed architecture respect to the best results of the other approaches achieves 41% (there are some cases in which the speed is more than double). In the case of the real application, the average speed improvement achieves 155%.

Finite state machines (FSM) are widely used in testing and verification of discrete event and hybrid systems. In order to guarantee fault coverage, various fault models and formal methods are utilized ranging from classical finite state machines (FSM) to logic circuits. The main problem for deriving an FSM-based test suite is the size of an FSM for real digital circuits, and thus, test engineers prefer to randomly derive a prefix of a test sequence and later on extend it with tests for hard detectable faults of a corresponding logic circuit. Usually at the latter step test sequences detecting hard detectable stuck-at faults are considered. In this paper, we are concerned about single functional faults, namely, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{a},\boldsymbol{b}-\mathbf{faults}$</tex> , of a system of Boolean functions of the combinational part of a sequential circuit such as deleting a variable or a cube in a SoP which represents the ON-set or the OFF-set of a corresponding Boolean function and experimentally evaluate fault coverage of a corresponding complete test suite with respect to fault coverage of tests returned by other methods. The experimental results show high fault coverage of test suites complete with respect to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{a},\boldsymbol{b}-\mathbf{faults}$</tex> and thus, it can be recommended to utilize such a test suite instead of a complete FSM-based test suite or to extend randomly generated test sequences for detecting <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{a},\boldsymbol{b}-\mathbf{faults}$</tex> not killed with the prefix.

Finite state machines (FSMs) control the functionality of System-on-Chip (SoC) designs. The security and trustworthiness of SoCs can be compromised by exploring the vulnerabilities of their FSMs. Any deviation from the specification of FSMs can endanger the security and trustworthiness of SoCs. This is a critical concern when an FSM is responsible for controlling the usage or propagation of protected information (e.g., secret keys) in a secure component. FSM vulnerabilities may be introduced by a rogue designer or an attacker who inserts hardware Trojans in the FSM implementation. Traditional FSM design flows as well as CAD tools may create unintentional security vulnerabilities in FSM designs (e.g., when a synthesis tool is trying to optimize a gate-level netlist). These vulnerabilities can also be introduced unintentionally by a CAD tool. In this chapter, we present an efficient formal analysis framework based on symbolic algebra to find FSM vulnerabilities. The proposed method tries to find inconsistencies between the specification and FSM implementation through manipulation of respective polynomials. Security properties (such as a safe transition to a protected state) are derived using specification polynomials and verified against implementation polynomials. In case of a failure, the vulnerability is reported. While existing methods can verify legal transitions, the proposed approach tries to solve the important and non-trivial problem of detecting illegal accesses to the design states (e.g., protected states).

Research demonstrated that faults seeded mutation using operators can be representative of faults in real systems. In this paper, we study the relationship between the different operators used to insert mutants in the fault domain of the system under test and the effectiveness of different state machine test suites at killing those mutants. We are particularly interested in the effectiveness of two interrelated state machine testing strategies at finding different types of faults. Those are the round-trip paths strategy and the transition tree strategy. Using empirical evaluation, we compare the effectiveness of more than two thousand unique test suites at killing mutants seeded using eight different mutation operators. We perform experiments on four experimental objects and provide qualitative analysis of the results. We conclude that neither of the two studied strategies is more effective than the other at killing a certain type of mutants. However, the structure of the finite state machine and the nature of the system under test affect the type of faults detected by the different testing strategies.

A new fast PWM modulation strategy for power electric converter based on finite set state machine theory

The finite state machine (FSM) needed for the low power system pulse frequency modulated (PFM) mode in a buck converter is usually asynchronous because the fast clock needed for a synchronous FSM consumes too much power, or is maybe even not available. However, the implementation, verification and testing of a asynchronous FSM is complicated compared to an synchronous one. This paper presents a concept of a semi-synchronous FSM that combines the benefits of both synchronous and asynchronous state machines. The result is a FSM which runs at high clock frequency, consumes very little power and can be implemented, verified and tested as a synchronous FSM. This concept has been used to design a PFM FSM in a field programmable gate array (FPGA) and in a 65 nm CMOS technology.

(Self-)reconfigurable finite state machine (FSM) had been introduced as a formal model to implement the sequential circuit which can be reconfigured during the operation. The reconfiguration is initiated by the external events or by the controller that generates the reconfiguration transitions. In this paper, we optimize the number of reconfiguration transitions required to reconfigure a given FSM into a new target FSM. A serious limitation of previously proposed techniques is that they depend on the number of essential transitions which are the different transitions between a given and a target FSMs. In this paper, we propose a new technique based on FSM decomposition to decompose a given and target FSMs into two sub machines. A part of essential transitions is implemented as a single decomposed submachine with the same functionality in two decomposed FSMs. This submachine is inactive and has no effect in the normal operation but reduces the number of essential transitions during the reconfiguration. We give a decomposition strategy, realize the proper hardware, and provide a set of experimental results that shows the reconfiguration transitions can be substantially reduced, in some cases up to 54%

In this paper we outline a theory for the environment-modeling problem , the problem of abstracting component finite state machines (FSMs)bordering a particular FSM of interest within a network of interacting FSMs. The goal is to lay a theoretical foundation for the automatic state reduction of large FSM networks. We feel this is a prerequisite for the efficient use of many verification techniques. We focus on computing conditions for the safe removal of a component FSM in a FSM network, where removal is safe if it preserves a certain well-defined trace equivalence. We present an optimized algorithm for determining language universality of a FSM, as well as determining independence of a FSM from those of its inputs connected to outputs of neighboring FSMs. These two properties, input independence and language universality, provide the necessary and sufficient conditions for safe removal. In addition, we show how simulation relations can be utilized, both to reduce the cost of computing safe removal and to create an appropriate abstract FSM when safe removal is not possible.

Extended Finite State Machines (EFSMs) are widely used when deriving tests for checking functional requirements for software implementations. However, the fault coverage of tests covering appropriate paths, variables, etc. of the specification EFSM, remains rather obscure and such tests do not detect many functional faults in EFSM implementations. In this paper, an approach is proposed for deriving complete tests with respect to functional faults of a proper Java EFSM implementation. First, an initial test suite derived against the specification EFSM is checked with respect to faults generated by a µJava tool. Since the EFSM software implementation is template based, each undetected fault can be easily mapped into a mutant EFSM of the specification machine. Thus, a distinguishing sequence is derived for two Finite State Machines modeling two EFSMs instead of deriving such a sequence for two programs. If the corresponding FSMs are too complex or cannot be completely derived, a test suite can be incomplete. However, the performed experiments clearly show that a test suite extended by such distinguishing sequences detects much more functional faults in software implementations of a system whose behaviour is described by the given EFSM.

Extended finite state machines (EFSMs) are widely used when deriving tests for checking the functional requirements for software implementations. However, the fault coverage of EFSMbased tests covering appropriate paths, variables, etc., remains rather obscure. Furthermore, these tests are known be incapable of detecting many functional faults frequently occurring in EFSM-based implementations. In this paper, an approach is proposed for deriving complete tests with the help of a proper Java EFSM implementation. Since the software is based on a template, the faults turn directly into EFSM faults. The method proposed here makes it possible to derive test suites that can detect functional faults. In the first step, the EFSM-based test suite derived by a well-known method is checked for completeness with respect to the faults generated by the μJava tool. Then, each undetected fault is easily mapped into an EFSM mutant. In the next step, some FSM abstraction is used to derive a distinguishing sequence for two finite-state machines (if such a sequence exists), which is added to the current test suite. The test derived in this way is complete with respect to the faults generated by μJava. If the corresponding FSM derived by EFSM modeling is too complex or no such FSM can be derived, the resulting test suite can be incomplete. However, the experiments performed by us clearly show that the original test suite extended by distinguishing sequences can detect many functional faults in software implementations when the given EFSM is used as a specification for the system.

Testing and incremental conformance testing of timed state machines

Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does not detect some (non-equivalent) mutants, T can be improved by adding test cases that detect those mutants. Mutants are created by applying mutation operators. Mutation operators are important because they define the characteristics of the system that are tested as well as the characteristics that are improved in the test suite. While mutation operators are well defined for a number of programming paradigms such as sequential or multi-threaded, to the best of our knowledge, mutation operators have not been defined for the actor programming model. In this paper, we define and classify mutation operators that can be used for mutation testing of actor programs.