Functional Test Case Generation based on model driven testing using FSM and UML Activity Diagram

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

UML activity and state chart diagrams are the most frequently UML models used for testing a system based on its specifications. Testers may choose different UML diagrams based on comprehensibility criteria. Previous studies, however, highlighted the lack of experts’ evaluation of comprehensibility of activity and state chart diagrams in test case generation. Thus, the main objective of this study is to evaluate the comprehensibility of the UML activity and state chart diagrams based on perspectives of experts. This study generated manual test cases from activity and state chart diagrams and used comprehensibility’s evaluation criteria during interviews with several software testing experts. The results of this study revealed that activity diagrams are more comprehensible than state chart diagrams in test case generation by considering comprehension variables (perceived difficulty and subjective confidence). The results of this study aim to unify the practical efforts of testers to choose MBT diagrams in generating test cases.

Context: Model-Based Testing (MBT) is a technique that employs formal models to represent reactive systems’ behav-ior and generates test cases. Such systems are mostly specified and verified using Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature. Most of them are graph-based once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. Objective: This study aims to compare graph-based algorithms employed to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known depth-first and breadth-first searches algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. Method: First, a systematic literature mapping was conducted to summarize the algorithms that have been adopted in MBT, considering FSM. The main algorithms found were implemented and analyzed in different experiments. Results: CPP presented better results with the FSMs used in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

The quality assurance of software product lines (SPL) achieved via testing is a crucial and challenging activity of SPL engineering. In general, the application of single-software testing techniques for SPL testing is not practical as it leads to the individual testing of a potentially vast number of variants. Testing each variant in isolation further results in redundant testing processes by means of redundant test-case executions due to the shared commonality. Existing techniques for SPL testing cope with those challenges, e.g., by identifying samples of variants to be tested. However, each variant is still tested separately without taking the explicit knowledge about the shared commonality and variability into account to reduce the overall testing effort. Furthermore, due to the increasing longevity of software systems, their development has to face software evolution. Hence, quality assurance has also to be ensured after SPL evolution by testing respective versions of variants. In this thesis, we tackle the challenges of testing redundancy as well as evolution by proposing a framework for model-based regression testing of evolving SPLs. The framework facilitates efficient incremental testing of variants and versions of variants by exploiting the commonality and reuse potential of test artifacts and test results. Our contribution is divided into three parts. First, we propose a test-modeling formalism capturing the variability and version information of evolving SPLs in an integrated fashion. The formalism builds the basis for automatic derivation of reusable test cases and for the application of change impact analysis to guide retest test selection. Second, we introduce two techniques for incremental change impact analysis to identify (1) changing execution dependencies to be retested between subsequently tested variants and versions of variants, and (2) the impact of an evolution step to the variant set in terms of modified, new and unchanged versions of variants. Third, we define a coverage-driven retest test selection based on a new retest coverage criterion that incorporates the results of the change impact analysis. The retest test selection facilitates the reduction of redundantly executed test cases during incremental testing of variants and versions of variants. The framework is prototypically implemented and evaluated by means of three evolving SPLs showing that it achieves a reduction of the overall effort for testing evolving SPLs.

The paper concerns automatic generation of usage models from real-time software UML models. Firstly, we define the reasonably constrained real-time software UML artifacts, which include use case diagrams, timed sequence diagrams and the execution probability of each sequence diagram in its associated use case. Secondly, the paper presents a method that derives the software usage model from the constrained UML artifacts. The method elicits the messages associated with the objects under testing and their occurrence probabilities to generate the usage model of each use case. Timing constraints in sequence diagrams are considered during usage model generation. Then the usage models of use cases are integrated into the software usage model by utilizing the execution sequence relations between use cases. The usage models can be used to generate real-time software statistical test cases and facilitate real-time software statistical testing.

In model-based testing using sequence diagrams, test cases are automatically derived from the sequence diagrams. For the generation of test cases, scenarios need to be found for representing as a sequence diagram, and to extract test paths satisfying the test coverage. However, it is hard to automatically extract test paths from the sequence diagram because a sequence diagram represents loop, opt, and alt information using CombinedFragments. To resolve this problem, we propose a transformation process that transforms a sequence diagram into an activity diagram which represents scenarios as a type of control flows. In addition, we generate test cases from the activity diagram by applying a test coverage concept. Finally, we present a case study for test cases generation from a sequence diagram.

This paper presents a prioritization technique that prioritizes the test cases using different design attributes like cohesion, coupling, the number of database access, and non-functional requirements. First, the system requirements are modeled using UML activity diagram (AD). The AD is turned into activity diagram graph (ADG), and the ADG is traversed to find out the test scenarios that are identified by the linearly independent paths in the ADG. Depending upon the different design attributes, weights for each node in the graph are identified and a final priority value is assigned to each node. The nodes executed for every test are identified, the priority value (PV) of the nodes is summed up, and the test case is assigned with a final priority value. Finally, depending on the priority value, the test cases are prioritized. The efficiency of the suggested approach is evaluated using the APFD metric.

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

Improvement of the Decay Heat Removal Characteristics of the Generation IV Gas-cooled Fast Reactor

Model-Based Testing: Achievements and Future Challenges

A Picture from the Model-Based Testing Area: Concepts, Techniques, and Challenges

The importance of renewable energy sources for the electrical power supply has grown rapidly in the past decades. Their often unpredictable nature however poses a threat to the stability of the existing electric grid. Hydroelectric powerplants play an important role in regulating the integration of renewable energy sources into the network by supplying on-demand load balancing as well as primary and secondary power network control. Therefore, the operating ranges of hydraulic machines has to be continuously extended, which potentially produces undesirable flow phenomena involving cavitation. An example is the formation of a gaseous volume in the swirling flow leaving a Francis turbine runner at off-design operating conditions. At high load, this so called vortex rope is shaped axisymmetrically and may enter a self-excited oscillation, measurable through significant fluctuations of the pressure throughout the system and the mechanical torque transferred to the generator. The main objective of the present work is the identification of the physical mechanisms governing this self-sustained, unstable behavior by measurement. Furthermore, the key parameters of numerical approaches using one-dimensional hydroacoustic flowmodels or CFD require experimental validation. For this purpose, the measurements provide a comprehensive data base of various flow and system parameters at varying operating conditions. Two test cases are studied, a small scale hydraulic circuit with a micro-turbine as well as a reduced scale physical model of an existing Francis turbine. On the first test case, the study of the flow rate fluctuations up- and downstream of the oscillating vortex rope in the draft tube, together with the volume of the cavity, revealed the destabilizing effect of the flow swirl in the draft tube inlet. The second test case accurately simulates the behavior of an actual hydraulic power plant. Investigations range from a local study of the flow field in the draft tube cone bymeans of LDV, PIV, high speed visualization and wall pressure measurements to a global analysis, considering the response of the hydraulic and mechanical system to the excitation by the vortex rope oscillation. Among the main observations is a periodical variation of the flow swirl in the draft tube, synchronized with the pressure oscillations. This is likely to be caused by a cyclically appearing volume of cavitation on the runner blades, modifying the relative flow angle at the outlet. The interaction of the blade cavitation and the vortex rope oscillation via the flow swirl is found to play a crucial role in the occurrence of self-excited pressure oscillations in Francis turbines.

The Model Driven Architecture (MDA) approach is emerged in the last years as a novel software design methodology for the development of software systems. In this approach the focus of software development is shifted from writing code to modeling. In MDA, developers implement models that are automatically transformed into the target code of the system. Alongside MDA, the Model Driven Testing (MDT) is emerging as a relevant research topic in both industrial and scientific communities. MDT is a methodology where test cases for the system are automatically obtained starting from test models to maximize specific model coverage criteria. Eventually, test cases are executed to verify the system code that is generated through an MDA approach. In this paper, we conduct an exploratory study in order to evaluate the differences that may exist between the model coverage guaranteed by the test cases and the code coverage reached when they are executed on the auto-generated code. Moreover, we identify the main factors that may influence these differences.

To effectively test parts of the Internet of Things (IoT) systems having a character of a state machine, Model-based Testing (MBT) approach can be taken. In MBT, a model of a system is created, and test cases generated automatically from the model, and a number of current strategies exist. In this paper we propose a novel alternative strategy, that concurrently allows to flexibly adjust the preferred length of the generated test cases, as well as to mark the states, in which the test case can start and end. Compared with an intuitive N-switch coverage-based strategy that aims at the same goals, our proposal generates a lower number of shorter test cases with less test step duplications.

Context: Model-Based Testing (MBT) is an approach that allows testers to represent the behavior of the system under test as models, specifying inputs and their expected outputs. From such models, existing tools might be employed to generate test cases automatically. While MBT represents a promising step towards the automation of test case generation, the quality of the model designed by the tester may impact, either positively or negatively, on its ability to reveal faults (i.e., the test effectiveness). Objective: In this context, we conducted a preliminary experiment to evaluate the impact caused by different testers when designing a test model for the same functionality. Method: In the experiment, the participants used Event Sequence Graphs and its supporting tool FourMA to create test models for two mobile apps: arXiv-mobile and WhoHasMyStuff. From the test models, test cases were generated using FourMA and concretized by means of the Robotium framework. In order to measure the impact of different testers, we employed code coverage (namely, instruction and branch coverage) as an estimation of test effectiveness. Results: Based on the results obtained, we observe high variation of code coverage among the testers. No tester was capable of producing a test model that subsumes all other testers' models with respect to code coverage. Moreover, factor learning seems not to reduce the code coverage variation. The relation between model size, modeling time, and code coverage were inconclusive. Conclusion: We conclude that further research effort on the MBT's modeling step is required to not only reduce the variation between testers, but also improving its effectiveness.