Industrial-Strength Model-Based Testing - State of the Art and Current Challenges

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

Welcome to the 13th edition of the Advances in Model-based testing (A-MOST 2017) held on March 17th, 2017 in beautiful Tokyo, Japan. Context of A-MOST: complexity of software systems is increasing rapidly and thus warrants the development of novel automated testing techniques that can manage the increased complexity of software. Model-Based Testing (MBT) is one such approach towards automated testing that allows managing the complexity of the Software under Test with the use of models. A recent survey shows that MBT has been increasing, being used both in academic and in industrial settings and thus still remains an interesting area of research. In the past 12 years, A-MOST has served as a good venue for interaction among model-based testing researchers from both academia and industry. In 2017, A-MOST received in total eight submissions, and six of them were accepted after a thorough review and discussion process. Each submission was reviewed by three members of the program committee. topics of accepted papers cover security testing, test case generation, conformance checking, software product line testing and usability testing. accepted papers are divided into two main sections, i.e., functional MBT and non-functional MBT that will be presented during the workshop. Moreover, we are also delighted to have two well-renowned MBT researchers as keynote speakers. first keynote speaker is Ina Schieferdecker who is the director of Fraunhofer FOKUS, Germany. topic of her keynote speech is The New Version of the UML Testing Profile. second keynote speaker is Shaukat Ali who is currently a senior research scientist from Simula Research Laboratory, Norway. His theme of the keynote speech is Uncertainty-Wise Testing. We thank them very much for accepting our invitations for their inspiring keynote speeches. We also appreciate the effort made by our program committee members for providing their timely reviews with high quality. Last but not least, we would like to thank the A-MOST steering committee for their support and advice. Last, we thank all the authors who submitted their high quality work to A-MOST 2017.

Model-Based Testing (MBT) was first introduced in 1970′s, and has the potential to improve efficiency and effectiveness of testing. However, its adoption—especially for web applications—has been hindered by the effort required to manually design MBT models, and keep them updated. Based on the above challenge in a real industrial context, this study introduces an automated approach to reduce that effort by reverse engineering MBT models from clickstream data captured during users' interaction with web applications. We have developed and present in this paper an open-source tool, named MBTModelGenerator , which logs user interactions via a lightweight JavaScript module in the front-end, and transmits them to a REST API backend. These interactions are then transformed into directly executable MBT models in the input format of an open-source MBT tool named GraphWalker . The tool was evaluated on two representative open-source web applications, Spring PetClinic and a Task Manager web app, and is under evaluation in several large-scale industrial testing projects. The generated MBT models accurately reflected user navigation flows and could be executed in the GraphWalker MBT tool without any manual changes. Using the tool has significantly reduced the effort of MBT model design by more than 90%, while still allowing test engineers to inspect and refine the generated models for completeness. Our approach facilitates lightweight adoption of MBT by automating model generation, which is the most effort intensive phase of MBT. To ensure correctness and completeness, the generated models should still be reviewed by test engineers —but that effort remains substantially lower than designing MBT models from scratch. The tool is in active industrial use and available as open-source for reuse and further development.

Requirements traceability enables the linkage between all development artifacts during the development process. Within model-based testing, requirements traceability links the original requirements with test model elements and generated test cases. Current approaches are either not practical or lack the necessary formal foundation for generating requirements-based test cases using model-checking techniques involving the requirements trace. This paper describes a practical and formal approach to ensure requirements traceability. The descriptions of the requirements are defined on path fragments of timed automata or timed state charts. The graphical representation of these paths is called a computation sequence chart (CSC). CSCs are automatically transformed into temporal logic formulae. A model-checking algorithm considers these formulae when generating test cases.

Feature Interaction Aware Test Case Generation for Embedded Control Systems

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Context: Model-Based Testing (MBT) is a technique that employs formal models to represent reactive systems’ behav-ior and generates test cases. Such systems are mostly specified and verified using Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature. Most of them are graph-based once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. Objective: This study aims to compare graph-based algorithms employed to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known depth-first and breadth-first searches algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. Method: First, a systematic literature mapping was conducted to summarize the algorithms that have been adopted in MBT, considering FSM. The main algorithms found were implemented and analyzed in different experiments. Results: CPP presented better results with the FSMs used in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Model-based testing generates test cases from a high-level model. Current models employ extensions to finite-state machines. This work proposes a separation of transitions in the model and their corresponding actions in the target implementation, and also includes special treatment of exceptional states.

Model-based testing has become increasingly popular in recent years. Major reasons include: (1) the need for quality assurance for increasingly complex systems, (2) the emerging model-centric development paradigm, e.g., UML and MDA, with its seemingly direct connection to testing, and (3) the advent of test-centered development methodologies. Model-based testing relies on execution traces of behavior models. They are used as test cases for an implementation: input and expected output. This complements the ideas of model-driven testing. The latter uses static models to derive test drivers to automate test execution. This assumes the existence of test cases, and is, like the particular intricacies of OO testing, not in the focus of this tutorial. We cover major methodological and technological issues: the business case of model-based testing within model-based development, the need for abstraction and inverse concretization, test selection, and test case generation. We (1) discuss different scenarios of model-based testing, (2) present common abstractions when building models, and their consequences for testing, (3) explain how to use functional, structural, and stochastic test selection criteria, and (4) describe today's test generation technology. We provide both practical guidance and a discussion of the state-of-the-art. Potentials of model-based testing in practical applications and future research are highlighted.

The occurrence of fault on embedded critical systems can result in financial damage and human losses, as well. Rigorous approaches based on formalisms, specially testing routines, have been proposed to guarantee more reliable systems. Model-based testing is a rigorous approach to support testing activities, e.g. test case generation. Theoretical results have been validated in the literature, but there are few tools to support practical applications. Thus, we develop a tool to support practical experiments which allows us to make a comparison among model-based test case generation methods by pointing more suitable sceneries for each one.

Acceptance test-driven development is widely used in practice. However, writing and maintaining acceptance tests is a costly and time-consuming activity, in particular when a system is tested via the GUI. In model-based testing, the tests are automatically generated from a model of the system. In this paper, we report our experience from applying a combination of acceptance test-driven development and model-based testing in several real-world projects from industry. With the application of model-based testing, we increased test coverage and extend testing to usage scenarios not exercised by the existing acceptance tests. In the industry projects, MBT was used as an enhancement rather than a replacement for ATDD. By creating a layered test automation architecture, we were able to reuse the established automation for model-based testing and to apply both approaches simultaneously. This strategy also helped us to minimize the risks and to reduce the effort involved in introducing MBT in the projects.

There is a resurgence of research in model-based testing, especially in the automated generation of test cases from abstract models. However this work largely remains theoretical: industrial adoption is low. This is partly due to the dominance of state-based approaches that often rely on global states that are problematic with respect to scalability and traceability. Developers and testers alike significantly prefer the intuitive nature, traceability and user-friendliness of scenarios, to the semantics of formal approaches. Proposals for scenario-driven testing exist but, as is the case for the vast majority of existing work on model-based testing, there is a considerable gap between the generated test cases and their corresponding IUT instrumentation. It is this problem we address here. In this paper we focus on modeling responsibilities and scenarios within a scenario-driven testing framework that generates fully-instrumented test cases. Our work proceeds from the scenario contracts proposed by Nebut et al.

In model-based testing, test cases are generated from a specification model. To avoid an exhaustive search for all possible test cases that can be obtained, usually an expensive and infeasible activity, test case generation may be guided by a test selection criterion. The objective of a test selection criterion is to produce a minimal test suite and yet effective to reveal faults. However, the choice of a criterion is not straightforward specially for real-time systems, because most criteria presented in the literature are general-purpose. Moreover, the relationship between general-purpose and specific criteria for real-time systems is not clear. In this paper, we investigate the criteria that can be applied for test case generation in the scope of model-based testing of real-time systems, specifically of Timed Input-Output Symbolic Transition Systems (TIOSTS) models. We formalize a family of 19 test selection criteria ordered by strict inclusion relation for TIOSTS models. The family combines general-purpose data-flow-oriented and transition-based criteria with specific reactive and real-time systems criteria. We also perform an empirical study to compare the effectiveness of selected criteria. Results of the empirical study indicate that failure detection capability of the generated test suite may vary, but differences are not significant for time failures. We conclude that more effective criteria for the model-based testing of real-time systems are still needed.

Recently, software engineering research has focused on IT system testing. Due to its practicality, academic and corporate research has presented various model-based testing methodologies. Model-based testing is a popular software technique that ensures system performance and evaluation. Automatic test case generation from requirements is intended to replace manual testing. Most of these approaches differ in critical areas like the specification paradigm and the model used to define requirements. Still, they share common principles, problems, and characteristics inherited from model-based testing. This paper proposes a meta-model that represents model-based testing principles and attributes to define the structure and entities of any new approach.

Model-Based Testing, the task of generating test inputs and oracles from a test model, has been successfully applied in the context of safety-critical real time systems. As these systems grow in complexity, test-models, designed to reflect the systems behaviour, will grow too. Currently testers face situations where test-models are too complex for present test generators. In this paper, we outline a software tool for the evaluation of the scalability of a combination of approaches for model-based test generation. We chose Networks of Timed Automata (NTA) as the modeling formalism because real-time properties can be specified and the semantics are well-defined. However, the tool input is given as a restricted UML statechart which is internally transformed. We expect this to increase industrial acceptance. The tool will provide the selection, parametrization and generation of a metaheuristic algorithm. The aim is to support test model specific generation algorithms. A simulator for NTAs will enable the metaheuristic to search for test goals in the model. For better performance, it will have an advanced parallelisation. Furthermore, input models will be used for search space reduction for even faster test case generation. The proposed approach allows the inclusion of an oracle generator that is able to provide expected outputs; this enables conformance checking between test models and systems under test. We plan to implement the outlined tool to enable test case generation even for models that are beyond the scope of currently available generators.