A Tool to Support Model-Based Testing Activities

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

Model-Based Testing, the task of generating test inputs and oracles from a test model, has been successfully applied in the context of safety-critical real time systems. As these systems grow in complexity, test-models, designed to reflect the systems behaviour, will grow too. Currently testers face situations where test-models are too complex for present test generators. In this paper, we outline a software tool for the evaluation of the scalability of a combination of approaches for model-based test generation. We chose Networks of Timed Automata (NTA) as the modeling formalism because real-time properties can be specified and the semantics are well-defined. However, the tool input is given as a restricted UML statechart which is internally transformed. We expect this to increase industrial acceptance. The tool will provide the selection, parametrization and generation of a metaheuristic algorithm. The aim is to support test model specific generation algorithms. A simulator for NTAs will enable the metaheuristic to search for test goals in the model. For better performance, it will have an advanced parallelisation. Furthermore, input models will be used for search space reduction for even faster test case generation. The proposed approach allows the inclusion of an oracle generator that is able to provide expected outputs; this enables conformance checking between test models and systems under test. We plan to implement the outlined tool to enable test case generation even for models that are beyond the scope of currently available generators.

Requirements traceability enables the linkage between all development artifacts during the development process. Within model-based testing, requirements traceability links the original requirements with test model elements and generated test cases. Current approaches are either not practical or lack the necessary formal foundation for generating requirements-based test cases using model-checking techniques involving the requirements trace. This paper describes a practical and formal approach to ensure requirements traceability. The descriptions of the requirements are defined on path fragments of timed automata or timed state charts. The graphical representation of these paths is called a computation sequence chart (CSC). CSCs are automatically transformed into temporal logic formulae. A model-checking algorithm considers these formulae when generating test cases.

Welcome to the 13th edition of the Advances in Model-based testing (A-MOST 2017) held on March 17th, 2017 in beautiful Tokyo, Japan. Context of A-MOST: complexity of software systems is increasing rapidly and thus warrants the development of novel automated testing techniques that can manage the increased complexity of software. Model-Based Testing (MBT) is one such approach towards automated testing that allows managing the complexity of the Software under Test with the use of models. A recent survey shows that MBT has been increasing, being used both in academic and in industrial settings and thus still remains an interesting area of research. In the past 12 years, A-MOST has served as a good venue for interaction among model-based testing researchers from both academia and industry. In 2017, A-MOST received in total eight submissions, and six of them were accepted after a thorough review and discussion process. Each submission was reviewed by three members of the program committee. topics of accepted papers cover security testing, test case generation, conformance checking, software product line testing and usability testing. accepted papers are divided into two main sections, i.e., functional MBT and non-functional MBT that will be presented during the workshop. Moreover, we are also delighted to have two well-renowned MBT researchers as keynote speakers. first keynote speaker is Ina Schieferdecker who is the director of Fraunhofer FOKUS, Germany. topic of her keynote speech is The New Version of the UML Testing Profile. second keynote speaker is Shaukat Ali who is currently a senior research scientist from Simula Research Laboratory, Norway. His theme of the keynote speech is Uncertainty-Wise Testing. We thank them very much for accepting our invitations for their inspiring keynote speeches. We also appreciate the effort made by our program committee members for providing their timely reviews with high quality. Last but not least, we would like to thank the A-MOST steering committee for their support and advice. Last, we thank all the authors who submitted their high quality work to A-MOST 2017.

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Model-based testing has become increasingly popular in recent years. Major reasons include: (1) the need for quality assurance for increasingly complex systems, (2) the emerging model-centric development paradigm, e.g., UML and MDA, with its seemingly direct connection to testing, and (3) the advent of test-centered development methodologies. Model-based testing relies on execution traces of behavior models. They are used as test cases for an implementation: input and expected output. This complements the ideas of model-driven testing. The latter uses static models to derive test drivers to automate test execution. This assumes the existence of test cases, and is, like the particular intricacies of OO testing, not in the focus of this tutorial. We cover major methodological and technological issues: the business case of model-based testing within model-based development, the need for abstraction and inverse concretization, test selection, and test case generation. We (1) discuss different scenarios of model-based testing, (2) present common abstractions when building models, and their consequences for testing, (3) explain how to use functional, structural, and stochastic test selection criteria, and (4) describe today's test generation technology. We provide both practical guidance and a discussion of the state-of-the-art. Potentials of model-based testing in practical applications and future research are highlighted.

There is a resurgence of research in model-based testing, especially in the automated generation of test cases from abstract models. However this work largely remains theoretical: industrial adoption is low. This is partly due to the dominance of state-based approaches that often rely on global states that are problematic with respect to scalability and traceability. Developers and testers alike significantly prefer the intuitive nature, traceability and user-friendliness of scenarios, to the semantics of formal approaches. Proposals for scenario-driven testing exist but, as is the case for the vast majority of existing work on model-based testing, there is a considerable gap between the generated test cases and their corresponding IUT instrumentation. It is this problem we address here. In this paper we focus on modeling responsibilities and scenarios within a scenario-driven testing framework that generates fully-instrumented test cases. Our work proceeds from the scenario contracts proposed by Nebut et al.

Test case generation, selection and coverage from natural language

Context: Model-Based Testing (MBT) is a technique that employs formal models to represent reactive systems’ behav-ior and generates test cases. Such systems are mostly specified and verified using Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature. Most of them are graph-based once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. Objective: This study aims to compare graph-based algorithms employed to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known depth-first and breadth-first searches algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. Method: First, a systematic literature mapping was conducted to summarize the algorithms that have been adopted in MBT, considering FSM. The main algorithms found were implemented and analyzed in different experiments. Results: CPP presented better results with the FSMs used in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

As of today, model-based testing (MBT) is considered as leading-edge\ntechnology in industry. We sketch the different MBT variants that - according\nto our experience - are currently applied in practice, with special emphasis on\nthe avionic, railway and automotive domains. The key factors for successful\nindustrial-scale application of MBT are described, both from a scientific and a\nmanagerial point of view. With respect to the former view, we describe the\ntechniques for automated test case, test data and test procedure generation for\nconcurrent reactive real-time systems which are considered as the most\nimportant enablers for MBT in practice. With respect to the latter view, our\nexperience with introducing MBT approaches in testing teams are sketched.\nFinally, the most challenging open scientific problems whose solutions are\nbound to improve the acceptance and effectiveness of MBT in industry are\ndiscussed.\n

Cost-effective test case generation with the hyper-heuristic for software product line testing

Input/Output Transition Systems (IOTSs) have been widely used as test models in model-based testing. Traditionally, input output conformance testing (IOCO) has been used to generate random test cases from IOTSs. A recent test case generation method for IOTSs, called Complete IOCO, applies fault models to obtain complete test suites with guaranteed fault coverage for IOTSs. This paper measures the efficiency of Complete IOCO in comparison with the traditional IOCO test case generation implemented in the JTorX tool. To this end, we use a case study involving five specification models from the automotive and the railway domains. Faulty mutations of the specifications were produced in order to compare the efficiency of both test generation methods in killing them. The results indicate that Complete IOCO is more efficient in detecting deep faults in large state spaces while IOCO is more efficient in detecting shallow faults in small state spaces.

In Model Based Testing (MBT), test cases are generated automatically from a partial representation of expected behaviour of the System Under Test (SUT) (i.e., the model). For most industrial systems, it is impossible to generate all the possible test cases from the model. The test engineer recourse to generation algorithms that maximize a given coverage criterion, a metric indicating the percentage of possible behaviours of the SUT covered by the test cases. Our previous work redefined classical Transition Systems (TSs) criteria for SPLs, using Featured Transition Systems (FTSs), a mathematical structure to compactly represent the behaviour of a SPL, as model for test case generation. In this paper, we provide one all-states coverage driven generation algorithm and discuss its scalability and efficiency with respect to random generation. All-states and random generation are compared on fault-seeded FTSs.

Model‐based testing (MBT) automates the design and generation of test cases from a model. This process includes model building, test selection criteria, test case generation, and test case execution stages. Current tools support this process at various levels of automation, most of them supporting three out of four stages. Among them is MBT4J, a platform that extends ModelJUnit with several techniques, offering a high level of automation for testing Java applications. In this study, the authors evaluate the efficacy of the MBT4J platform, in terms of the number of test cases generated, errors detected, and coverage metrics. A case study is conducted using two open‐source Java systems from public repositories, and 15 different configurations. MBT4J was able to automatically generate five models from the source code. It was also able to generate up to 2025 unique test cases for one system and up to 1044 for the other, resulting in 167 and 349 failed tests, respectively. Transition and transition pair coverage reached 100% for all models. Code coverage ranged between 72 and 84% for the one system and between 59 and 76% for the other. The study found that Greedy and Random were the most effective testers for finding errors.

Extended finite state machine (EFSM) is among the most popular models for model-based testing. However, automated test case generation on EFSM models is still a challenging task since an EFSM model may contains infeasible paths. This paper proposed a novel approach (ATGEM) to generate test case and construct oracle information from EFSM automatically. To address the infeasible problem, a metric based on data flow analysis is presented to predict the infeasible probability so as to bypass the infeasible paths as far as possible and improve the test case generation efficiency. Afterwards, an executable EFSM model is developed to obtain runtime feedback information as a fitness function in order to generate test data and construct oracle information automatically. This approach, which can generate various types data and has a wide range of applications, combines static analysis and dynamic analysis aims to find a preferable feasible path subset to generate test cases and meet adequacy coverage criteria. The experimental results on several EFSM models show that test case generation method and path feasibility metric have good effectiveness. Utilizing path feasibility metric can speed up the process of test case generation greatly, and ATGEM is more efficient than existing method.