Modeling Enhanced Scenarios for Automated Instrumentation

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

Model‐based testing (MBT) automates the design and generation of test cases from a model. This process includes model building, test selection criteria, test case generation, and test case execution stages. Current tools support this process at various levels of automation, most of them supporting three out of four stages. Among them is MBT4J, a platform that extends ModelJUnit with several techniques, offering a high level of automation for testing Java applications. In this study, the authors evaluate the efficacy of the MBT4J platform, in terms of the number of test cases generated, errors detected, and coverage metrics. A case study is conducted using two open‐source Java systems from public repositories, and 15 different configurations. MBT4J was able to automatically generate five models from the source code. It was also able to generate up to 2025 unique test cases for one system and up to 1044 for the other, resulting in 167 and 349 failed tests, respectively. Transition and transition pair coverage reached 100% for all models. Code coverage ranged between 72 and 84% for the one system and between 59 and 76% for the other. The study found that Greedy and Random were the most effective testers for finding errors.

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Requirements traceability enables the linkage between all development artifacts during the development process. Within model-based testing, requirements traceability links the original requirements with test model elements and generated test cases. Current approaches are either not practical or lack the necessary formal foundation for generating requirements-based test cases using model-checking techniques involving the requirements trace. This paper describes a practical and formal approach to ensure requirements traceability. The descriptions of the requirements are defined on path fragments of timed automata or timed state charts. The graphical representation of these paths is called a computation sequence chart (CSC). CSCs are automatically transformed into temporal logic formulae. A model-checking algorithm considers these formulae when generating test cases.

Test case generation, selection and coverage from natural language

Model-based testing has become increasingly popular in recent years. Major reasons include: (1) the need for quality assurance for increasingly complex systems, (2) the emerging model-centric development paradigm, e.g., UML and MDA, with its seemingly direct connection to testing, and (3) the advent of test-centered development methodologies. Model-based testing relies on execution traces of behavior models. They are used as test cases for an implementation: input and expected output. This complements the ideas of model-driven testing. The latter uses static models to derive test drivers to automate test execution. This assumes the existence of test cases, and is, like the particular intricacies of OO testing, not in the focus of this tutorial. We cover major methodological and technological issues: the business case of model-based testing within model-based development, the need for abstraction and inverse concretization, test selection, and test case generation. We (1) discuss different scenarios of model-based testing, (2) present common abstractions when building models, and their consequences for testing, (3) explain how to use functional, structural, and stochastic test selection criteria, and (4) describe today's test generation technology. We provide both practical guidance and a discussion of the state-of-the-art. Potentials of model-based testing in practical applications and future research are highlighted.

In Model Based Testing (MBT), test cases are generated automatically from a partial representation of expected behaviour of the System Under Test (SUT) (i.e., the model). For most industrial systems, it is impossible to generate all the possible test cases from the model. The test engineer recourse to generation algorithms that maximize a given coverage criterion, a metric indicating the percentage of possible behaviours of the SUT covered by the test cases. Our previous work redefined classical Transition Systems (TSs) criteria for SPLs, using Featured Transition Systems (FTSs), a mathematical structure to compactly represent the behaviour of a SPL, as model for test case generation. In this paper, we provide one all-states coverage driven generation algorithm and discuss its scalability and efficiency with respect to random generation. All-states and random generation are compared on fault-seeded FTSs.

In model-based testing using sequence diagrams, test cases are automatically derived from the sequence diagrams. For the generation of test cases, scenarios need to be found for representing as a sequence diagram, and to extract test paths satisfying the test coverage. However, it is hard to automatically extract test paths from the sequence diagram because a sequence diagram represents loop, opt, and alt information using CombinedFragments. To resolve this problem, we propose a transformation process that transforms a sequence diagram into an activity diagram which represents scenarios as a type of control flows. In addition, we generate test cases from the activity diagram by applying a test coverage concept. Finally, we present a case study for test cases generation from a sequence diagram.

Context: Model-Based Testing (MBT) is a technique that employs formal models to represent reactive systems’ behav-ior and generates test cases. Such systems are mostly specified and verified using Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature. Most of them are graph-based once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. Objective: This study aims to compare graph-based algorithms employed to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known depth-first and breadth-first searches algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. Method: First, a systematic literature mapping was conducted to summarize the algorithms that have been adopted in MBT, considering FSM. The main algorithms found were implemented and analyzed in different experiments. Results: CPP presented better results with the FSMs used in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Cost-effective test case generation with the hyper-heuristic for software product line testing

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

In model-based testing, test cases are generated from a specification model. To avoid an exhaustive search for all possible test cases that can be obtained, usually an expensive and infeasible activity, test case generation may be guided by a test selection criterion. The objective of a test selection criterion is to produce a minimal test suite and yet effective to reveal faults. However, the choice of a criterion is not straightforward specially for real-time systems, because most criteria presented in the literature are general-purpose. Moreover, the relationship between general-purpose and specific criteria for real-time systems is not clear. In this paper, we investigate the criteria that can be applied for test case generation in the scope of model-based testing of real-time systems, specifically of Timed Input-Output Symbolic Transition Systems (TIOSTS) models. We formalize a family of 19 test selection criteria ordered by strict inclusion relation for TIOSTS models. The family combines general-purpose data-flow-oriented and transition-based criteria with specific reactive and real-time systems criteria. We also perform an empirical study to compare the effectiveness of selected criteria. Results of the empirical study indicate that failure detection capability of the generated test suite may vary, but differences are not significant for time failures. We conclude that more effective criteria for the model-based testing of real-time systems are still needed.

Testing is the fundamental technique to assess the correctness of software systems, but it is cost-labored to generate test cases. One solution to change the situation is to automatize some parts of the testing process, especially the generation of test cases using formal theory and technology. The research work in the direction shows the good perspective. This paper targets on the automatic generation of test cases based on IOSTS, which is widely used to model reactive systems with data. When selecting test cases based on a set of test purposes specified by IOSTS or temporal logic, in general, the redundancy phenomena are unavoidable in the derived test suite. Hence, some strategies are presented for eliminating the redundancies in order to reduce the cost of implementing testing. More importantly, the strategies are directly applied to test cases in form of IOSTS, such that it can reduce not only the size of test suite, but also the cost of deriving test cases.

Welcome to the 13th edition of the Advances in Model-based testing (A-MOST 2017) held on March 17th, 2017 in beautiful Tokyo, Japan. Context of A-MOST: complexity of software systems is increasing rapidly and thus warrants the development of novel automated testing techniques that can manage the increased complexity of software. Model-Based Testing (MBT) is one such approach towards automated testing that allows managing the complexity of the Software under Test with the use of models. A recent survey shows that MBT has been increasing, being used both in academic and in industrial settings and thus still remains an interesting area of research. In the past 12 years, A-MOST has served as a good venue for interaction among model-based testing researchers from both academia and industry. In 2017, A-MOST received in total eight submissions, and six of them were accepted after a thorough review and discussion process. Each submission was reviewed by three members of the program committee. topics of accepted papers cover security testing, test case generation, conformance checking, software product line testing and usability testing. accepted papers are divided into two main sections, i.e., functional MBT and non-functional MBT that will be presented during the workshop. Moreover, we are also delighted to have two well-renowned MBT researchers as keynote speakers. first keynote speaker is Ina Schieferdecker who is the director of Fraunhofer FOKUS, Germany. topic of her keynote speech is The New Version of the UML Testing Profile. second keynote speaker is Shaukat Ali who is currently a senior research scientist from Simula Research Laboratory, Norway. His theme of the keynote speech is Uncertainty-Wise Testing. We thank them very much for accepting our invitations for their inspiring keynote speeches. We also appreciate the effort made by our program committee members for providing their timely reviews with high quality. Last but not least, we would like to thank the A-MOST steering committee for their support and advice. Last, we thank all the authors who submitted their high quality work to A-MOST 2017.

Model Based Testing is one of the most critical area to be addressed efficiently to ensure effective testing of the given project. The system implemented combines UML with FSM to cover all scenarios with all possible paths. As Finite Machine also works on the trigger where conditions are framed and if these conditions are satisfied then next action is executed; this phenomenon motivates to build a framework for generating the test cases automatically covering all paths (activity diagram in UML helps to cover all paths) and conditions ( Finite State Machine helps to frame set of conditions). Model Based Testing designed considering all paths and conditions to check all scenarios to generate detailed test cases for given project or application.