Enhancing Acceptance Test-Driven Development with Model-Based Test Generation

In this article we state that cyber-physical systems and the Internet of Things pose challenges to software quality research. In the emerging real-time digital economy companies need to gain deep customer insight. The state of the art in modelbased systems and model-based testing allows software engineers for product-based and quantitative control of quality and for increased productivity. These gains can be invested to better understand the domain and business conditions. We argue that cyber-physical systems will pose an excellent basis for conducting collaborative research in model-based systems and model-based testing in the near future and report on lessons learnt in three areas of software (quality) research: (1) process-oriented quality, (2) model-based systems and (3) model-based testing.

Model-Based Testing (MBT) was first introduced in 1970′s, and has the potential to improve efficiency and effectiveness of testing. However, its adoption—especially for web applications—has been hindered by the effort required to manually design MBT models, and keep them updated. Based on the above challenge in a real industrial context, this study introduces an automated approach to reduce that effort by reverse engineering MBT models from clickstream data captured during users' interaction with web applications. We have developed and present in this paper an open-source tool, named MBTModelGenerator , which logs user interactions via a lightweight JavaScript module in the front-end, and transmits them to a REST API backend. These interactions are then transformed into directly executable MBT models in the input format of an open-source MBT tool named GraphWalker . The tool was evaluated on two representative open-source web applications, Spring PetClinic and a Task Manager web app, and is under evaluation in several large-scale industrial testing projects. The generated MBT models accurately reflected user navigation flows and could be executed in the GraphWalker MBT tool without any manual changes. Using the tool has significantly reduced the effort of MBT model design by more than 90%, while still allowing test engineers to inspect and refine the generated models for completeness. Our approach facilitates lightweight adoption of MBT by automating model generation, which is the most effort intensive phase of MBT. To ensure correctness and completeness, the generated models should still be reviewed by test engineers —but that effort remains substantially lower than designing MBT models from scratch. The tool is in active industrial use and available as open-source for reuse and further development.

In the Software Development Life Cycle, modelling plays a most significant role in designing and developing software efficiently. Acceptance Test-Driven Development (ATDD) is a powerful agile software development model where a customer provides user acceptance test suits as a part of Software Requirements Specifications. A design has to develop a system so that User Acceptance Tests will be successful. In some systems, the Combinatorial Logic and Combinatorial Testing play a very crucial role. The authors have proposed a novel approach to enhance the existing Acceptance Test Driven Development model to Combinatorial Logic Oriented-ATDD model by incorporating combinatorial logic. Refinement with respect to combinatorial logic needs to be incorporated in all the stages of Software Development Life Cycle, i.e. starting from Software Requirement Specifications to User Acceptance Tests. This comprehensive approach derives the acceptance tests from user requirements effectively and efficiently. In this paper, the existing Indian Railway Reservation System is considered as a case study, and it was fully implemented as per proposed Combinatorial Logic Oriented-ATDD model.

As of today, model-based testing (MBT) is considered as leading-edge\ntechnology in industry. We sketch the different MBT variants that - according\nto our experience - are currently applied in practice, with special emphasis on\nthe avionic, railway and automotive domains. The key factors for successful\nindustrial-scale application of MBT are described, both from a scientific and a\nmanagerial point of view. With respect to the former view, we describe the\ntechniques for automated test case, test data and test procedure generation for\nconcurrent reactive real-time systems which are considered as the most\nimportant enablers for MBT in practice. With respect to the latter view, our\nexperience with introducing MBT approaches in testing teams are sketched.\nFinally, the most challenging open scientific problems whose solutions are\nbound to improve the acceptance and effectiveness of MBT in industry are\ndiscussed.\n

Model-based testing is of high practical relevance and many model-based testing approaches have been developed during the last years. But the key question under which conditions model-based testing pays off and a related decision support procedure for its application has not been sufficiently addressed. In this paper we develop a generic decision support procedure whether to apply model-based testing in a project or not. The decision support procedure compares estimated costs and benefits of model-based testing throughout all phases of the test process and is derived on the basis of a case study performed at the European Space Agency.

Test-Driven Development (TDD) is probably the most important agile engineering practice. Since it was first described in detail in [1], this development practice has been adopted widely. This adoption has been also well supported with tools that provide a framework for defining and executing unit tests on the different development platforms. Test-Driven Development provides a guideline how to develop applications unit by unit, resulting in well-designed, well maintainable quality software. TDD focuses on units and it ensures that the atomic building blocks and their interactions are specified and implemented correctly. There is certainly a need for automating other tests as well in order to ensure a working integration environment, to validate performance or to ensure that the application finally behaves as it was specified by the customer. Usually for automating these non-unit tests, developers (mis-)use the unit test frameworks and the unit test execution tools. This paper investigates the potential problems caused by misusing unit test tools for automated functional tests in cases when these functional tests are defined through the development technique called Acceptance Test Driven Development (ATDD). The misuse of the unit testing tools may have direct impact on the testing efficiency and it may also “close the doors” for features specialized for automated functional tests. Some results of this investigation have been prototyped in a tool called SpecRun, which aims to provide better automated functional testing efficiency.

This work describes the capabilities of a model-based analysis and automatic test generation system. Model checking ensures that an automatically generated code is free from contradictions that are difficult to identify manually in complex models. It generates test vectors and test drivers that can execute against model simulations or auto-generated code, and produces test sequence vectors for testing dynamic system behavior that support feedback, such as integrators or time delays, which are common in control systems. The paper briefly describes tool qualification support, and processes for using this model-based testing tool with modeling and code coverage tools to produce verification evidence that meets the FAA standards for certification of these systems, while reducing the verification costs by as much as 50%.

Model-Based Testing: Achievements and Future Challenges

This paper investigates the applicability of User Interaction Diagrams (UIDs) as user scenarios for specifying requirements of software built by non-technical customers. User scenarios represent an alternative to representation of Acceptance Test-Driven Development (ATDD). Two methods for building user scenarios using UIDs were proposed: the progressive and the regressive methods. The progressive method for construction of scenarios provides a description from any starting point until the expected result is reached. The regressive method is based on the Assert-First technique, introduced in Test-Driven Development (TDD), where the user scenario is constructed the other way round, that is, from the expected result to the starting point. These two methods were applied in an experiment where the results demonstrated that the regressive method requires significantly less effort as compared to the progressive method. The quality criteria of the two methods were different, where the regressive method yielded better results.

Recently, software engineering research has focused on IT system testing. Due to its practicality, academic and corporate research has presented various model-based testing methodologies. Model-based testing is a popular software technique that ensures system performance and evaluation. Automatic test case generation from requirements is intended to replace manual testing. Most of these approaches differ in critical areas like the specification paradigm and the model used to define requirements. Still, they share common principles, problems, and characteristics inherited from model-based testing. This paper proposes a meta-model that represents model-based testing principles and attributes to define the structure and entities of any new approach.

This research investigates how the codebase and acceptance tests have changed over time using version control. An essential requirement for this evaluation is analyzing software development and test cases used in real-world projects. The data stored in software repositories is vital for software systems. Access to prior versions of the program and the ability to differentiate between adjacent versions using acceptance tests and source code modifications can provide insight into how the software evolves. A skillful visualization method is required to achieve this level of awareness. This research presents a visualization method for tracking the concurrent evolution of source code and acceptance tests within software development projects. The case study looks at four open-source projects taken from the real world that have acceptance tests hosted on GitHub. The proposed visualization method enables the negative slope on two projects for the source code and acceptance test co-evolution to be quickly observed, and with such an observation, the project team can take necessary actions to change the course of the co-evolution.

Fishbone analysis process is a conceptual stage in engineering role and product design. It’s a systematic method to analysis a failure system that were longer used in electronic industry application but not limit to use in oil and gas industry. In risk management concept that a failure as a risk shall prevent and identify in beginning process that may require to minimize them to happen in product and application in oil field equipment. The main interest of this paper is to propose for the potential of fishbone model can be insert and enclose onto risk management system and then implement in oil field development project in oil and gas industry. This paper is provided base on research and experiences into risk management and learned from practical team and their opinion in industry which matched by an application in oil field development equipment such as actuators, valves and Christmas tree that follow to industry standard such as API 6A guidelines. A frame work of implementation phase for risk management fishbone in oil field development project in industry were reviewed and added value by improving an effective communication and skills of the project and engineering team to the stakeholder and organization. The frame work that added a Fishbone analysis method in engineering role may become a standard model and may work as reference for academic and practitioners to help evaluate which risk management need to emphasize in order to maximize the organization in industry outcome from each stage of model. The frame work model may also use for reference of develop the oil field development project with simply process and can be develop with same concept for the next wide development project in industry.

Model Based Testing (MBT) is a relatively an approach widely discussed in software testing. It extends test automation from test design to test execution using automatic test generation from models. The effective use of this approach requires new skills, technology and knowledge, such as test modelling skills, but also good programs and tool support. This paper explores a design of a Behaviour driven test automation framework using MBT and how it can be effectively used during Agile Development. The automation framework is experimented upon the integration of the Graphwalker, a Model Based Graphical User Interface test generator with behaviour driven development framework and Robot Framework.

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements. Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests. Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests. We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code. Given the complexity of testing, the model-based testing approach was used in conjunction with test automation harnesses. Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing. The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications. Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

The increasing complexity and wide applicability of software control systems requires the use of advanced and automated development and testing methodologies to meet time-to-market constraints, quality assurance and safety standards. In this context, the applicability of model-based development (MBD) and model-based testing (MBT) methodologies to such systems has gained attention in the last decade. The paper presents an integrated model-based approach to testing software control systems whose models are described by linear differential equations. The approach presented concerns conformance testing and includes the notation of tests, the implementation of a test comparator, the calculation of test coverage, the generation of test cases, and the execution of the tests. Finally, an example of a two-dimensional system is used for an experimental evaluation of the theoretical analysis and to present a perspective on the applicability of the approach for industrial projects.