Model-Based Testing: Achievements and Future Challenges

Model-based testing (MBT) of software systems is considered and well-known MBT methods and tools are reviewed. These methods make it possible to automate procedures performed by software test engineers and significantly increase the quality of testing software systems. The MBT technology, its components, and main tools are described. A list of problems arising in using MBT is compiled. A symbolic approach that makes it possible to solve some of existing MBT problems and the Generic Trace Generator based on the symbolic approach are presented. This tool was developed at the V. M. Glushkov Institute of Cybernetics. The symbolic approach is extended for the first time as a result of addition of the test execution stage that considerably improves the MBT technology.

The evolution of software testing technologies has significantly reduced software testing execution times, but the test design and generation are still often implemented with slow manual-oriented methods. Model-based testing (MBT) offers automation to test design and generation, and different MBT solutions are familiar from research, but more effort needs to be done to adopt MBT for industrial use. Hardware in the loop (HIL) is a simulation and testing technique used in the development and testing of embedded systems. HIL is a challenging application field for MBT due to complex and non-deterministic nature of some embedded systems. To tackle this problem, the authors present a novel prototype platform in which online and offline MBT is applied to HIL environment. MBT in general has been introduced for HIL in scientific literature before, but the application of online MBT in HIL is a novel approach. The whole novel MBT in HIL prototype platform along with the used MBT tool, platform architecture and MBT process are presented accompanied by experimental results and analysis of two case studies with an example embedded system under test.

Background & Problem Statement - Software testing is a critical phase in the software development lifecycle (SDLC), ensuring that applications function correctly, meet user requirements, and maintain high- quality standards. Traditional software testing approaches, including manual testing and rule-based automation, often face challenges in scalability, efficiency, and adaptability to dynamic software environments. Traditional testing methods are overwhelmed by complex software systems which slows down defect detection and extends both testing costs and release schedules. Machine Learning (ML) has emerged as a transformative solution, introducing predictive and adaptive capabilities that optimize test case selection, automate defect detection, and enhance overall software quality assurance (QA). This study explores the integration of ML in software testing, addressing the challenges of traditional QA methodologies and demonstrating how AI-driven frameworks improve testing efficiency. Methodology - To investigate the impact of ML in software testing, this research adopts a systematic approach by analyzing ML-driven test automation techniques, including predictive testing, adaptive test execution, and automated test case generation. Research reviews how Google Microsoft Facebook IBM and Deep Code put ML-based quality assurance frameworks into operation. The study leverages supervised learning, reinforcement learning, deep learning, and NLP-based techniques to demonstrate how ML models predict software defects, dynamically adapt test cases, and optimize testing resources. The research tests how ML-based testing models operate within CI/CD pipelines to improve ongoing testing and deployment flow. Analysis & Results - The analysis of ML-driven software testing reveals that predictive analytics improves early defect detection rates. It helps developers spend 37% less time debugging their work. Adaptive testing models, including self-healing test scripts, minimize maintenance costs by 50% and enhance test reliability in agile environments. The integration of NLP-based test case generation increases test coverage. NLP technology enables automatic connection between requirements and test cases at 89% success rate. Additionally, reinforcement learning techniques improve test case selection, reducing redundant test executions by 43%. Our research shows different ML methods work well to lessen incorrect error alerts. ML integration for QA surely increasing defect prediction accuracy and optimizing test execution time. Findings & Contributions - This research contributes to the field of AI-driven software testing by providing a comprehensive framework for ML-based QA methodologies. Our study shows that machine learning helps find more software problems better adapts test cases and lowers testing expenses to solve present software development needs. The study also identifies critical challenges, including data availability, model interpretability, and computational overhead, suggesting future research directions in Explainable AI (XAI), hybrid AI-ML testing models, and AI-driven security testing. As the industry moves toward AI-first software testing, this research paves the way for fully autonomous QA frameworks, enabling intelligent, scalable, and cost- effective software validation techniques. Keywords - Machine Learning, Software Testing, Quality Assurance, Predictive Testing, Adaptive Testing, Test Automation, Defect Prediction, Self- Healing Test Scripts, AI-Driven QA, Reinforcement

Recently, enterprises, organizations, and software companies are building more and more web applications to provide their services over the Internet. In order to fulfill various requirements, the complexity of web applications nowadays is increasing dramatically. As a result, the performance characteristics of web applications, including response time, throughput, etc, become more critical than before and should be taken into careful consideration. If the response time of a web application is poor, users may lose their interests even the function of the web application is correct. Therefore, how to execute performance testing on a complex web application systematically and efficiently will be an important issue. In this paper, a performance testing framework for REST-based web applications is introduced. The performance testing framework aims to provide software testers with an integrated process from test cases design, test scripts generation, to test execution. Based on the test cases designed by software testers and the appropriate software artifacts preserved by the framework (e.g., API document), the framework generates the corresponding performance test scripts, which can be executed by specific performance test tools. This helps software testers to focus more in the design of performance test cases. In addition, effort needed to understand the design and implementation of the application and to learn the operation of testing tools decrease. Thus, the efficiency of performance testing can be highly facilitated.

Model-based testing is of high practical relevance and many model-based testing approaches have been developed during the last years. But the key question under which conditions model-based testing pays off and a related decision support procedure for its application has not been sufficiently addressed. In this paper we develop a generic decision support procedure whether to apply model-based testing in a project or not. The decision support procedure compares estimated costs and benefits of model-based testing throughout all phases of the test process and is derived on the basis of a case study performed at the European Space Agency.

To effectively test parts of the Internet of Things (IoT) systems having a character of a state machine, Model-based Testing (MBT) approach can be taken. In MBT, a model of a system is created, and test cases generated automatically from the model, and a number of current strategies exist. In this paper we propose a novel alternative strategy, that concurrently allows to flexibly adjust the preferred length of the generated test cases, as well as to mark the states, in which the test case can start and end. Compared with an intuitive N-switch coverage-based strategy that aims at the same goals, our proposal generates a lower number of shorter test cases with less test step duplications.

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements. Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests. Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests. We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code. Given the complexity of testing, the model-based testing approach was used in conjunction with test automation harnesses. Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing. The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications. Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

The evolution of software testing has led to the adoption of AI-driven tools to optimize the efficiency, accuracy, and coverage of test case generation. This paper introduces an AI Test Design and Script Generator, a cutting-edge solution that automates requirement and document reviews, enhances test coverage through generative AI, and ensures secure segmentation of customers, organizations, and departments. Unlike traditional automation tools, this AI-driven approach integrates with private Large Language Models (LLMs), providing intelligent test generation without collecting personal or sensitive data. By leveraging AI for test case design, this tool allows business analysts and QA professionals to quickly assess risks, generate optimized test cases, and streamline testing efforts. This paper explores the challenges of traditional test case creation, the role of AI in overcoming these limitations, and the key benefits of AI-driven testing in enterprise environments.

As we know, software testing is an important part of software development lifecycle. More than 50% of the whole system development work and total cost were spend on the software testing. And it’s estimated that almost 60% of the total test time and cost were spent on the design of test cases. In recent years, the automated testing gradually replaces the traditional manual testing to become an important branch of software testing. In automated testing, the most important thing is to design and generate valid test case automatically. In this paper, In order to reduce the workload of testers on the test case generation, improve the test efficiency and makes test experiences be passed on, we design a common description method for the test case based on ontology; propose a semantic similarity measure method to retrieve the usable test cases from test case library based on WordNet; establish the relationship between the test case and test requirement though rules to modify the retrieved test case; generate the final test case sequence. At last, the practical applicability of the approach is evaluated through an experiment. Keywords-software testing; ontology; test case generation; semantic similarity measure

Model‐based testing (MBT) automates the design and generation of test cases from a model. This process includes model building, test selection criteria, test case generation, and test case execution stages. Current tools support this process at various levels of automation, most of them supporting three out of four stages. Among them is MBT4J, a platform that extends ModelJUnit with several techniques, offering a high level of automation for testing Java applications. In this study, the authors evaluate the efficacy of the MBT4J platform, in terms of the number of test cases generated, errors detected, and coverage metrics. A case study is conducted using two open‐source Java systems from public repositories, and 15 different configurations. MBT4J was able to automatically generate five models from the source code. It was also able to generate up to 2025 unique test cases for one system and up to 1044 for the other, resulting in 167 and 349 failed tests, respectively. Transition and transition pair coverage reached 100% for all models. Code coverage ranged between 72 and 84% for the one system and between 59 and 76% for the other. The study found that Greedy and Random were the most effective testers for finding errors.

This paper describes a versatile test facility developed by AECL for validation and reliability (V&R) testing of safety-critical software used in the process trip computers for CANDU reactors. It describes the hardware and software aspects of the test facility. The test hardware consists of a test rig with a test computer used for executing the test software and a process trip computer emulator. The test software is comprised of an operating system, a test interpreter, a test oracle, and a man-machine interface. This paper also discusses the application of the test facility in V&R testing of the process trip computer, how test scripts are prepared and automatically run on the test computer, and how test results are automatically generated by the test computer, thus eliminating potential human errors. The test scripts, which contain specific instructions for testing, are text files written in a special AECL test language. An AECL Test Language Interpreter (ATLIN) program interprets the test scripts and translates structured English statements in the test scripts into test actions. The intuitive nature of the special AECL test language, the version controlled test scripts in text format and automatic test logging feature facilitate the preparation of test cases, which are easy to repeat, review and readily modifiable, and production of consistent results. This paper presents the concept of adding a process trip computer emulator for use in preparation of V&R testing. The process trip computer emulator is designed independently from the actual process trip computer but based on the same functional specification as for the process trip computer. The use of the process trip computer emulator allows the test scripts to be exercised before the actual process trip computers are available for V&R testing, thereby, resulting in a significant improvement to the project schedule. The test facility, with the built-in process trip computer emulator, is also a valuable training tool for the V&R staff and plant personnel.

A Picture from the Model-Based Testing Area: Concepts, Techniques, and Challenges

In recent years, Model-Based Testing (MBT) has attracted an increasingly wide interest from industry and academia. MBT allows automatic generation of a large and comprehensive set of test cases from system models (e.g., state machines), which leads to the systematic testing of the system. However, even when using simple test strategies, applying MBT in large industrial systems often leads to generating large sets of test cases that cannot possibly be executed within time and cost constraints. In this situation, test case selection techniques are employed to select a subset from the entire test suite such that the selected subset conforms to available resources while maximizing fault detection. In this paper, we propose a new similarity-based selection technique for state machine-based test case selection, which includes a new similarity function using triggers and guards on transitions of state machines and a genetic algorithm-based selection algorithm. Applying this technique on an industrial case study, we show that our proposed approach is more effective in detecting real faults than existing alternatives. We also assess the overall benefits of model-based test case selection in our case study by comparing the fault detection rate of the selected subset with the maximum possible fault detection rate of the original test suite.

The Model Driven Architecture (MDA) approach is emerged in the last years as a novel software design methodology for the development of software systems. In this approach the focus of software development is shifted from writing code to modeling. In MDA, developers implement models that are automatically transformed into the target code of the system. Alongside MDA, the Model Driven Testing (MDT) is emerging as a relevant research topic in both industrial and scientific communities. MDT is a methodology where test cases for the system are automatically obtained starting from test models to maximize specific model coverage criteria. Eventually, test cases are executed to verify the system code that is generated through an MDA approach. In this paper, we conduct an exploratory study in order to evaluate the differences that may exist between the model coverage guaranteed by the test cases and the code coverage reached when they are executed on the auto-generated code. Moreover, we identify the main factors that may influence these differences.