A Rule-Based Approach for Identifying Failure Inducing Combinations in Combinatorial Testing

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Combinatorial testing (CT) is a well-known technique for improving the quality of test plans while reducing testing costs. Traditionally, CT is used by testers at testing phase to design a test plan based on a manual definition of the test space. In this work, we extend the traditional use of CT to other parts of the development life cycle. We use CT at early design phase to improve design quality. We also use CT after test cases have been created and executed, in order to find gaps between design and test. For the latter use case we deploy a novel technique for a semi-automated definition of the test space, which significantly reduces the effort associated with manual test space definition. We report on our practical experience in applying CT for these use cases to three large and heavily deployed industrial products. We demonstrate the value gained from extending the use of CT by (1) discovering latent design flaws with high potential impact, and (2) correlating CT-uncovered gaps between design and test with field reported problems.

In last three decades, lot of research work has been done by statisticians and academicians (D. Richard Kuhn, Raghu N. Kacker, Vu Lei, Ljubomir Lazic, Sloane, Neil J. A., Madhav Phadke, D. M. Cohen, James Bach and others) in the field of Combinatorial testing (CT) and its use in software testing. However, its tremendous potential in terms of test optimization and fault detection is yet to be exploited to its fullest. This paper showcases some of the real-world examples from iGATE Corporation, where CT has been used successfully to achieve breakthrough results in software testing. The paper, through three case studies taken from different domains describes the challenges faced by testing teams in terms of test scenario complexities, schedule and budget constraints; and how CT helped resolve those challenges successfully. Our experience shows 60 to 70 percent benefits in terms of optimization of test efforts while enhancing test case coverage. In the end, the paper highlights some learnings from our experiences in implementing CT approach which may be of help to fellow practitioners.

Combinatorial testing has lots of test cases, but software testers hope to get the best test coverage with the smallest test case suite. For the scale of produced test cases is so large that researchers have considered the implementation of the critical test cases. This article researches the classic combinatorial test methods and proposes methods to generate pair-wise testing cases with a priority. Firstly, we design formulas to compute the weights of priorities. Secondly, we adopt a greed algorithm to solve the combinatorial testing problems. Furthermore, we integrate the greed strategy into a genetic algorithm to improve the efficiency. It improves the testing efficiency while securing the detection rate of defects under limited resources.

Combinatorial test case generation faces a problem on how to reduce the test cases by uncover the unnecessary test cases. So, there is a need for expert applications or strategies that generate the most optimum test cases keeping in mind the most important combinations. Complementing existing work on combinatorial test case generation strategies, also known as t-way testing strategies (i.e., where t represents interaction degree), this paper presents the design and the implementation of a new combinatorial test list generation strategy based on Harmony Search (HS) algorithm, called General T-way Harmony Search-based Strategy (GTHS). HS has been chosen to be the main engine of test generation because it could balance between intensification and diversification. Benchmarking experimental results show that GTHS produces competitive results as compared to other existing well-known optimization-based strategies and provides the support for high combination degrees (i.e., t ≤ 12).

Although combinatorial testing has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test whose test cases need to be performed contiguously. For thorough testing, the testing requirements are not only to cover all the interactions among factors but also to cover all the value sequences of every factor. Generally, systems under test always involve constraints and dependencies in or among test cases. The constraints among test cases have not been effectively specified. First, we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage, and propose a clocked computation tree logic-based formal specification method for specifying constraints. Then, Particle Swarm Optimization (PSO) based Extended covering array Generator (PEG) is elaborated. To evaluate the constructed test suites, a method for verifying the constraints’ validity is presented, and kernel functions for measuring the coverage are also introduced. Finally, the performance of the proposed PEG is evaluated using several sets of benchmark experiments for some common constraints, and the feasibility and usefulness of PEG is validated.

Combinatorial testing is a prevailing approach to finding out faults in the extremely complex systems of train operation for railway. While combinatorial testing is applied to Chinese Train Control Systems (CTCS) for revealing faults, some potential faults tend to be arduous to trigger due to the interaction of multiple test cases. For the sake of accurate and efficient fault localization of CTCS under the circumstances of masking effects, a novel method of locating the faults of CTCS using the relationship matrix with frequent terms has been proposed in the paper. To begin with, failed test cases from CTCS were made full use of to calculate the frequentness of failure‐inducing elements for ordered tables. Then, the relationship matrix with frequent terms for CTCS was established to work out the suspiciousness of failure‐inducing combinations by the following relation of failure‐inducing elements in the above tables. Furthermore, an iterative approach to locating failure‐inducing combinations was adopted to obtain the full list of rankings of failure‐inducing combinations, among which top combinations, especially MFS (Minimal Failure‐causing Schema), were prone to discover the faults of CTCS. Finally, the effectiveness and accuracy of the proposed method for CTCS was confirmed on the simulation platform of CTCS for Beijing‐Shenyang High‐speed Rail. Empirical studies suggest that compared to the method of fault forest, the average suspiciousness in the proposed method is 12.17% higher than that and the number of additional test cases are reduced by 31.7% on average, which can produce more precise and efficient results of identifying the faults of CTCS in case of masking effects. The research achievements can provide some references for the safety evaluation and system optimization of CTCS for high‐speed rail. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

A test scenario generation method based on combinatorial testing (CT) and Bayesian Network for autonomous vehicles is proposed in this paper. Firstly, some parameters are selected to describe the test scenarios which are classified according to road types and driving tasks. Then, the constraint sets for the scenarios with forbidden tuples are established to avoid the generated cases do not conform to the reality, in which the construct constraint set (CCS) algorithm is utilized to compute implied constraints. Furthermore, the Bayesian networks is used as the probabilistic models of the scenarios, where the traffic participants are represented as object nodes and the relative relationships between the participants are converted into the network structures. Finally, an improved automatic efficient test case generator (AETG) is developed to generate test cases. By considering both probability and frequency, the select function is designed for determining the values of scenario parameters. And the generation mode can be changed by modifying the weight and target parameters. The effectiveness of the proposed method is evaluated by generating six typical test scenarios. Compared with other algorithms, the numbers of test cases in the sets generated by this method are less and the probability deviations are smaller.

Usage of combinatorial testing is wide spreading as an effective technique to reveal unintended feature interaction inside a given system. To this aim, test cases are constructed by combining tuples of assignments of the different input parameters, based on some effective combinatorial strategy. The most commonly used strategy is two-way (pairwise) coverage, requiring all combinations of valid assignments for all possible pairs of input parameters to be covered by at least one test case. In this paper a new heuristic strategy developed for the construction of pairwise covering test suites is presented, featuring a new approach to support expressive constraining over the input domain. Moreover, it allows the inclusion or exclusion of ad-hoc combinations of parameter bindings to let the user customize the test suite outcome. Our approach is tightly integrated with formal logic, since it uses test predicates to formalize combinatorial testing as a logic problem, and applies an external model checker tool to solve it. The proposed approach is supported by a prototype tool implementation, and early results of experimental assessment are also presented.

Due to the complexity of the nuclear industrial distributed control system (DCS), input-domain testing techniques, including random testing and combinatorial testing, are usually utilized to test the control logics in nuclear industrial DCS. To improve the fault detection efficiency of random testing, the adaptive random testing technique selects a test case that significantly differs from all existing test cases. Similarly, to improve the fault detection efficiency of combinatorial testing, the greedy combinatorial testing technique adopts a greedy strategy to generate test cases that cover more uncovered tuple-combinations of parametric values. In this paper, we designed an experiment to compare the fault detection efficiency between adaptive random testing technique and greedy combinatorial testing technique for control logics of nuclear industrial DCS. Through the analysis of the fault detection ratios, the f-measure values, and the values of average percent of faults detected (APFD) on two experimental subjects, including the commonly used benchmarks in the field of Boolean-specification testing as well as a group of Boolean expressions extracted from the control logics in nuclear industrial DCS, the experimental results give us the following conclusions: (1) If the test suites' sizes are relatively small, the fault detection efficiencies of the two techniques are very close though there is a slight advantage in adaptive random testing; (2) With the gradual increase of test suites' sizes, the fault detection efficiency of greedy combinatorial testing is beyond adaptive random testing gradually. Such a result can help us select the appropriate testing techniques in the testing of the control logics in nuclear industry DCS.

International standards for software testing have been published in 2013 and are now widely used in industry and academia. However, these standards are difficult to use for the analysis of testing for complex software. Complex software has a huge number of components and an enormous number of relations between them. If we analyze such complex software according to the software testing standards, the results of the analysis will have a huge number of testing patterns or cases. Therefore, planning a project of complex software testing is impossible, and even if we could plan it, the plan would be infeasible. We propose a method of testing analysis for complex software by using definitions of software architecture in international standards. We believe that the outcomes of the proposed method are software-test-architecture. The software-test-architecture helps to analyze complex software, plan software testing, and develop test cases. We select existing case studies that have testing on a large and complex software, and discuss techniques to apply the proposed method for the case studies.

Background & Problem Statement - Software testing is a critical phase in the software development lifecycle (SDLC), ensuring that applications function correctly, meet user requirements, and maintain high- quality standards. Traditional software testing approaches, including manual testing and rule-based automation, often face challenges in scalability, efficiency, and adaptability to dynamic software environments. Traditional testing methods are overwhelmed by complex software systems which slows down defect detection and extends both testing costs and release schedules. Machine Learning (ML) has emerged as a transformative solution, introducing predictive and adaptive capabilities that optimize test case selection, automate defect detection, and enhance overall software quality assurance (QA). This study explores the integration of ML in software testing, addressing the challenges of traditional QA methodologies and demonstrating how AI-driven frameworks improve testing efficiency. Methodology - To investigate the impact of ML in software testing, this research adopts a systematic approach by analyzing ML-driven test automation techniques, including predictive testing, adaptive test execution, and automated test case generation. Research reviews how Google Microsoft Facebook IBM and Deep Code put ML-based quality assurance frameworks into operation. The study leverages supervised learning, reinforcement learning, deep learning, and NLP-based techniques to demonstrate how ML models predict software defects, dynamically adapt test cases, and optimize testing resources. The research tests how ML-based testing models operate within CI/CD pipelines to improve ongoing testing and deployment flow. Analysis & Results - The analysis of ML-driven software testing reveals that predictive analytics improves early defect detection rates. It helps developers spend 37% less time debugging their work. Adaptive testing models, including self-healing test scripts, minimize maintenance costs by 50% and enhance test reliability in agile environments. The integration of NLP-based test case generation increases test coverage. NLP technology enables automatic connection between requirements and test cases at 89% success rate. Additionally, reinforcement learning techniques improve test case selection, reducing redundant test executions by 43%. Our research shows different ML methods work well to lessen incorrect error alerts. ML integration for QA surely increasing defect prediction accuracy and optimizing test execution time. Findings & Contributions - This research contributes to the field of AI-driven software testing by providing a comprehensive framework for ML-based QA methodologies. Our study shows that machine learning helps find more software problems better adapts test cases and lowers testing expenses to solve present software development needs. The study also identifies critical challenges, including data availability, model interpretability, and computational overhead, suggesting future research directions in Explainable AI (XAI), hybrid AI-ML testing models, and AI-driven security testing. As the industry moves toward AI-first software testing, this research paves the way for fully autonomous QA frameworks, enabling intelligent, scalable, and cost- effective software validation techniques. Keywords - Machine Learning, Software Testing, Quality Assurance, Predictive Testing, Adaptive Testing, Test Automation, Defect Prediction, Self- Healing Test Scripts, AI-Driven QA, Reinforcement

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

Internet of things (IoT) systems comprise of multiple devices connected together, to perform an intelligent task in real time. Such systems have to be meticulously tested in order to avoid hazards situations. Combinatorial testing technique can effectively test such complex IoT systems with reduced effort as it generates fewer test cases with adequate coverage. Prioritisation of combinatorial tests deals with finding an ideal order of the test cases so that faults are detected early. Recent approaches to prioritization problem are either coverage based or parameter-value weight based for twoway or three-way interaction strengths separately. In this paper, we present a hybrid prioritisation technique for combinatorial testing that combines both weight based and interaction coverage based approaches. To demonstrate the effectiveness of our algorithm, we have conducted initial synthetic experiments on various covering arrays, and measured the effectiveness with t-Rate of fault of detection metric. The results are promising in covering the combinations early.

Software testing is required to verify and validate systems. Combinatorial testing in one of the significant testing techniques. Design and select test cases for combinatorial testing considered as combinatorial problem. Even though there are some existing optimization algorithm based combinatorial testing strategies, that minimize the number of test cases, but most of these strategies based on one-test-at-a-time (OTAT) approach and none of them has adopted the one-parameter-at-a-time (OPAT) approach. Therefore, this paper will propose a new OPAT strategy based on Harmony Search Algorithm (HS) called OPAT-HS. OPAT-HS was originally designed only to support Covering Array (CA) and Mixed Covering Array (MCA) for uniform interaction strength. The result obtained in the experiments appears that OPAT-HS is always best at configurations with MCA notations.