Combinatorial Testing in an Industrial Environment -- Analyzing the Applicability of a Tool

A Case Study in Matching Test and Proof Coverage

In last three decades, lot of research work has been done by statisticians and academicians (D. Richard Kuhn, Raghu N. Kacker, Vu Lei, Ljubomir Lazic, Sloane, Neil J. A., Madhav Phadke, D. M. Cohen, James Bach and others) in the field of Combinatorial testing (CT) and its use in software testing. However, its tremendous potential in terms of test optimization and fault detection is yet to be exploited to its fullest. This paper showcases some of the real-world examples from iGATE Corporation, where CT has been used successfully to achieve breakthrough results in software testing. The paper, through three case studies taken from different domains describes the challenges faced by testing teams in terms of test scenario complexities, schedule and budget constraints; and how CT helped resolve those challenges successfully. Our experience shows 60 to 70 percent benefits in terms of optimization of test efforts while enhancing test case coverage. In the end, the paper highlights some learnings from our experiences in implementing CT approach which may be of help to fellow practitioners.

This paper puts privacy cybersecurity protection into framework of an Internet of Things industrial environment (IIoT) and analyzes on two tools’ groups in particular: (1) Standards and technical norms specifically designed to be applied in industrial environments, and (2) assessment tools for cybersecurity of Intermet of Things (IoT) devices usable in industrial infrastructures and environments. These case studies show the cybersecurity challenges and concerns triggered by these limited technologies and how they are addressed by regulations, standards, and technical mechanisms. The goal of these case studies is to illustrate how regulation and technologies attempt to protect the cybersecurity interests in industrial contexts. Finally, the case studies aim to generate an understanding of challenges and how standards and technology address cybersecurity issues in these industrial contexts.

In this paper we describe how the test design can be done by using the Combinatorial Testing approach for internet of things operating systems. Contiki operating system is taken as a case study but we discuss what can be the approach for RIOT and Tiny OS operating systems. We discuss how the combinatorial coverage measurement can be gathered in addition to the traditional metrics code coverage. The test design generated by using Advanced Combinatorial Testing for Software is analyzed for Contiki operating system. We elaborate the code coverage gathering technique for Contiki simulator which happens to be in Java. We explain the usage of Combinatorial Coverage Measurement tool. Although we have explained the test design methodology for internet of things operating systems, the approach explained can be followed for other open source software.

Semi-automatic methodology for augmented panorama development in industrial outdoor environments

In situ atmospheric particulate matter (PM) monitoring networks play a critical role in advancing understanding of air quality dynamics across local to regional scales by providing continuous, site-resolved observations. In contrast to short-term measurement campaigns, sustained monitoring networks enable the characterization of long-term trends, episodic events, and source-specific variability, while providing essential data for model validation and exposure assessment. These capabilities are especially useful in industrial environments, where emissions are spatially heterogeneous, temporally variable, and chemically complex.We present the design and implementation of the Environmental Monitoring for Industrial Sites (ÉMIS) network, a ground-based, distributed system developed to monitor complex industrial emission environments. The network is designed to operate under challenging conditions, including high-latitude environments with extreme seasonal variability, limited access to grid power, and constrained connectivity. Each station integrates measurements of particulate matter across multiple size fractions (PM2.5, PM5, and PM10) and volatile organic compounds using low-cost optical sensors, local meteorology, and visual documentation via a conditionally triggered camera. Stations are additionally equipped with a modified Wilson and Cooke (MWAC) bottle sampler to collect long-term, sector-representative samples for chemical characterization.Stations transmit data using long-range radio (LoRa) at user-defined intervals (e.g., two-minute resolution) to a hub node, which aggregates and relays data via cellular or satellite communication to an online dashboard. Unlike Bluetooth or Wi-Fi, LoRa enables kilometer-scale data transmission without the cost or infrastructure requirements of cellular modems. This architecture provides near–real-time insight into emission dynamics while supporting long-term data continuity. Emphasis on low-cost, modular instrumentation reduces financial and logistical barriers associated with traditional monitoring systems, enabling high-density deployments accessible to researchers, industries and public stakeholders.The initial field deployment consisted of fifteen (15) stations distributed across a copper smelter in Quebec, Canada, spanning more than 1 km². This case study demonstrates the network’s ability to resolve spatial gradients and localized emission signals, while dealing with complex topography and climate conditions. Analysis reveals persistent PM hotspots associated with heavy machinery traffic, ore handling operations, and slag cooling. Periodic PM spikes linked to train transport, as well as clear relationships between wind speed, wind direction, and plume occurrence were identified.The multi-level data output from the ÉMIS network supports a wide range of applications, including PM source attribution, evaluation of emission dynamics, integration with receptor and dispersion modeling, and validation of satellite-derived products. The network also serves as a testbed for sensor development, quality assurance refinement, and cross-network harmonization under real-world industrial conditions. This work highlights how adaptable, cost-effective ground-based monitoring networks can expand observational capacity in industrial environments and support advances in atmospheric science, air-quality management, and community-informed decision-making.

Combinatorial testing (CT) is a well-known technique for improving the quality of test plans while reducing testing costs. Traditionally, CT is used by testers at testing phase to design a test plan based on a manual definition of the test space. In this work, we extend the traditional use of CT to other parts of the development life cycle. We use CT at early design phase to improve design quality. We also use CT after test cases have been created and executed, in order to find gaps between design and test. For the latter use case we deploy a novel technique for a semi-automated definition of the test space, which significantly reduces the effort associated with manual test space definition. We report on our practical experience in applying CT for these use cases to three large and heavily deployed industrial products. We demonstrate the value gained from extending the use of CT by (1) discovering latent design flaws with high potential impact, and (2) correlating CT-uncovered gaps between design and test with field reported problems.

Due to the complexity of the nuclear industrial distributed control system (DCS), input-domain testing techniques, including random testing and combinatorial testing, are usually utilized to test the control logics in nuclear industrial DCS. To improve the fault detection efficiency of random testing, the adaptive random testing technique selects a test case that significantly differs from all existing test cases. Similarly, to improve the fault detection efficiency of combinatorial testing, the greedy combinatorial testing technique adopts a greedy strategy to generate test cases that cover more uncovered tuple-combinations of parametric values. In this paper, we designed an experiment to compare the fault detection efficiency between adaptive random testing technique and greedy combinatorial testing technique for control logics of nuclear industrial DCS. Through the analysis of the fault detection ratios, the f-measure values, and the values of average percent of faults detected (APFD) on two experimental subjects, including the commonly used benchmarks in the field of Boolean-specification testing as well as a group of Boolean expressions extracted from the control logics in nuclear industrial DCS, the experimental results give us the following conclusions: (1) If the test suites' sizes are relatively small, the fault detection efficiencies of the two techniques are very close though there is a slight advantage in adaptive random testing; (2) With the gradual increase of test suites' sizes, the fault detection efficiency of greedy combinatorial testing is beyond adaptive random testing gradually. Such a result can help us select the appropriate testing techniques in the testing of the control logics in nuclear industry DCS.

In complex software systems, identification of parameter combinations that lead to failures is important for effective debugging and robust software quality assurance. Combinatorial testing (CT) can greatly reduce the number of test cases (TC) used for testing a complex system by generating TCs based on various parameter combinations, while still maintaining high fault-detection capabilities. In this paper, a rule-based approach is presented that will aid in identifying Failure Inducing Combinations (FIC) of parameters that caused a fault in CT generated TCs. The approach uses heuristics to identify rules to methodically split parameter combinations into more likely or less probable combinations causing failure, thus reducing the set of failure sources. This approach is tested on two different case studies i) Three factor authentication system and ii) Existing literature-based input. The results obtained show significant accuracy and time saving for program debugging, thereby indicating applicability of the approach to real-world problems. The approach was found to be successful in identifying the pair-wise combinations of parameters that are likely to cause failure.

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Verification tool for Vienna Development Method (BWDM) is a test case generation tool for the VDM++ specification. The existing BWDM could cause a combinatorial explosion of the generated test cases. To reduce the number of test cases, there is Pairwise Independent Combinatorial Testing Tool (PICT): a pairwise testing tool. We apply pairwise testing into BWDM. Here, BWDM cannot call PICT library directly. Hence, we have developed PICT-wrapper. It is an interface to connect PICT and BWDM. The extended BWDM eliminate the possibility of the combinatorial explosion.

Combinatorial testing has been widely utilized in testing softwares, e.g. Siemens Suite. This paper aims to investigate the reason why combinatorial testing works in Siemens Suite. Experiments are designed to get the MFS(minimal failure-causing schema) for Siemens Suite, which has been used as a benchmark to evaluate the effectiveness of many testing techniques. The lowerbound of fault-detecting probability of T-way combinatorial test suite for each program is calculated by analyzing the strengths and the number of MFS for each faulty version. Computational results could explain the effectiveness of combinatorial testing in Siemens Suite.

Combinatorial test case generation faces a problem on how to reduce the test cases by uncover the unnecessary test cases. So, there is a need for expert applications or strategies that generate the most optimum test cases keeping in mind the most important combinations. Complementing existing work on combinatorial test case generation strategies, also known as t-way testing strategies (i.e., where t represents interaction degree), this paper presents the design and the implementation of a new combinatorial test list generation strategy based on Harmony Search (HS) algorithm, called General T-way Harmony Search-based Strategy (GTHS). HS has been chosen to be the main engine of test generation because it could balance between intensification and diversification. Benchmarking experimental results show that GTHS produces competitive results as compared to other existing well-known optimization-based strategies and provides the support for high combination degrees (i.e., t ≤ 12).

Internet of things (IoT) systems comprise of multiple devices connected together, to perform an intelligent task in real time. Such systems have to be meticulously tested in order to avoid hazards situations. Combinatorial testing technique can effectively test such complex IoT systems with reduced effort as it generates fewer test cases with adequate coverage. Prioritisation of combinatorial tests deals with finding an ideal order of the test cases so that faults are detected early. Recent approaches to prioritization problem are either coverage based or parameter-value weight based for twoway or three-way interaction strengths separately. In this paper, we present a hybrid prioritisation technique for combinatorial testing that combines both weight based and interaction coverage based approaches. To demonstrate the effectiveness of our algorithm, we have conducted initial synthetic experiments on various covering arrays, and measured the effectiveness with t-Rate of fault of detection metric. The results are promising in covering the combinations early.

A test scenario generation method based on combinatorial testing (CT) and Bayesian Network for autonomous vehicles is proposed in this paper. Firstly, some parameters are selected to describe the test scenarios which are classified according to road types and driving tasks. Then, the constraint sets for the scenarios with forbidden tuples are established to avoid the generated cases do not conform to the reality, in which the construct constraint set (CCS) algorithm is utilized to compute implied constraints. Furthermore, the Bayesian networks is used as the probabilistic models of the scenarios, where the traffic participants are represented as object nodes and the relative relationships between the participants are converted into the network structures. Finally, an improved automatic efficient test case generator (AETG) is developed to generate test cases. By considering both probability and frequency, the select function is designed for determining the values of scenario parameters. And the generation mode can be changed by modifying the weight and target parameters. The effectiveness of the proposed method is evaluated by generating six typical test scenarios. Compared with other algorithms, the numbers of test cases in the sets generated by this method are less and the probability deviations are smaller.