Combinatorial Security Testing—10 Years Later

This article demonstrates how institutional weakness makes amnesty, as a conflict management strategy, ineffective as found in Zamfara State; and how prioritising human security can help combat insecurity. Using Weak Institutions of State as a framework of analysis, the article argued that factors such as loss of territorial control or the sole use of force, crisis of legitimacy, mass poverty, corruption and inability to provide basic services to the citizens as characteristic of states with institutional weakness, create very complex security problems that amnesty is incapable of addressing effectively. Rather than mitigate insecurity, weak institutions in a state further fuel it by creating opportunities for both state and non-state actors to exploit the offer of amnesty for personal enrichment. This article adopted the documentary method of data collection, using deduction and inferences as analytical methods. Based on its findings, the study recommends the need for a more sustainable approach to conflict management that can benefit a larger number of citizens.

In last three decades, lot of research work has been done by statisticians and academicians (D. Richard Kuhn, Raghu N. Kacker, Vu Lei, Ljubomir Lazic, Sloane, Neil J. A., Madhav Phadke, D. M. Cohen, James Bach and others) in the field of Combinatorial testing (CT) and its use in software testing. However, its tremendous potential in terms of test optimization and fault detection is yet to be exploited to its fullest. This paper showcases some of the real-world examples from iGATE Corporation, where CT has been used successfully to achieve breakthrough results in software testing. The paper, through three case studies taken from different domains describes the challenges faced by testing teams in terms of test scenario complexities, schedule and budget constraints; and how CT helped resolve those challenges successfully. Our experience shows 60 to 70 percent benefits in terms of optimization of test efforts while enhancing test case coverage. In the end, the paper highlights some learnings from our experiences in implementing CT approach which may be of help to fellow practitioners.

Observation systems known as errors-in-variables (EIV) models with model parameters estimated by total least squares (TLS) have been discussed for more than a century, though the terms EIV and TLS were coined much more recently. So far, it has only been shown that the inequality-constrained TLS (ICTLS) solution can be obtained by the combinatorial methods, assuming that the weight matrices of observations involved in the data vector and the data matrix are identity matrices. Although the previous works test all combinations of active sets or solution schemes in a clear way, some aspects have received little or no attention such as admissible weights, solution characteristics and numerical efficiency. Therefore, the aim of this study was to adjust the EIV model, subject to linear inequality constraints. In particular, (1) This work deals with a symmetrical positive-definite cofactor matrix that could otherwise be quite arbitrary. It also considers cross-correlations between cofactor matrices for the random coefficient matrix and the random observation vector. (2) From a theoretical perspective, we present first-order Karush–Kuhn–Tucker (KKT) necessary conditions and the second-order sufficient conditions of the inequality-constrained weighted TLS (ICWTLS) solution by analytical formulation. (3) From a numerical perspective, an active set method without combinatorial tests as well as a method based on sequential quadratic programming (SQP) is established. By way of applications, computational costs of the proposed algorithms are shown to be significantly lower than the currently existing ICTLS methods. It is also shown that the proposed methods can treat the ICWTLS problem in the case of more general weight matrices. Finally, we study the ICWTLS solution in terms of non-convex weighted TLS contours from a geometrical perspective.

Search based algorithms for test sequence generation in functional testing

Combinatorial testing has lots of test cases, but software testers hope to get the best test coverage with the smallest test case suite. For the scale of produced test cases is so large that researchers have considered the implementation of the critical test cases. This article researches the classic combinatorial test methods and proposes methods to generate pair-wise testing cases with a priority. Firstly, we design formulas to compute the weights of priorities. Secondly, we adopt a greed algorithm to solve the combinatorial testing problems. Furthermore, we integrate the greed strategy into a genetic algorithm to improve the efficiency. It improves the testing efficiency while securing the detection rate of defects under limited resources.

Pairwise testing is a combinatorial testing technique that tests all possible pairs of input values. Although, finding a smallest set of test cases for pairwise testing is NP-complete, pairwise testing is regarded as a reasonable cost-benefit compromise among combinatorial testing methods. In this paper we formulate the problem of finding a pairwise test set as a search problem and apply a genetic algorithm to solve it. We also describe an open-source tool called PWiseGen for generating pairwise test sets. PWiseGen produces competitive results compared with existing pairwise testing tools. Besides, it provides a framework and a research platform for generating pairwise test sets using genetic algorithms; it is configurable, extensible, and reusable.

Due to the complexity of the nuclear industrial distributed control system (DCS), input-domain testing techniques, including random testing and combinatorial testing, are usually utilized to test the control logics in nuclear industrial DCS. To improve the fault detection efficiency of random testing, the adaptive random testing technique selects a test case that significantly differs from all existing test cases. Similarly, to improve the fault detection efficiency of combinatorial testing, the greedy combinatorial testing technique adopts a greedy strategy to generate test cases that cover more uncovered tuple-combinations of parametric values. In this paper, we designed an experiment to compare the fault detection efficiency between adaptive random testing technique and greedy combinatorial testing technique for control logics of nuclear industrial DCS. Through the analysis of the fault detection ratios, the f-measure values, and the values of average percent of faults detected (APFD) on two experimental subjects, including the commonly used benchmarks in the field of Boolean-specification testing as well as a group of Boolean expressions extracted from the control logics in nuclear industrial DCS, the experimental results give us the following conclusions: (1) If the test suites' sizes are relatively small, the fault detection efficiencies of the two techniques are very close though there is a slight advantage in adaptive random testing; (2) With the gradual increase of test suites' sizes, the fault detection efficiency of greedy combinatorial testing is beyond adaptive random testing gradually. Such a result can help us select the appropriate testing techniques in the testing of the control logics in nuclear industry DCS.

Software product lines achieve much shorter time to market by system level reuse and code variability. A possible way to achieve this flexibility is to use generic components, including the core system, in different products in alternative configurations. The focus of testing efforts for such complex and highly variable systems often shifts from testing specific products to assessing the overall quality of the core system or potential new configurations. As a complementary approach to feature models and related combinatorial testing methods optimizing for feature coverage, we apply a source code oriented analysis of variability. We present two algorithms that optimize for high coverage of the common code base in terms of C++ preprocessor-based configurations with a limited set of actual configurations selected for testing. The methods have been evaluated on iGO Navigation, a large industrial system with typical configuration support for product lines, hence we believe the approach can be generalized to other systems as well.

Correct behaviour of X.509 certificate validation code in SSL/TLS implementations is crucial to ensure secure communication channels. Recently, there have been major efforts in testing these implementations, namely frankencerts and mucerts, which provide new ways to generate test certificates which are likely to reveal errors in the implementations of X.509 validation logic. However, it remains a significant challenge to generate effective test certificates. In this paper, we explore the applicability of a prominent combinatorial method, namely combinatorial testing, for testing of X.509 certificates. We demonstrate that combinatorial testing provides the theoretical guarantees for revealing errors in the certificate validation logic of SSL/TLS implementations. Our findings indicate that the introduced combinatorial testing constructs, coveringcerts, compare favorably to existing testing methods by encapsulating the semantics of the validation logic in the input model and employing combinatorial strategies that significantly reduce the number of tests needed. Besides the foundations of our approach, we also report on experiments that indicate its practical use.

In the testing of a software component-based robot system, generating test cases for the system is a time-consuming and difficult task that requires the combining of test data. This paper proposes an adaptive combinatorial testing method which is based on the input/output relationship among components and which automatically generates the test cases for the system. The proposed algorithm first generates an input/output relationship graph in order to analyze the input/output relationship of the system. It then generates the reduced set of test cases according to the analyzed type of input/output relationship. To validate the proposed algorithm some comparisons are given in terms of the time complexity and the number of test cases.

As a rapidly emerging technology, Web services offer a brand-new mechanism for program-to-program interactions over the Internet. Inevitably, this programming pattern brings great challenge to the improvement of the quality and reliability of Web services. In the paper, a two-level testing framework for WSBS is proposed through deeply analyzing the architecture of WSBS and black-box character of Web service unit. In service unit level, combinatorial testing method is used to ensure single service's reliability through extracting interface information from WSDL file. In system level, BPEL specification is converted into state diagram at first, and then state transition-based test cases generation algorithm is presented. In addition, the feasibility and effectiveness of our approach is validated by some examples.

The humanitarian vehicle routing problem with non-routineness of trips

Security issue is becoming increasingly more important in the field of P2P (Peer-to-Peer) technology due to its intrinsic properties, e.g. anonymous, dynamic, open etc. Reputation and trust mechanism is an effective way to handle the complex security problems of P2P network. Lots of prominent systems have been proposed in recent years, the purpose of this paper is to analysis the main properties of these proposals (i.e. architecture of the network, metric, calculation approach, incentive mechanism etc.), to compare some typical systems to bring better understanding of the reputation and trust mechanism, and to give directions to future studies in this area.

Combinatorial testing is a practical software testing approach that has been widely used in practice. Most research and applications of such approach focus on N-way combinatorial testing that provides a minimum coverage of all N-way interactions among factor. However, the strengths of different interactions may not be a fixed integer N, but a variable. Therefore, variable strength combinatorial testing approach is necessary in applications. Existing variable strength combinatorial testing, which allows some interactions have a higher strength than others, has a limitation that such higher-strength interactions must be disjoint. To avoid such a limitation, an improved variable strength combinatorial testing approach, which makes a more sufficient consideration on actual interaction relationship, is proposed in this article. Furthermore, two greedy heuristic algorithms are also proposed to generate combinatorial test suite. Compared to some existing algorithms and tools, the proposed algorithms have advantages on both the execution effectiveness and the optimality of generated test suite. Experimental results can prove such advantages.

SummaryIn this paper, we report on applying combinatorial testing to Internet of Things (IoT) home automation hub systems. We detail how to create a dedicated input parameter model of an IoT home automation hub system for use with combinatorial test case generation strategies. Further, we developed an automated test execution framework and two test oracles for evaluation purposes. We applied and evaluated our proposed methodological approach to a real‐world IoT system and analysed the obtained results of various combinatorial test sets with different properties generated based on the derived input model. Additionally, we compare these results to a random testing approach. Our empirical testing evaluations revealed multiple errors in the tested devices and also showed that all considered approaches performed nearly equally well.