Controller-Centric Combinatorial Wrap-Around Interaction Testing to Evaluate a Stateful PCE-Based Transport Network Architecture

An active stateful Path Computation Element (PCE) is capable of computing as well as controlling via Path Computation Element Communication Protocol (PCEP) Multiprotocol Label Switching Traffic Engineering (MPLS-TE) Label Switched Paths (LSPs). Furthermore, it is also possible for an active stateful PCE to create, maintain, and delete LSPs. This document defines the PCEP extension to associate two or more LSPs to provide end-to-end path protection.

The process of regression testing is generally done under critical time and resource constraints. In the existing approaches, Test Case Prioritization (TCP), Test Case Selection (TCS), and Test Suite Reduction (TSR) have been treated as different optimization problems. This is based on simple heuristics that do not use valuable information that can be extracted from the program under test. Moreover, they do not allow for making internally consistent budget-conscious decisions. This paper introduces a unified approach that combines various variants of Graph Neural Networks (GNNs) and Reinforcement Learning (RL) for the solutions of TCP, TCS, and TSR. In our proposal, the representation of regression artifacts such as test cases, code entities, and faults is used as nodes in the typed graph, while the edges are used to show the relations between these artifacts. The relation-aware GNN is used for generating test case embeddings, reflecting the distance between the test cases and the changed areas as well as the areas known to have issues in the past. The actor-critic RL agent uses these test case embeddings and the budget to decide whether to run, skip, or discard test cases. The performance of our proposal outstrips coverage-based heuristics, history-based ranking, a GA-based search, an enhanced QPSO, and two learning-based ablations, as validated by our experiments on four Java projects from Defects4J. In terms of various programs and budgets, using GNN–RL has led to an improvement in the Average Percentage of Faults Detected (APFD) and Cost-cognizant Average Percentage of Faults Detected (APFDc) by approximately 6–9 percentage points on lower budgets. In addition, it is possible to obtain a 42% suite reduction and a 48% cost reduction while still retaining 98% of all fault-revealing tests in the final suite. The results show that it is possible to obtain promising solutions for adaptive regression testing using budget-aware reinforcement learning and graph-based representation learning.

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

The path computation element (PCE) architecture was originally proposed with a stateless condition, i.e., considering only network reserved resources during constraint-based path computations. More recently, a stateful architecture was introduced to additionally maintain the state of computed and established label switch paths (LSPs). Furthermore, the PCE architecture evolved to active functionality, enabling the PCE to directly issue recommendations to the network. In this study, we present and discuss several use cases where the active stateful architecture can provide some benefits. They include impairment-aware path computations in the context of multirate optical networks, recovery solutions, global defragmentation, and dynamic LSP adaptations. The latter use case is then specifically demonstrated in a network testbed including a flexigrid optical network operated with a multicarrier 1 Tb/s transmission with coherent detection. Novel advanced digital signal processing (DSP) monitoring functionalities are introduced and experimentally demonstrated. These monitoring functionalities are utilized to trigger a new hitless dynamic adaptation technique operating on the applied low-density parity check (LDPC) transmitted coding. The technique has been successfully demonstrated to increase transmission robustness upon impairment degradation, such that no traffic disruption is experienced. Moreover, to accommodate the LSP coding adaptation, network reconfiguration has been performed, successfully driven by the PCE thanks to the active functionality.

Combinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in practical applications (e.g. limited testing resources), for example in combinatorial interaction regression testing, prioritized interaction test suites (called interaction test sequences) are often employed. Consequently, many strategies have been proposed to guide the interaction test suite prioritization. It is, therefore, important to be able to evaluate the different interaction test sequences that have been created by different strategies. A well-known metric is the Average Percentage of Combinatorial Coverage (shortly APCCλ), which assesses the rate of interaction coverage of a strength λ (level of interaction among parameters) covered by a given interaction test sequence S. However, APCCλ has two drawbacks: firstly, it has two requirements (that all test cases in S be executed, and that all possible λ-wise parameter value combinations be covered by S); and secondly, it can only use a single strength λ (rather than multiple strengths) to evaluate the interaction test sequence - which means that it is not a comprehensive evaluation. To overcome the first drawback, we propose an enhanced metric Normalized APCCλ (NAPCC) to replace the APCCλ Additionally, to overcome the second drawback, we propose three new metrics: the Average Percentage of Strengths Satisfied (APSS); the Average Percentage of Weighted Multiple Interaction Coverage (APWMIC); and the Normalized APWMIC (NAPWMIC). These metrics comprehensively assess a given interaction test sequence by considering different interaction coverage at different strengths. Empirical studies show that the proposed metrics can be used to distinguish different interaction test sequences, and hence can be used to compare different test prioritization strategies.

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Numerous commercial electronic devices such as computers and mobile phones are composed of embedded systems. In general, these systems present complex functionalities, making the process of developing and testing a challenging task for engineers and designers. Testing is essential for validating any embedded device in order to guarantee that their features are operational and reliable. Improved testing methods could increase the reliability and quality of the devices. In this paper, a test methodology for evaluating commercial embedded devices is proposed. The proposed approach is based on the use of historical knowledge acquired by analyzing and modeling a growing test database composed by information on prior regular test routines. This approach aims to track the similarities among test cases applied in past products in order to optimize the analysis of test cases necessary for the evaluation of recent products that will be submitted to testing processes. A comparative case study was conducted involving the proposed test methodology and a standard test methodology based on test cases in a test routine involving embedded devices produced by a company in the energy solutions business. The empirical results showed a significant improvement in the efficiency of failure and defect detection in the embedded devices, allowing improved management of the processes of development, test, and production.

Low power circuits have become a necessary part in modern designs. Retention flip-flop is one of the most important components in low power designs. Conventional test methodologies are not sufficient to test the retention flip-flop thoroughly. This paper presents four new fault models and the testing of retention flip-flop. The four fault models are awake-mode stuck-at fault, sleep-mode stuck-at fault, awake-mode transition fault, and sleep-mode transition fault. The four faults model the defects that affect the retained value, wakeup time, and sleep time of retention flip-flops. Based on the new fault models, test patterns for retention flip-flop can be easily generated by current automatic test pattern generation tools. The proposed test methodology is validated by performing experiments on ISCAS'89 benchmark circuits and some realistic industrial low power designs. The experimental results show that the faults of retention flip-flops can be easily detected by our method and the average fault coverage is higher than 98%. The fault coverage of conventional single stuck-at fault and transition fault test can be increased by more than 1%.

In this article, we leverage the power of machine learning algorithms to propose a test methodology for millimeter-wave (mm-wave) integrated circuits. The proposed test strategy is based on identifying the main process degradation mechanisms in a particular device under test (DUT) and then designing dedicated process monitor circuits to characterize this degradation and infer the DUT performance. The resulting process monitors do not load or couple to any of the DUT nodes, and the methodology can be adapted to any mm-wave device without complex codesign. The proposed test methodology is illustrated on a set of 21 fabricated samples of a 65-GHz PA designed in STMicroelectronics 55-nm CMOS technology.

Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highly-configurable software systems is complicated by the fact that, in many such systems, there are constraints between specific configuration parameters that render certain combinations invalid. Many CIT algorithms lack a mechanism to avoid these. In recent work, automated constraint solving methods have been combined with search-based CIT construction methods to address the constraint problem with promising results. However, these techniques can incur a non-trivial overhead. In this paper, we build upon our previous work to develop a family of greedy CIT sample generation algorithms that exploit calculations made by modern Boolean satisfiability (SAT) solvers to prune the search space of the CIT problem. We perform a comparative evaluation of the cost-effectiveness of these algorithms on four real-world highly-configurable software systems and on a population of synthetic examples that share the characteristics of those systems. In combination our techniques reduce the cost of CIT in the presence of constraints to 30 percent of the cost of widely-used unconstrained CIT methods without sacrificing the quality of the solutions.

Ensuring the quality of Highly Configurable Systems (HCSs) during its evolution and maintenance is challenging. As an HCS evolves, new features are added, changed, or removed, which makes the test case selection for regression testing a difficult task. The use of test traceability can help in this task, but there is a lack of studies exploring the use of trace links for HCS testing. Existing work is usually based on the variability model, which is not always available or updated. Yet, the few existing approaches rely on links between test cases and files/lines of code, limiting the selection to test cases related to file changes, not considering the whole implementation of features, which can be spread in many files other than the changed ones. Considering this limitation, this work presents a test case selection approach, namely FeaTestSel, that links test cases to features using HCS pre-processor directives. Then, the selection of test cases is based on features affected by changes in each commit. In addition to the selected test cases, the approach also produces the following reports to support the test activity: the lines of code that correspond to each feature, the lines exercised by each test case, and the test cases linked to each feature. To validate the approach, we rely on Libssh, a real open-source HCS in constant evolution. By adding the execution time of the approach to the execution time of the selected test cases, we achieved a reduction of approximately ≈50%, in comparison with the retest-all technique. Furthermore, the approach was able to maintain quality by selecting 100% of failed test files. The traceability and reports produced by our approach can also be used for further work by researchers, analysis of the test quality by engineers, or as a source of information for tool builders.

Combinatorial interaction testing (CIT) is an important technique for testing highly configurable software systems with demonstrated effectiveness in practice. The goal of CIT is to generate test cases covering the interactions of configuration options, under certain hard constraints. In this context, constrained covering arrays (CCAs) are frequently used as test cases in CIT. Constrained Covering Array Generation (CCAG) is an NP-hard combinatorial optimization problem, solving which requires an effective method for generating small CCAs. In particular, effectively solving t-way CCAG with t>=4 is even more challenging. Inspired by the success of automated algorithm configuration and automated algorithm selection in solving combinatorial optimization problems, in this paper, we investigate the efficacy of automated algorithm configuration and automated algorithm selection for the CCAG problem, and propose a novel, automated CCAG approach called AutoCCAG. Extensive experiments on public benchmarks show that AutoCCAG can find much smaller-sized CCAs than current state-of-the-art approaches, indicating the effectiveness of AutoCCAG. More encouragingly, to our best knowledge, our paper reports the first results for CCAG with a high coverage strength (i.e., 5-way CCAG) on public benchmarks. Our results demonstrate that AutoCCAG can bring considerable benefits in testing highly configurable software systems.

Software needs modifications and requires revisions regularly. Owing to these revisions, retesting software becomes essential to ensure that the enhancements made, have not affected its bug-free functioning. The time and cost incurred in this process, need to be reduced by the method of test case selection and prioritization. It is observed that many nature-inspired techniques are applied in this area. African Buffalo Optimization is one such approach, applied to regression test selection and prioritization. In this paper, the proposed work explains and proves the applicability of the African Buffalo Optimization approach to test case selection and prioritization. The proposed algorithm converges in polynomial time (O(n2)). In this paper, the empirical evaluation of applying African Buffalo Optimization for test case prioritization is done on sample data set with multiple iterations. An astounding 62.5% drop in size and a 48.57% drop in the runtime of the original test suite were recorded. The obtained results are compared with Ant Colony Optimization. The comparative analysis indicates that African Buffalo Optimization and Ant Colony Optimization exhibit similar fault detection capabilities (80%), and a reduction in the overall execution time and size of the resultant test suite. The results and analysis, hence, advocate and encourages the use of African Buffalo Optimization in the area of test case selection and prioritization.

This work introduces a model-based approach for minimization of test costs for interactive systems. Results known from state-based conformance testing and graph theory are used and extended to construct algorithms for test case generation and selection to cover the behavioral model of the system under test (SUT). The test case selection on the basis of the established model is ruled by an adequacy criterion, which provides a measure of how effective a given set of test cases is in terms of its potential to reveal faults. The approach is specification-oriented; i.e., the underlying model represents the system behavior interacting with the user's actions, that are viewed here as events.

Good test cases are those test case with high failure detecting capability. The effectiveness of testing is relied on the test case selection process. The earlier good test cases being selected, the higher the effectiveness of testing. Partitioning is one of the method to improve the effectiveness of test case selection. Dynamic Partitioning is method that has been proved empirically can improve the effectiveness of random selection. This method is proposed based on the intuition that a successful test case has high probability in detecting failure in the next execution. On the other hand, Addition Branch Coverage method has been proposed to improve the effectiveness of testing. It selects test case if it can add the covered branch during test execution. The empirical results show that the Additional Branch Coverage is the best method compare with othe coverage-based method. In this research, the combination of those two methods, Dynamic Partitioning and Additional Branch coverage is proposed with the hypothesis that this combination will combine the advantages of those two methods, and hence produce better method. The effectiveness of the proposed method has been investigated empirically by comparing it with the Random Testing, Dynamic Partitioning and Additional Branch Coverage. The Random Testing usually used as the benchmark in software testing research. The results show that the new method performed more effective that all other studied methods. The combination of Dynamic Partitioning and Additional Branch Coverage improve the effectiveness of test case selection in term of the number of executed test cases to detect all existing failures.