Reducing the Costs of Bounded-Exhaustive Testing

Despite recent progress in automated test generation research, significant challenges remain for applying these techniques on large-scale software systems. These systems under test often require structurally complex test inputs within a large input domain. It is challenging to automatically generate a reasonable number of tests that are both legal and behaviorally-diverse to exercise these systems. Constraint-based test generation is an effective approach for generating structurally complex inputs for systematic testing. While this approach can typically generate large numbers of tests, it has limited scalability – tests generated are usually only up to a small bound on input size. Combinatorial test generation, e.g., pair-wise testing, is a more scalable approach but is challenging to apply on commercial software systems that require complex input structures that cannot be formed by using arbitrary combinations. This paper introduces comKorat, which unifies constraint-based generation of structurally complex tests with combinatorial testing. Specifically, comKorat integrates Korat and ACTS test generators to generate test suites for large scale software systems with structurally complex test inputs. We have successfully applied comKorat on four software applications developed at eBay and Yahoo!. The experimental results show that comKorat outperforms existing solutions in execution time and test coverage. Furthermore, comKorat found a total of 59 previously unknown bugs in the four applications.

A software product line is a set of software systems that share some common features. Several recent works have been reported that apply combinatorial testing, a very effective testing strategy, to software product lines. A unique challenge in these efforts is dealing with a potentially large number of constraints among different features. In this paper, we propose a novel constraint-handling strategy that uses minimum invalid tuples (MITs) as an alternative to traditional constraint solvers. Our approach systematically derives all MITs from a software product line, and uses them to quickly determine the validity of a test configuration during test generation. We implemented a test generation research tool called LOOKUP that integrates the proposed constraint-handling strategy with a general test generation algorithm called IPOG-C. Experimental results show that LOOKUP performs considerably better than two existing test generation tools in terms of test size and execution time.

Cloud-based elastic computing systems dynamically change their resources allocation to provide consistent quality of service and minimal usage of resources in the face of workload fluctuations. As elastic systems are increasingly adopted to implement business critical functions in a cost-efficient way, their reliability is becoming a key concern for developers. Without proper testing, cloud-based systems might fail to provide the required functionalities with the expected service level and costs. Using system testing techniques, developers can expose problems that escaped the previous quality assurance activities and have a last chance to fix bugs before releasing the system in production. System testing of cloud-based systems accounts for a series of complex and time demanding activities, from the deployment and configuration of the elastic system, to the execution of synthetic clients, and the collection and persistence of execution data. Furthermore, clouds enable parallel executions of the same elastic system that can reduce the overall test execution time. However, manually managing the concurrent testing of multiple system instances might quickly overwhelm developers' capabilities, and automatic support for test generation, system test execution, and management of execution data is needed. In this demo we showcase AUToCLES, our tool for automatic testing of cloud-based elastic systems. Given specifications of the test suite and the system under test, AUToCLES implements testing as a service (TaaS): It automatically instantiates the SUT, configures the testing scaffoldings, and automatically executes test suites. If required, AUToCLES can generate new test inputs. Designers can inspect executions both during and after the tests.

The RC-5 algorithm is an algorithm with the encryption method using the symmetric method and processing in the form of a cipher block, the same keywords are used for the encryption and decryption processes. The testing and analysis phase consists of a). Testing the execution time of the key generation process (set up key) b). Testing the success of the encryption and decryption process on the client-server integration method c). Testing the success of the encryption and decryption process using the file operation method d). Comparison of overhead/file size before and after passing through the system. The testing phase and execution time analysis are carried out when using the file operation method where the test will be divided into 3 parts, namely the execution time for the key setup process, the encryption process, and the decryption process. Testing the success of the encryption and decryption process on the client-server integration method is divided into 2 parts, namely testing the encryption process and the decryption process. In testing the encryption process, the characters that will be sent to the server will be encrypted first with a key to becoming a ciphertext in the form of integer data. The test results obtained are 1). The execution time for key generation (set up key) is very fast, around 9-10 ns. 2). In the encryption and decryption process, the execution time depends on the size of the plaintext file. The larger the plaintext file size, the longer the execution time. Where for a file with a size of 1 kb requires an execution time of 50.6 ms in the encryption process. 3). There is no difference between the size of the file before entering encryption and after decryption. Where for files with a size of 500 bytes before entering the encryption process, a file of 500 bytes will be generated from the decryption process.

State justification is a time-consuming operation in test generation for sequential circuits. In this paper, we present a technique to rapidly identify state elements (hip-hops) that are either difficult to set or unsettable. This is achieved by performing test generation on certain transformed circuits to identify state elements that are not settable to specific logic values. Two applications that benefit from this identification are sequential circuit test generation and partial scan design. The knowledge of the state space is shown to be useful in creating early backtracks in deterministic test generation. Partial scan selection is also shown to benefit from the knowledge of the difficult-to-set hip-hops. Experiments on the ISCAS89 circuits are presented to show the reduction in time for test generation and the improvements in the testability of the resulting partial scan circuits.

Software testing is an important stage of the software development life cycle. However, the test execution is time-consuming and costly. To accelerate the test execution, researchers have applied several methods to run the testing in parallel. One method of parallelizing the test execution is by using a GPU machine to distribute test case inputs among several threads running in parallel. However, when testing a program on a GPU, different test case inputs could have different control flow paths. This leads to branch divergence which may increase the execution time of testing. In addition, transferring data between host and device may increase the execution time of the test execution when data are allocated statically instead of dynamically. Also, different data sizes from different test case inputs may increase the number of memory transactions. To address these challenges, three studies were conducted. The first study investigated three programming models: CUDA Unified Memory, CUDA Non-Unified Memory, and OpenMP GPU Offloading to parallelize the test execution. This study utilized 11 benchmarks and parallelized their test suites by using these models. The evaluation was based on their performance in terms of execution time. The results showed that using CUDA Unified Memory to implement a test suite for GPU was the easiest and most productive programming model among the other two models. The second study discussed three challenges (control flow, memory access pattern, and input size of a program) when parallelizing test execution on two major computing platforms: multi-core CPU and GPU. It analyzed 14 programs from different domains to evaluate parallelizing test execution on the two platforms. The results showed that the cache size of a GPU and branch divergence significantly increase the execution time of many test suites of programs. The last study proposed a novel algorithm to minimize the branch divergence when testing an application on a GPU. Essentially, the algorithm arranges test inputs based on the warp size of a GPU. Test inputs that have similar control flow paths are grouped within the same warp to be executed in parallel. The validation and evaluation were conducted on six different domains of benchmarks (57 programs in total). The results showed that the algorithm speeds up the testing execution up to 3.8x and improves the warp execution efficiency up to 15x. Therefore, the algorithm reduces the branch divergence by minimizing the number of inactive threads per warp.

We describe the tools and interfaces created by the AGEDIS project, a European Commission sponsored project for the creation of a methodology and tools for automated model driven test generation and execution for distributed systems. The project includes an integrated environment for modeling, test generation, test execution, and other test related activities. The tools support a model based testing methodology that features a large degree of automation and also includes a feedback loop integrating coverage and defect analysis tools with the test generator and execution framework. Prototypes of the tools have been tried in industrial settings providing important feedback for the creation of the next generation of tools in this area.

We describe the tools and interfaces created by the AGEDIS project, a European Commission sponsored project for the creation of a methodology and tools for automated model driven test generation and execution for distributed systems. The project includes an integrated environment for modeling, test generation, test execution, and other test related activities. The tools support a model based testing methodology that features a large degree of automation and also includes a feedback loop integrating coverage and defect analysis tools with the test generator and execution framework. Prototypes of the tools have been tried in industrial settings providing important feedback for the creation of the next generation of tools in this area.

We describe the tools and interfaces created by the AGEDIS project, a European Commission sponsored project for the creation of a methodology and tools for automated model driven test generation and execution for distributed systems. The project includes an integrated environment for modeling, test generation, test execution, and other test related activities. The tools support a UML based testing methodology that features a large degree of automation and also includes a feedback loop integrating coverage and defect analysis tools with the test generator and execution framework.

A hybrid DFT method is proposed to reduce the hardware penalty of traditional DFT methods and test generation time. It takes advantages of both traditional DFT methods like scan and DFT methods which control clocks for testability. The hardware scheme and test generation algorithm are presented. Test generation results for ISCAS '89 circuits have been generated and showed an improvement in test generation time and fault coverage.

Cost-effective test case generation with the hyper-heuristic for software product line testing

Test generation has proven to provide an effective way of identifying programming errors. Unfortunately, current test generation techniques are challenged by higher-order functions in dynamic languages, such as JavaScript functions that receive callbacks. In particular, existing test generators suffer from the unavailability of statically known type signatures, do not provide functions or provide only trivial functions as inputs, and ignore callbacks triggered by the code under test. This paper presents LambdaTester, a novel test generator that addresses the specific problems posed by higher-order functions in dynamic languages. The approach automatically infers at what argument position a method under test expects a callback, generates and iteratively improves callback functions given as input to this method, and uses novel test oracles that check whether and how callback functions are invoked. We apply LambdaTester to test 43 higher-order functions taken from 13 popular JavaScript libraries. The approach detects unexpected behavior in 12 of the 13 libraries, many of which are missed by a state-of-the-art test generator.

Automatic test pattern generation on parallel processors

Deep learning (DL) frameworks are essential to DL-based software systems, and framework bugs may lead to substantial disasters, thus requiring effective testing. Researchers adopt DL models or single interfaces as test inputs and analyze their execution results to detect bugs. However, floating-point errors, inherent randomness, and the complexity of test inputs make it challenging to analyze execution results effectively, leading to existing methods suffering from a lack of suitable test oracles. Some researchers utilize metamorphic testing to tackle this challenge. They design Metamorphic Relations (MRs) based on input data and parameter settings of a single framework interface to generate equivalent test inputs, ensuring consistent execution results between original and generated test inputs. Despite their promising effectiveness, they still face certain limitations. (1) Existing MRs overlook structural complexity, limiting test input diversity. (2) Existing MRs focus on limited interfaces, which limits generalization and necessitates additional adaptations. (3) Their detected bugs are related to the result consistency of single interfaces and far from those exposed in multi-interface combinations and runtime metrics (e.g., resource usage). To address these limitations, we propose ModelMeta, a model-level metamorphic testing method for DL frameworks with four MRs focused on the structure characteristics of DL models. ModelMeta augments seed models with diverse interface combinations to generate test inputs with consistent outputs, guided by the QR-DQN strategy. It then detects bugs through fine-grained analysis of training loss/gradients, memory/GPU usage, and execution time. We evaluate the effectiveness of ModelMeta on three popular DL frameworks (i.e., MindSpore, PyTorch, and ONNX) with 17 DL models from ten real-world tasks ranging from image classification to object detection. Results demonstrate that ModelMeta outperforms state-of-the-art baselines from the perspective of test coverage and diversity of generated test inputs. Regarding bug detection, ModelMeta has identified 31 new bugs, of which 27 have been confirmed, and 11 have been fixed. Among them, seven bugs existing methods cannot detect, i.e., five wrong resource usage bugs and two low-efficiency bugs. These results demonstrate the practicality of our method.

In online model-based testing, test execution is interleaved with test generation. Test cases should be generated and executed with minimal delay, while still achieving targeted coverage criteria quickly. Extensive model analysis in such case is not possible as any delays in choosing the next step will immediately impact the response times of test execution. The algorithms thus need to be as fast as possible, where a limiting factor is the available computing power. Experts working on the test models used for the generation often need to be able to quickly edit the models, generate test cases, and use the feedback to further evolve the models. Reserving large-scale computing resources while editing the model is unnecessary, but performing the analysis on them for test generation can improve the execution response time significantly. In this paper, we present an approach and algorithm for distributing the online test generation analysis part concurrently over the network, while enabling the expert to work on the models and execute the test cases locally at the same time.