Test case generation, selection and coverage from natural language

Services are highly reusable, flexible and loosely coupled, which makes the evolution and the maintenance of composite services more complex. Evolution of BPEL composite service covers changes of processes, bindings and interfaces. In this paper, an approach is proposed to select and generate test cases during the evolution of BPEL composite service. The approach identifies the changes by using control-flow analysis technique and comparing the paths in new composite service version and the old one using extensible BPEL flow graph (or XBFG). Message flow is appended to the control flow so that XBFG can describe the behavior of composite service integrally. The binding and predicate constraint information added in XBFG elements can be used in path selection and test case generation. Theory analysis and case study both show that the approach is effective, and test cases coverage rate is high for the changes of processes, bindings and interfaces.

This paper adresses the generation of test cases for testing the conformance of a reactive black-box implementation with respect to its specification. We aim at extending the principles and algorithms of model-based testing for recursive interprocedural specifications that can be modeled by Push-Down Systems (PDS). Such specifications may be more compact than non-recursive ones and are more expressive.The generated test cases are selected according to a test purpose, a (set of) scenario of interest that one wants to observe during test execution. The test generation method we propose in this paper is based on program transformations and a coreachability analysis, which allows to decide whether and how the test purpose can still be satisfied. However, despite the possibility to perform an exact analysis, the inability of test cases to inspect their own stack prevents it from using fully the coreachability information. We discuss this partial observation problem, its consequences, and how to minimize its impact.KeywordsTest PurposeLabel Transition SystemProgram TransformationGenerate Test CaseTest ExecutionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Cost-effective test case generation with the hyper-heuristic for software product line testing

In this paper, we present an automatic grammar-based test generation approach which takes a symbolic grammar as input, requires zero control input from users, and produces well-distributed test cases. Our approach utilizes a novel dynamic stochastic model where each variable is associated with a tuple of probability distributions, which are dynamically adjusted along the derivation. The adjustment is based on a tabling strategy to keep track of the recursion of each grammar variable. We further present a test generation coverage tree illustrating the distribution of generated test cases and their detailed derivations, more importantly, it provides various implicit balance control mechanisms. We implemented this approach in a Java-based system, named Gena. Experimental results demonstrate the effectiveness of our test generation approach and show the balanced distribution of generated test cases over grammatical structures.KeywordsTest GenerationCoverage TreeProduction RuleSoftware Product LineGenerate Test CaseThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

Embedded systems have become increasingly connected and communicate with each other, forming large-scaled and complicated network systems. To make their design and testing more reliable and robust, this paper proposes a formal specification language calledSENSand aSENS-based automatic test generation tool calledTGSENS. Our approach is summarized as follows: (1) A user describes requirements of target embedded network systems by logical property-based constraints usingSENS. (2) GivenSENSspecifications, test cases are automatically generated using a SAT-based solver. Filtering mechanisms to select efficient test cases are also available in our tool. (3) In addition, given a testing goal by the user, test sequences are automatically extracted from exhaustive test cases. We’ve implemented our approach and conducted several experiments on practical case studies. Through the experiments, we confirmed the efficiency of our approach in design and test generation of real embedded air-conditioning network systems.

With Internet applications spreading like wildfire, software testing is challenged with new topics related to the distributed nature of Web applications. We apply code based testing techniques to the testing of Web applications, specifically Java programs. Source code based automatic test generation is difficult because most previous methods use constraint satisfaction models as a solution, which is an NP complete problem [M. J. Gallagher et al. (1997)]. We present a method of guiding users through test case generations. Instead of automating the entire procedure, our method aims at generating a framework of test cases and providing instructions for users to instantiate the framework into actual executable test cases. An early experimental study of this method shows the effectiveness of this method.

Maximum coverage with minimum testing time is the main objective of a test case generation activity which leads to a multi-objective problem. Search-Based Testing (SBT) technique is a demanding research area for test case generation. Researchers have applied various metaheuristic (searching) algorithms to generate efficient and effective test cases in many research works. Out of these existing search-based algorithms, Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) are the most widely used algorithms in automatic test case generation. In this paper, a test case generation approach is proposed using Cuckoo Search (CS) algorithm. CS has a controlling feature, Levy flights, which makes it more efficient in searching the best candidate solution. It helps to generate efficient test cases in terms of code coverage and execution time. In our proposed method, test cases are generated based on path coverage criteria. Fitness of a test case is evaluated using branch distance and approximation level combined functions. The result is compared with PSO and with its variant Adaptive PSO (APSO). The experimental result shows that both the algorithms give nearly equal to the same result. Though the results are nearly equal, the implementation of CS is simple as it requires only one parameter to be tuned.

Controlled Natural Languages (CNLs) have many applications including document authoring, automatic reasoning on texts and reliable machine translation, but their application is not limited to these areas. We explore a new application area of CNLs, the use of CNLs in computer-assisted language learning. In this paper we present a a web application for language learning using CNLs as well as a detailed description of the properties of the family of CNLs it uses.

Natural language (NL) is pervasive in software requirements specifications (SRSs). However, despite its popularity and widespread use, NL is highly prone to quality issues such as vagueness, ambiguity, and incompleteness. Controlled natural languages (CNLs) have been proposed as a way to prevent quality problems in requirements documents, while maintaining the flexibility to write and communicate requirements in an intuitive and universally understood manner. In collaboration with an industrial partner from the financial domain, we systematically develop and evaluate a CNL, named Rimay, intended at helping analysts write functional requirements. We rely on Grounded Theory for building Rimay and follow well-known guidelines for conducting and reporting industrial case study research. Our main contributions are: (1) a qualitative methodology to systematically define a CNL for functional requirements; this methodology is intended to be general for use across information-system domains, (2) a CNL grammar to represent functional requirements; this grammar is derived from our experience in the financial domain, but should be applicable, possibly with adaptations, to other information-system domains, and (3) an empirical evaluation of our CNL (Rimay) through an industrial case study. Our contributions draw on 15 representative SRSs, collectively containing 3215 NL requirements statements from the financial domain. Our evaluation shows that Rimay is expressive enough to capture, on average, 88% (405 out of 460) of the NL requirements statements in four previously unseen SRSs from the financial domain.

On the order of test goals in specification-based testing

In online model-based testing, test execution is interleaved with test generation. Test cases should be generated and executed with minimal delay, while still achieving targeted coverage criteria quickly. Extensive model analysis in such case is not possible as any delays in choosing the next step will immediately impact the response times of test execution. The algorithms thus need to be as fast as possible, where a limiting factor is the available computing power. Experts working on the test models used for the generation often need to be able to quickly edit the models, generate test cases, and use the feedback to further evolve the models. Reserving large-scale computing resources while editing the model is unnecessary, but performing the analysis on them for test generation can improve the execution response time significantly. In this paper, we present an approach and algorithm for distributing the online test generation analysis part concurrently over the network, while enabling the expert to work on the models and execute the test cases locally at the same time.

Model-Based Testing, the task of generating test inputs and oracles from a test model, has been successfully applied in the context of safety-critical real time systems. As these systems grow in complexity, test-models, designed to reflect the systems behaviour, will grow too. Currently testers face situations where test-models are too complex for present test generators. In this paper, we outline a software tool for the evaluation of the scalability of a combination of approaches for model-based test generation. We chose Networks of Timed Automata (NTA) as the modeling formalism because real-time properties can be specified and the semantics are well-defined. However, the tool input is given as a restricted UML statechart which is internally transformed. We expect this to increase industrial acceptance. The tool will provide the selection, parametrization and generation of a metaheuristic algorithm. The aim is to support test model specific generation algorithms. A simulator for NTAs will enable the metaheuristic to search for test goals in the model. For better performance, it will have an advanced parallelisation. Furthermore, input models will be used for search space reduction for even faster test case generation. The proposed approach allows the inclusion of an oracle generator that is able to provide expected outputs; this enables conformance checking between test models and systems under test. We plan to implement the outlined tool to enable test case generation even for models that are beyond the scope of currently available generators.

BZ‐TESTING‐TOOLS (BZ‐TT) is a tool set for automated test case generation from B and Z specifications. BZ‐TT uses boundary and cause–effect testing on the basis of the formal model. It has been used and validated on several industrial applications in the domain of critical software, particularly smart card and transport systems. This paper presents the test coverage criteria supported by BZ‐TT. On the one hand, these correspond to various classical structural coverage criteria, but specialized to the case of B abstract machines. The paper gives algorithms for these in Prolog. On the other hand, BZ‐TT introduces new coverage criteria for complex data structures, based on boundary analysis: this paper defines weak and strong state‐boundary coverage, input‐boundary coverage and output‐boundary coverage. Finally, the paper describes how BZ‐TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable). Copyright © 2004 John Wiley & Sons, Ltd.

Software systems are not considered complete unless properly tested and verified. In existing literature, a growing interest on establishment of automated testing techniques has been observed. However, tedious manual process of test case generation largely depends upon domain knowledge and formalized representation of user requirements. The advent of semantic web engineering has led the foundation for developing ontologies as a mean to express information and knowledge semantics regarding particular domain efficiently. In software testing, ontologies can be significantly helpful to automate testing phase as they encode domain knowledge in machine interpretable format. We have proposed automatic test case generation framework that involves ontology-based requirement specification and learning based methods for conducting black box testing. Our approach integrates knowledge-based system (ontology) with learning-based testing algorithm to automate: generation of test cases, test execution and test verdict construction. Proposed framework includes, requirement ontology to formalize requirement specification, Dialogue Manager that enables selection of available test cases and Learning Based Testing to generate counter examples of test cases through system learning. The contribution of this paper is to enable 1) requirement elicitation and specification using ontologies 2) test data selection from existing ontologies and 3) automatic test case generation from existing test cases. To represent the applicability of this research, ontology for requirement elicitation and specification is developed. Framework proposed in this research paper is an effort to provide software testing tools to save time, cost and efforts during test design phase.