Automatic Loop Summarization via Path Dependency Analysis

In software engineering, models are used for many different things. In this paper, we focus on program verification, where we use models to reason about the correctness of systems. There are many different types of program verification techniques which provide different correctness guarantees. We investigate the domain of program verification tools, and present a concise megamodel to distinguish these tools. We also present a data set of almost 400 program verification tools. This data set includes the category of verification tool according to our megamodel, practical information such as input/output format, repository links, and more. The categorisation enables software engineers to find suitable tools, investigate similar alternatives and compare them. We also identify trends for each level in our megamodel based on the categorisation. Our data set, publicly available at https://doi.org/10.4121/20347950, can be used by software engineers to enter the world of program verification and find a verification tool based on their requirements.

Satisfiability Modulo Theories (SMT) solvers are used in many modern program verification, analysis and testing tools. They owe their scale and efficiency thanks to advances in search algorithms underlying modern SAT solvers and first-order theorem provers. They owe their versatility in software development applications thanks to specialized algorithms supporting theories, such as numbers and algebraic data-types, of relevance for software engineering. This lecture introduces algorithmic principles of SMT solving, taking as basis modern SAT solvers and integration with specialized theory solvers and quantifier reasoning. We detail some of the algorithms used for main theories used in current SMT solvers and survey newer theories and approaches to integrating solvers. The lectures also outline some application scenarios where SMT solvers have found use, including program verification, network analysis, symbolic model checking, test-case generation, and white-box fuzzing.

Services are highly reusable, flexible and loosely coupled, which makes the evolution and the maintenance of composite services more complex. Evolution of BPEL composite service covers changes of processes, bindings and interfaces. In this paper, an approach is proposed to select and generate test cases during the evolution of BPEL composite service. The approach identifies the changes by using control-flow analysis technique and comparing the paths in new composite service version and the old one using extensible BPEL flow graph (or XBFG). Message flow is appended to the control flow so that XBFG can describe the behavior of composite service integrally. The binding and predicate constraint information added in XBFG elements can be used in path selection and test case generation. Theory analysis and case study both show that the approach is effective, and test cases coverage rate is high for the changes of processes, bindings and interfaces.

Software testing is the key to ensuring software quality. Test case generation and software fault localization are two important research objects of testing. Spectrum-based fault localization (SBFL) method is a mainstream dynamic fault localization method, which inputs test cases into the program for execution and collects statement coverage information and execution results. By analysing this information, a fault report can be generated. The performance of SBFL is affected by test case quality. Therefore, this paper proposes a test case generation method based on improved genetic algorithm to assist SBFL. This method uses a small number of initial test cases to execute the program, then uses the XGBoost model to pre-process the spectrum information and generates a list of potential fault statements to guide the iterative process of the genetic algorithm. The experimental results show that the test cases generated using the improved genetic algorithm have certain advantages over the test cases generated by general genetic algorithm and the baseline test cases.

SummaryThis article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Back's action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright © 2014 John Wiley & Sons, Ltd.

Test cases are essential parts to perform software testing as they play very significant role defining the test steps to work. Test cases can be generated either manually or automatically. Creating test cases manually has several disadvantages, such as higher probability in producing inaccurate results. Furthermore, generating test cases manually is ineffective since manual methods will take longer and require high human efforts. Therefore, generating test cases automatically is more preferable to limit the previous problems. Automated testing can also provide more accurate results than manual testing because it is less prone to human errors encouraging software developers to generate test cases automatically. Test cases can be generated from various sources, including UML diagrams. In this paper, we propose a method to generate test cases automatically by utilising depth-first search algorithms. The UML diagram used in this study is an activity diagram. This paper also provides a method to parse activity diagram that is created using the visual paradigm application that has been converted into an XML format. Our method is able to parse and convert the XML format into graph or tree form. Using our proposed method, a test case search will be carried out using the depth-first search on the tree form to generate test cases. Lastly, we also evaluate our method by comparing the generated test cases with cyclomatic complexity, the method will be performed manually and automatically to validate the results of many scenarios. Our result showed that our method could generate the same scenario (100% coverage) as the manual validation method.

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

Maximum coverage with minimum testing time is the main objective of a test case generation activity which leads to a multi-objective problem. Search-Based Testing (SBT) technique is a demanding research area for test case generation. Researchers have applied various metaheuristic (searching) algorithms to generate efficient and effective test cases in many research works. Out of these existing search-based algorithms, Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) are the most widely used algorithms in automatic test case generation. In this paper, a test case generation approach is proposed using Cuckoo Search (CS) algorithm. CS has a controlling feature, Levy flights, which makes it more efficient in searching the best candidate solution. It helps to generate efficient test cases in terms of code coverage and execution time. In our proposed method, test cases are generated based on path coverage criteria. Fitness of a test case is evaluated using branch distance and approximation level combined functions. The result is compared with PSO and with its variant Adaptive PSO (APSO). The experimental result shows that both the algorithms give nearly equal to the same result. Though the results are nearly equal, the implementation of CS is simple as it requires only one parameter to be tuned.

As we know, software testing is an important part of software development lifecycle. More than 50% of the whole system development work and total cost were spend on the software testing. And it’s estimated that almost 60% of the total test time and cost were spent on the design of test cases. In recent years, the automated testing gradually replaces the traditional manual testing to become an important branch of software testing. In automated testing, the most important thing is to design and generate valid test case automatically. In this paper, In order to reduce the workload of testers on the test case generation, improve the test efficiency and makes test experiences be passed on, we design a common description method for the test case based on ontology; propose a semantic similarity measure method to retrieve the usable test cases from test case library based on WordNet; establish the relationship between the test case and test requirement though rules to modify the retrieved test case; generate the final test case sequence. At last, the practical applicability of the approach is evaluated through an experiment. Keywords-software testing; ontology; test case generation; semantic similarity measure

Software engineers working in industry seldom try to apply formal methods to solve problems. There are various reasons for this. Sometimes these reasons are understandable---the cost of using formal methods does not make economic sense in many contexts. However, formal methods are also often greeted with scepticism. Formal methods are assumed to take too much time, require tools that are too academic, or to be too mathematical to be understood by practice-oriented software engineers. We tested these assumptions by designing a small course around a framework for program verification, aimed at regular computer science students enrolled in a Master's programme. After four lectures and associated exercises, students were given a small verification task where they had to model and verify a real, non-trivial, C function in Why3. A significant majority of students managed to prove a non-trivial functional specification of this C function in the time allotted, and many also pointed out inherent flaws of this function discovered during formalization. Participants reported no major difficulties or mental hurdles in learning Why3, and considered its approach to be appropriate for selected components of safety-critical software. While formal verification tools such as Why3 still have lots of room for improvement, this experience shows that in a short amount of time, software engineers can be taught to use a program verification tool, and obtain usable results without being fully proficient in it. We further recommend that courses on formal methods should also let students explore these as techniques to be applied, instead of only focusing on the theory behind them, as we expect this to gradually lower the barrier to wider acceptance.

Testing of product is perform to discover or detect the errors and defects in the developed system. But testing is usually time consuming especially when complex projects are canvass. Testing of a product lead off with generation of test cases. The Test case generation are based on three parts coding, design and specification. The Specification based testing deals with generation of test cases from the functional requirements. The proposed system deals with automatic generation of test cases from functional requirement using Natural Language Processing (NLP). The proposed system constructs test cases based on keywords in context from functional requirement of Software Requirement Specification document. The proposed system is beneficial as it can automatically analyze the functional requirement from Software Requirement Specification in order to extract test cases for testing. The goal of the proposed system is to reduce effort and time consumed by software tester to test the product.

Testing is an important and challenging part of software development and its effectiveness depends on the quality of test cases. However, there exists no means of measuring quality of tests developed for GPU programs and as a result, no test case generation techniques for GPU programs aiming at high test effectiveness. Existing criteria for sequential and multithreaded CPU programs cannot be directly applied to GPU programs as GPU follows a completely different memory and execution model. We surveyed existing work on GPU program verification and bug fixes of open source GPU programs. Based on our findings, we define barrier, branch and loop coverage criteria and propose a set of mutation operators to measure fault finding capabilities of test cases. CLTestCheck, a framework for measuring quality of tests developed for GPU programs by code coverage analysis, fault seeding and work-group schedule amplification has been developed and evaluated using industry standard benchmarks. Experiments show that the framework is able to automatically measure test effectiveness and reveal unusual behaviours. Our planned work includes data flow coverage adopted for GPU programs to probe the underlying cause of unusual kernel behaviours and a more comprehensive work-group scheduler. We also plan to design and develop an automatic test case generator aiming at generating high quality test suites for GPU programs.

Extended finite state machine (EFSM) is among the most popular models for model-based testing. However, automated test case generation on EFSM models is still a challenging task since an EFSM model may contains infeasible paths. This paper proposed a novel approach (ATGEM) to generate test case and construct oracle information from EFSM automatically. To address the infeasible problem, a metric based on data flow analysis is presented to predict the infeasible probability so as to bypass the infeasible paths as far as possible and improve the test case generation efficiency. Afterwards, an executable EFSM model is developed to obtain runtime feedback information as a fitness function in order to generate test data and construct oracle information automatically. This approach, which can generate various types data and has a wide range of applications, combines static analysis and dynamic analysis aims to find a preferable feasible path subset to generate test cases and meet adequacy coverage criteria. The experimental results on several EFSM models show that test case generation method and path feasibility metric have good effectiveness. Utilizing path feasibility metric can speed up the process of test case generation greatly, and ATGEM is more efficient than existing method.

This paper presents work on generation of specification-driven test cases based on quasirandom (low-discrepancy) sequences instead of pseudorandom numbers. This approach is novel in software testing. This enhanced uniformity of quasirandom sequences leads to faster generation of test cases covering all possibilities. We demonstrate by examples that quasirandom sequences can be a viable alternative to pseudorandom numbers in generating test cases. In this paper, we present a method that can generate test cases from a decision table specification more effectively via quasirandom numbers. Analysis of a simple problem in this paper shows that quasirandom sequences achieve better data than pseudorandom numbers, and have the potential to converge faster and so reduce the computational burden. The use of different quasirandom sequences for generating test cases is presented in this paper.

Generating test cases from the formal specification description VDM++, which is a method for disambiguating specifications, is time-consuming and labor intensive.Therefore, our laboratory has developed BWDM, a tool that automatically generates test cases from VDM++ specifications.However, existing BWDM has problems that it only supports integer types and cannot generate test cases for operation and function definitions including recursive structure.Therefore, in order to improve the usefulness of BWDM, this paper extends BWDM to solve the above problems.Consequently, it has confirmed that the use of extended BWDM can reduce the test case generation time compared to manual test case generation.