Evaluating the Impact of Different Testers on Model-based Testing

The Model Driven Architecture (MDA) approach is emerged in the last years as a novel software design methodology for the development of software systems. In this approach the focus of software development is shifted from writing code to modeling. In MDA, developers implement models that are automatically transformed into the target code of the system. Alongside MDA, the Model Driven Testing (MDT) is emerging as a relevant research topic in both industrial and scientific communities. MDT is a methodology where test cases for the system are automatically obtained starting from test models to maximize specific model coverage criteria. Eventually, test cases are executed to verify the system code that is generated through an MDA approach. In this paper, we conduct an exploratory study in order to evaluate the differences that may exist between the model coverage guaranteed by the test cases and the code coverage reached when they are executed on the auto-generated code. Moreover, we identify the main factors that may influence these differences.

Coverage-Guided fuzzing is the type of fuzzing which focuses on the code or branch coverage. It is mainly efficient in detecting buffer overflow. For the code coverage, it is an important parameter of performance evaluation of the Coverage-Guided fuzzing tools, since the higher coverage means higher possibility of bug detection. However, the timeout set for the fuzzing also affects the efficiency, due to the fact that the growth rate of code coverage will gradually become slower as the running time becoming longer. Setting a timeout that is too long cannot help improving the coverage nor the bug detected, and it will be a complete waste of time. Since selecting the fuzzing time might be quite confusing for testers, in this paper, the author decided to discover the relationship of the growth of the coverage and the running time of the. AFL and FuzzFactory were used for the evaluation and the timeout were set as 1 second, 1 minute, 1 hour, 6 hours and 12 hours respectively. The results showed the relationship between fuzzing time and the performance, for all fuzzers and the effect between different fuzzers.

This work aims at defining a procedure and a set of mechanisms able to improve the quality of the code coverage in automated software reverse engineering processes, and specifically in automated GUI-driven testing of Android apps. Existing automated model-based testing techniques, based on reverse engineering, generate test cases which can be executed directly on the software's GUI. We propose to augment the code coverage of these techniques, by exploiting information from patterns, defined at different levels (application design, state-based model, interaction with Android services), and generating additional test cases that may increase the coverage capability of GUI-Ripping based testing technique. The generation of the additional test cases is accomplished by defining an automatable procedure which exploits an existing GUI testing approach and a pattern based approach used in a different context.

Previous works have shown that high-level decision diagrams (HLDD-s) are suitable for system representation for analyzing code coverage metrics. This is due to the fact that HLDD models implicitly represent classical code coverage items, such as statement and branch coverage. However, research on the properties of HLDD-s, which contribute to the accuracy of coverage assessment, is missing. Current paper proposes a set of HLDD manipulations in order to generate diagrams that would allow more stringent code coverage measurement without sacrificing performance, i.e., computation time and memory requirements. The techniques include generation of HLDD-trees from Hardware Description Language (HDL)descriptions and two types of HLDD collapsing methods, which are a generalization of the BDD reduction rules. Experiments on ITC99 benchmark circuits show that the code coverage assessment based on the proposed HLDD manipulation is more stringent than what can be achieved with classical methods. At the same time, the model is well scalable because HLDD generation is terminated in the HDL variables.

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.

SummaryTo deliver reliable software, developers may rely on the fault detection capability of test suites. To evaluate this capability, they can apply code coverage metrics before a software release. However, recent research results have shown that these metrics may not provide a solid basis for this evaluation. Moreover, the fixing of a fault has a cost, and not all faults have the same impact regarding software reliability. In this sense, operational testing aims at assessing parts of the system that are more valuable for users. The goal of this work is to investigate whether traditional code coverage and code coverage merged with operational information can be related to post‐release bug detection. We focus on the scope of proprietary software under continuous delivery. We performed an exploratory case study where code branch and statement coverage metrics were collected for each version of a proprietary software together with real usage data of the system. We then measured the ability to explain the bug‐fixing activity after version release using code coverage levels. We found that traditional statement coverage has a moderate negative correlation with bug‐fixing activities, whereas statement coverage merged with the operational profile has a large negative correlation with higher confidence. Developers can consider operational information as an important factor of influence that should be analysed, among other factors, together with code coverage to assess the fault detection capability of a test suite.

Automotive infotainment systems were getting more and more features in recent years. The usability of their HMIs (human-machine interfaces) has been improved considerably. However the complexity of the HMI software is growing. Testing the HMI became very demanding and time consuming. Because of multiplicity of HMI variants, a better code coverage is a goal for the development process of most manufacturers. Model-based testing is one way to achieve a better code coverage and keep the costs and complexity acceptable. However, the existing research approaches in the area of model-based HMI testing can not satisfy the needs for our testing purposes. In the work a model-based testing approach will be proposed for testing both the logical behavior and the graphical interface of the automotive infotainment HMI. As an important part of the testing approach a test-oriented HMI specification model will be designed. It is a model, which describes the required behavior of the HMI and contains the necessary information for the testing process. Test generation methods and the design of tests will also be proposed. These results can be generally used for testing advanced GUI-driven applications. Specific coverage criteria for infotainment HMIs, methods for automatic test generation and verification of the system behavior are also focuses of the work. The paper introduces the ideas and the goals of our model-based testing approach for infotainment HMIs.

This work describes the working of white-box testing for the Service-Oriented Architecture (SOA) based Application. Now, a days it is very essential to perform code coverage testing to understand the quality of software. This paper deals with the measurement of branch coverage percentage for the BPEL architecture that orchestrates all the services, which are distributed geographically. Here, we are testing the code coverage of BPEL architecture, which actually shows the invocations of services when it is required. This work integrates the existing open source tools, to automate the computation of branch coverage for SOA based application. This paper shows a novel technique for generation of test cases and computing branch coverage through our proposed tool.

Unit testing has a significant role in software development and its impacts depend on the quality of test cases and test data used. To reduce time and effort, unit test generator systems can help automatically generate test cases and test data. However, there is currently no unit test generator for Kotlin programming language even though this language is popularly used for android application developments. In this study, we propose and develop a test generator system that utilizes genetic algorithm (GA) and ANTLR4 parser. GA is used to obtain the most optimal test cases and data for a given Kotlin code. ANTLR4 parser is used to optimize the mutation process in GA so that the mutation process is not totally random. Our model results showed that the average value of code coverage in generated unit tests against instruction coverage is 95.64%, with branch coverage of 76.19% and line coverage of 96.87%. In addition, only two out of eight generated classes produced duplicate test cases with a maximum of one duplication in each class. Therefore, it can be concluded that our optimization with GA on the unit test generator is able to produce unit tests with high code coverage and low duplication.

Code coverage-guided unit test generation (CGTG) and large language model-based test generation (LLMTG) are two principal approaches for the generation of unit tests. Each of these approaches has its inherent advantages and drawbacks. Tests generated by CGTG have been shown to exhibit high code coverage and high executability. However, they lack the capacity to comprehend code intent, which results in an inability to identify deviations between code implementation and design intent (i.e., functional defects). Conversely, although LLMTG demonstrates an advantage in terms of code intent analysis, it is generally characterized by low executability and necessitates iterative debugging. In order to enhance the ability of unit test generation to identify functional defects, a novel framework has been proposed, entitled the intent analysis-guided unit test generation and refinement (IGTG&R) model. The IGTG&R model consists of a two-stage process for test generation. In the first stage, we introduce coverage path entropy to enhance CGTG to achieve high executability and code coverage of test cases. The second stage refines the test cases using LLMs to identify functional defects. We quantify and verify the interference of incorrect code implementation on intent analysis through conditional entropy. In order to reduce this interference, the focal method body is excluded from the code context information during intent analysis. Using these two-stage process, IGTG&R achieves a more profound comprehension of the intent of the code and the identification of functional defects. The IGTG&R model has been demonstrated to achieve an identification rate of functional defects ranging from 65% to 89%, with an execution success rate of 100% and a code coverage rate of 75.8%. This indicates that IGTG&R is superior to the CGTG and LLMTG approaches in multiple aspects.

Software testing is one of the most challenging and arduous phase of software development life cycle (SDLC) which helps in determining the software quality. Code coverage is a widely used testing paradigm, which describes the degree to which the code has been tested. Aim of the current paper is to propose an optimised code coverage algorithm with the help of an emerging technique, i.e., intelligent water drop (IWD). This approach uses dynamic parameters for finding all the optimal paths using basic properties of natural water drops. It proposes how test cases can be considered as an IWD moving on the edges of the control dependency graph for finding the optimal paths. The algorithm guarantees complete code coverage by generating automated test sequences.

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most of these techniques require data on dynamic execution in the form of code coverage information for test cases. However, the collection of dynamic code coverage information on test cases has several associated drawbacks including cost increases and reduction in prioritization precision. In this paper, we propose an approach to prioritizing test cases in the absence of coverage information that operates on Java programs tested under the JUnit framework-an increasingly popular class of systems. Our approach, JUnit test case Prioritization Techniques operating in the Absence of coverage information (JUPTA), analyzes the static call graphs of JUnit test cases and the program under test to estimate the ability of each test case to achieve code coverage, and then schedules the order of these test cases based on those estimates. To evaluate the effectiveness of JUPTA, we conducted an empirical study on 19 versions of four Java programs ranging from 2K-80K lines of code, and compared several variants of JUPTA with three control techniques, and several other existing dynamic coverage-based test case prioritization techniques, assessing the abilities of the techniques to increase the rate of fault detection of test suites. Our results show that the test suites constructed by JUPTA are more effective than those in random and untreated test orders in terms of fault-detection effectiveness. Although the test suites constructed by dynamic coverage-based techniques retain fault-detection effectiveness advantages, the fault-detection effectiveness of the test suites constructed by JUPTA is close to that of the test suites constructed by those techniques, and the fault-detection effectiveness of the test suites constructed by some of JUPTA's variants is better than that of the test suites constructed by several of those techniques.

Model‐based testing (MBT) automates the design and generation of test cases from a model. This process includes model building, test selection criteria, test case generation, and test case execution stages. Current tools support this process at various levels of automation, most of them supporting three out of four stages. Among them is MBT4J, a platform that extends ModelJUnit with several techniques, offering a high level of automation for testing Java applications. In this study, the authors evaluate the efficacy of the MBT4J platform, in terms of the number of test cases generated, errors detected, and coverage metrics. A case study is conducted using two open‐source Java systems from public repositories, and 15 different configurations. MBT4J was able to automatically generate five models from the source code. It was also able to generate up to 2025 unique test cases for one system and up to 1044 for the other, resulting in 167 and 349 failed tests, respectively. Transition and transition pair coverage reached 100% for all models. Code coverage ranged between 72 and 84% for the one system and between 59 and 76% for the other. The study found that Greedy and Random were the most effective testers for finding errors.

Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite’s fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies’ results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.