Code Coverage Criteria for Asynchronous Programs

The argument of correctness in refinement-based formal software design often disregards source code analysis and code generation. To mitigate the risk of errors in these phases, certifications issued by regulation entities demand or recommend testing the generated software using a code coverage criteria. We propose improvements for the BTestBox, a tool for automatic generation of tests for software components developed with the B method. BTestBox supports several code coverage criteria and code generators for different languages. The tool uses a constraint solver to produce tests, thus being able to identify dead code and tautological branching conditions. It also generates reports with different metrics and may be used as an extension to the Atelier B. Our tool performs a double task: first, it acts on the B model, by checking the code coverage. Second, the tool performs the translation of lower level B specifications into programming language code, runs tests and compares their results with the expected output of the test cases. The present version of BTestBox uses parallelisation techniques that significantly improve its performance. The results presented here are encouraging, showing performance numbers that are one order of magnitude better than the ones obtained in the tool’s previous version.

Software testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed an empirical study to shed light on this problem. Our study focuses on test code coverage, in particular that of automatic testing tools. Code coverage is commonly used for various purposes, such as evaluating test adequacy, assessing test quality, and analyzing testability. Our study uses code coverage as the concrete measurement of testability. By analyzing code coverage of two state-of-the-art tools, in comparison with that of developer-written tests, we have discovered that code visibility does not necessarily have effects on its code coverage, but significantly affects automatic testing tools. Low code visibility often leads to low code coverage for automatic tools. In addition, different treatments on code visibility can result in significant differences in overall code coverage for automatic tools. Using a tool enhancement specific to code visibility, we demonstrate the great potential to improve existing tools.

Regression testing is an important and costly strategy in software testing. To decrease its cost, many methods have been proposed including the test case prioritization methods. The aim of the prioritization methods is to define an ideal order of test cases that allows for higher coverage and early fault detection with minimal number of executed test cases. However, the problem with most of the existing test case prioritization methods is the random sorting of test cases when two or more test cases record equal priority values. In this paper, the effectiveness of an enhanced weighted method using a unique priority value, UniVal, is proposed. Unival prioritizes test cases based on code coverage criteria with information from history of previous executions. In addition, a controlled experiment was executed, and the results were statistically analyzed to assess the effectiveness of the proposed method. The results indicates better performance with regard to prioritize test cases and achieve higher APFD values.  Â

SummaryTo deliver reliable software, developers may rely on the fault detection capability of test suites. To evaluate this capability, they can apply code coverage metrics before a software release. However, recent research results have shown that these metrics may not provide a solid basis for this evaluation. Moreover, the fixing of a fault has a cost, and not all faults have the same impact regarding software reliability. In this sense, operational testing aims at assessing parts of the system that are more valuable for users. The goal of this work is to investigate whether traditional code coverage and code coverage merged with operational information can be related to post‐release bug detection. We focus on the scope of proprietary software under continuous delivery. We performed an exploratory case study where code branch and statement coverage metrics were collected for each version of a proprietary software together with real usage data of the system. We then measured the ability to explain the bug‐fixing activity after version release using code coverage levels. We found that traditional statement coverage has a moderate negative correlation with bug‐fixing activities, whereas statement coverage merged with the operational profile has a large negative correlation with higher confidence. Developers can consider operational information as an important factor of influence that should be analysed, among other factors, together with code coverage to assess the fault detection capability of a test suite.

Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite’s fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies’ results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.

BETA (Bbased testing approach) is a toolsupported approach to generate test cases from Bmethod specifications through the application of input space partitioning and logical coverage criteria. The BETA tool automates the whole process, from the design of abstract test cases to the generation of executable test scripts. In this paper, we present an empirical study that was performed to evaluate and contribute to the development of BETA. The study evaluated the applicability of BETA on problems with different characteristics and used techniques of code coverage and mutation analysis to measure the quality of the generated test cases. The study was carried out in different rounds, and the results of each round were used as a reference for the improvement of the approach and its supporting tool. These case studies were relevant not only for the evaluation of BETA but also to evaluate how different features affect the usability of the approach and the quality of the test cases and to compare the quality of the test cases generated using different coverage criteria. The results of this study showed that (1) BETAgenerated test scenarios for the different criteria follow theoretical expectations in terms of criteria subsumption; (2) the BETA implementation of the logical criteria generates more efficient test sets regarding code and mutation coverage than the input space partitioning ones; (3) it is important to go beyond the strict requirements of the criteria by adding some additional variation (randomization) of the input data; and (4) the algorithms designed to combine test requirements into test cases need to deal carefully with infeasible (logically unsatisfiable) combinations.

Graph and Logic Expression Coverage Criteria plays a dominant role towards testing embedded Safety Critical Systems. Although structural code coverage criteria in the safety critical application context have been explored by researchers to certain extent, the identification of the most suitable Graph and Logic Expression Coverage Criteria according to the control flow graph structure is challengeable and has not has been addressed much in available literature. This research proposes a methodology for graph coverage techniques and logic coverage techniques based exclusively on different Control Flow Graph clusters and dependencies between the decision nodes, together with the control paths. With an industry based case study validation, suggested methodology can be extended to automated metric collection tool for safety critical systems.

Automotive infotainment systems were getting more and more features in recent years. The usability of their HMIs (human-machine interfaces) has been improved considerably. However the complexity of the HMI software is growing. Testing the HMI became very demanding and time consuming. Because of multiplicity of HMI variants, a better code coverage is a goal for the development process of most manufacturers. Model-based testing is one way to achieve a better code coverage and keep the costs and complexity acceptable. However, the existing research approaches in the area of model-based HMI testing can not satisfy the needs for our testing purposes. In the work a model-based testing approach will be proposed for testing both the logical behavior and the graphical interface of the automotive infotainment HMI. As an important part of the testing approach a test-oriented HMI specification model will be designed. It is a model, which describes the required behavior of the HMI and contains the necessary information for the testing process. Test generation methods and the design of tests will also be proposed. These results can be generally used for testing advanced GUI-driven applications. Specific coverage criteria for infotainment HMIs, methods for automatic test generation and verification of the system behavior are also focuses of the work. The paper introduces the ideas and the goals of our model-based testing approach for infotainment HMIs.

Code coverage-guided unit test generation (CGTG) and large language model-based test generation (LLMTG) are two principal approaches for the generation of unit tests. Each of these approaches has its inherent advantages and drawbacks. Tests generated by CGTG have been shown to exhibit high code coverage and high executability. However, they lack the capacity to comprehend code intent, which results in an inability to identify deviations between code implementation and design intent (i.e., functional defects). Conversely, although LLMTG demonstrates an advantage in terms of code intent analysis, it is generally characterized by low executability and necessitates iterative debugging. In order to enhance the ability of unit test generation to identify functional defects, a novel framework has been proposed, entitled the intent analysis-guided unit test generation and refinement (IGTG&R) model. The IGTG&R model consists of a two-stage process for test generation. In the first stage, we introduce coverage path entropy to enhance CGTG to achieve high executability and code coverage of test cases. The second stage refines the test cases using LLMs to identify functional defects. We quantify and verify the interference of incorrect code implementation on intent analysis through conditional entropy. In order to reduce this interference, the focal method body is excluded from the code context information during intent analysis. Using these two-stage process, IGTG&R achieves a more profound comprehension of the intent of the code and the identification of functional defects. The IGTG&R model has been demonstrated to achieve an identification rate of functional defects ranging from 65% to 89%, with an execution success rate of 100% and a code coverage rate of 75.8%. This indicates that IGTG&R is superior to the CGTG and LLMTG approaches in multiple aspects.

The feedback provided by current testing education tools about the\ndeficiencies in a student's test suite either mimics industry code coverage\ntools or lists specific instructor test cases that are missing from the\nstudent's test suite. While useful in some sense, these types of feedback are\nakin to revealing the solution to the problem, which can inadvertently\nencourage students to pursue a trial-and-error approach to testing, rather than\nusing a more systematic approach that encourages learning. In addition to not\nteaching students why their test suite is inadequate, this type of feedback may\nmotivate students to become dependent on the feedback rather than thinking for\nthemselves. To address this deficiency, there is an opportunity to investigate\nalternative feedback mechanisms that include a positive reinforcement of\ntesting concepts. We argue that using an inquiry-based learning approach is\nbetter than simply providing the answers. To facilitate this type of learning,\nwe present Testing Tutor, a web-based assignment submission platform that\nsupports different levels of testing pedagogy via a customizable feedback\nengine. We evaluated the impact of the different types of feedback through an\nempirical study in two sophomore-level courses. We use Testing Tutor to provide\nstudents with different types of feedback, either traditional detailed code\ncoverage feedback or inquiry-based learning conceptual feedback, and compare\nthe effects. The results show that students that receive conceptual feedback\nhad higher code coverage (by different measures), fewer redundant test cases,\nand higher programming grades than the students who receive traditional code\ncoverage feedback.\n

The only tool that you need in order to develop applications or customizations using Salesforce DX is the Salesforce CLI, a command-line interface for executing Salesforce DX commands. Having said that, to improve productivity, an integrated development environment (IDE) is preferred. There are several IDEs to choose from, including Illuminated Cloud (a plug-in for the IntelliJ IDEA IDE), The Welkin Suite, and others. In this book, we are going to standardize on using Visual Studio Code with the Salesforce Extensions for Visual Studio Code; the extension pack is published by Salesforce, and both the extension pack and Visual Studio Code are free.

The purpose of this journal is to design and build a web-based computer accessories information and sales system at Toko Sarjana Komputer. This system is expected to facilitate the sales management process, inventory, and customer service more effectively and efficiently. The methodology used in developing this system is the waterfall method, which includes the stages of requirements analysis, system design, implementation, testing, and maintenance. This system was built using the PHP programming language by utilizing the Laravel framework, with Visual Studio Code as the Integrated Development Environment (IDE) and MySQL as the database. PHP was chosen because of its flexibility and good ability to process data and easily integrate with various types of databases. Visual Studio Code provides a development environment that supports various extensions to simplify coding and debugging MySQL is used to manage large-scale data and support complex transactions. With this system, it is hoped that Toko Sarjana Komputer can improve operational efficiency and provide better service to customers. This system also provides a platform that is more accessible to customers to purchase computer accessories online.

The Model Driven Architecture (MDA) approach is emerged in the last years as a novel software design methodology for the development of software systems. In this approach the focus of software development is shifted from writing code to modeling. In MDA, developers implement models that are automatically transformed into the target code of the system. Alongside MDA, the Model Driven Testing (MDT) is emerging as a relevant research topic in both industrial and scientific communities. MDT is a methodology where test cases for the system are automatically obtained starting from test models to maximize specific model coverage criteria. Eventually, test cases are executed to verify the system code that is generated through an MDA approach. In this paper, we conduct an exploratory study in order to evaluate the differences that may exist between the model coverage guaranteed by the test cases and the code coverage reached when they are executed on the auto-generated code. Moreover, we identify the main factors that may influence these differences.

The term grammar-based software describes software whose input can be specified by a context-free grammar. This grammar may occur explicitly in the software, in the form of an input specification to a parser generator, or implicitly, in the form of a hand-written parser. Grammar-based software includes not only programming language compilers, but also tools for program analysis, reverse engineering, software metrics and documentation generation. Hence, ensuring their completeness and correctness is a vital prerequisite for their use. In this paper we propose a strategy for the construction of test suites for grammar based software, and illustrate this strategy using the ISO C + + grammar. We use the concept of grammar-rule coverage as a pivot for the reduction of an implementation-based test suite, and demonstrate a significant decrease in the size of this suite. The effectiveness of this reduced test suite is compared to the original test suite with respect to code coverage and more importantly, fault detection. This work greatly expands upon previous work in this area and utilises large scale mutation testing to compare the effectiveness of grammar-rule coverage to that of statement coverage as a reduction criterion for test suites of grammar-based software. This work finds that when grammar rule coverage is used as the sole criterion for reducing test suites of grammar based software, the fault detection capability of that reduced test suite is greatly diminished when compared to other coverage criteria such as statement coverage.

Testing is an important and challenging part of software development and its effectiveness depends on the quality of test cases. However, there exists no means of measuring quality of tests developed for GPU programs and as a result, no test case generation techniques for GPU programs aiming at high test effectiveness. Existing criteria for sequential and multithreaded CPU programs cannot be directly applied to GPU programs as GPU follows a completely different memory and execution model. We surveyed existing work on GPU program verification and bug fixes of open source GPU programs. Based on our findings, we define barrier, branch and loop coverage criteria and propose a set of mutation operators to measure fault finding capabilities of test cases. CLTestCheck, a framework for measuring quality of tests developed for GPU programs by code coverage analysis, fault seeding and work-group schedule amplification has been developed and evaluated using industry standard benchmarks. Experiments show that the framework is able to automatically measure test effectiveness and reveal unusual behaviours. Our planned work includes data flow coverage adopted for GPU programs to probe the underlying cause of unusual kernel behaviours and a more comprehensive work-group scheduler. We also plan to design and develop an automatic test case generator aiming at generating high quality test suites for GPU programs.