Software reliability modeling based on test coverage

In order to overcome the limitations of debugging process, insufficient consideration of imperfect debugging and testing-effort (TE) in software reliability modeling and analysis, a software reliability growth model (SRGM) explicitly incorporating imperfect debugging and TE is developed. From the point of view of incomplete debugging and introduction of new fault, software testing process is described and a relatively unified SRGM framework is presented considering TE. The proposed framework models are fairly general models that cover a variety of the previous works on SRGM with ID and TE. Furthermore, a special SRGM incorporating an improved Logistic testing-effort function (TEF) into imperfect debugging modeling is proposed. The effectiveness and reasonableness of the proposed model are verified by published failure data set. The proposed model closer to real software testing has better descriptive and predictive power than other models

Many software reliability growth models have been proposed in the past decade. Those models tacitly assume that testing-effort expenditures are constant throughout software testing. This paper develops realistic software reliability growth models incorporating the effect of testing-effort. The software error detection phenomenon in software testing is modeled by a nonhomogeneous Poisson process. The software reliability assessment measures and the estimation methods of parameters are investigated. Testing-effort expenditures are described by exponential and Rayleigh curves. Least-squares estimators and maximum likelihood estimators are used for the reliability growth parameters. The software reliability data analyses use actual data. The software reliability growth models with testing-effort can consider the relationship between the software reliability growth and the effect of testing-effort. Thus, the proposed models will enable us to evaluate software reliability more realistically.

Software reliability growth models (SRGMs) have been developed to estimate software reliability measures such as the number of remaining faults, software failure rate, and software reliability. Issues such as imperfect debugging and the learning phenomenon of developers have been considered in these models. However, most SRGMs assume that faults detected during tests will eventually be removed. Consideration of fault removal efficiency in the existing models is limited. In practice, fault removal efficiency is usually imperfect. This paper aims to incorporate fault removal efficiency into software reliability assessment. Fault removal efficiency is a useful metric in software development practice and it helps developers to evaluate the debugging effectiveness and estimate the additional workload. In this paper, imperfect debugging is considered in the sense that new faults can be introduced into the software during debugging and the detected faults may not be removed completely. A model is proposed to integrate fault removal efficiency, failure rate, and fault introduction rate into software reliability assessment. In addition to traditional reliability measures, the proposed model can provide some useful metrics to help the development team make better decisions. Software testing data collected from real applications are utilized to illustrate the proposed model for both the descriptive and predictive power. The expected number of residual faults and software failure rate are also presented.

A large number of software reliability growth models (SRGMs) have been proposed during the past 30 years to estimate software reliability measures such as the number of remaining faults, software failure rate, and software reliability. Selection of an optimal SRGM for use in a particular case has been an area of interest for researchers in the field of software reliability. Tools and techniques for software reliability model selection found in the literature cannot be used with high confidence as they use a limited number of model selection criteria. For the first time, we developed a deterministic quantitative model based on a distance based approach (DBA) method, then applied it for evaluation, optimal selection, and ranking of SRGMs. DBA recognizes the need for relative importance of criteria for a given application, without which inter-criterion comparison could not be accomplished. It requires a set of model selection criteria, along with a set of SRGMs, and their level of criteria for optimal selection; and it successfully presents the results in terms of a merit value which is used to rank the SRGMs. We use two distinct, real data sets for demonstration of the DBA method. The result of this study will be a ranking of SRGMs based on the Euclidean composite distance of each alternative to the designated optimal SRGM.

Software testing is an essential activity in all software projects. The key issue in testing is determining the sufficiency of tests and, traditionally this has been done by using the Software Reliability Growth Models (SRGMs). However, SRGMs are not applicable when there are no stabilization phases as required by most software reliability models. Test-Defect Coverage Analytics Model (TDCAM) has been proposed to address this problem. This paper proposes the application of visual analytic techniques as an approach for supporting informed decision making in deciding the sufficiency of tests. Visual analytics research considers interactive visualization as the common platform for combining various computational data analysis techniques to support the analytical reasoning process. We presented four visual representations of TDCAM to demonstrate how analytical models have been applied to indicate the adequacy of tests in relation to sufficient and efficient test coverage. The techniques provide an effective and generally applicable test estimation on the basis of a general trend that higher test coverage correlates with higher probability of detecting more defects.

With the growth of complexity in the software system, to deliver reliable software products on time becomes a critical issue. Many software reliability growth models (SRGMs) have been proposed in the past three decades. However, most software reliability assessment processes and parameter estimations of models depend on the computations of the general-purposed numerical software. In this paper, we will present a powerful computer-aided reliability assessment tool for software based on object-oriented analysis and design-CARATS. CARATS can use both traditional SRGMs and neural-network methods to assess software reliability. This would greatly help project managers to make decisions during software development life cycle. Due to the characteristics of the special-purposed and object-oriented design, CARATS can analyze the software reliability easily, and is also more flexible to adopt different SRGMs than other existing software reliability analysis tools.

Software reliability models are used to estimate and predict the reliability, number of remaining faults, failure intensity, total software development cost, etc., of a software. Testing coverage is very important for both software developers and customers of software products. Testing converge is a measure that enables software developers to evaluate the quality of tested software and determine how much additional effort is needed to improve the reliability of the software. This paper proposes a software reliability growth model based on a non-homogeneous Poisson process (NHPP) that incorporates a logistic–exponential testing coverage function with imperfect debugging. The proposed model relates the test coverage to fault detection phenomena in debugging. Goodness-of-fit test of the proposed model is conducted using different criteria for two sets of software failure data. The proposed model is compared with other existing NHPP models. A software cost model incorporating testing coverage and an optimal release policy based on the number of remaining faults are developed.

In this chapter we discuss the software reliability growth models (SRGMs) that describe the relationship between the number of faults removed and the number of test cases used. Firstly we describe the discrete exponential and S-shaped models in a perfect debugging environment. We also discuss flexible discrete SRGM, which can depict either exponential or S-shaped growth curves, depending upon the parameter values estimated from the past failure data. Further, we describe an SRGM for the fault removal phenomenon in a perfect debugging environment. Most testing processes are imperfect in practice, therefore we also discuss a discrete model that incorporates the impact of imperfect debugging and fault generation into software reliability growth modeling. Faults in the software are generally not of the same type, rather the faults contained in a large software may differ from each other in terms of the amount of time and skill of the removal team required to remove them. Three discrete models: the generalized Erlang model, modeling severity of faults with respect to testing time, and a model with faults of different severity incorporating logistic learning function have been discussed. A discrete model in a distributed environment is also discussed. The above discrete SRGMs assume a constant fault detection rate while testing the software under consideration. In practice, however, the fault detection rate varies because of changes in the testing skill, the system environment, and the testing strategy used to test the software. SRGMs for the fault removal phenomenon, the generalized Erlang model, and the generalized Erlang model with logistic function are discussed, incorporating the concept of change point. It is also shown how equivalent continuous models can be derived. This chapter describes the state-of-the-art in discrete modeling.

It is one of the important issues in modern software development to make a highly reliable software system by efficient and economical testing. As one of these solutions, many companies have taken in testing-progress management and quality/reliability assessment based on software reliability growth models. Consequently, many software reliability growth models reflecting various development factors and operational environment of the software system have been proposed. Most of these models have assumed the software system as one domain. That is, the models have assumed that the set of the testing-paths influenced by execution of test-cases is extended with testing-progress and it expands to whole software system finally. However, a software system has the structure that the main-component calls plural sub-components ordinarily. Therefore, unless the main-component is tested, the sub-component called from it is not tested. In this chapter, we propose software reliability growth models based on such components' characteristics. Especially, these models reflecting the different testing-environment for composed components are formulated by nonhomogeneous Poisson processes. Furthermore, using fault-detection data observed in actual development projects, we show numerical examples of software reliability assessment and results of goodness-of-fit comparisons with existing software reliability growth models.

In the software industry, Software Reliability Growth Models (SRGMs) with confidence intervals (C.I.) are frequently employed as valuable tools to assist manager to determine the optimal timing for software releases and enhancing the effectiveness of testing processes. However, many SRGMs lack clear explanations regarding the estimation of variance for cumulative software errors. This limitation can impede their ability to accurately derive C.I. for the mean value function (MVF), making it challenging for developers to evaluate potential risks and uncertainties within the software system. Consequently, the practical application of these models may be compromised. To address this issue, this paper proposes the utilization of Ohba’s Inflection S-shaped model and Yamada’s delayed S-shaped model to construct SRGMs with C.I., thereby providing developers with improved guidance for making optimal release time decisions.

This paper discusses two-dimensional software reliability measurement technologies, which describe a software reliability growth process depending on two-types of software reliability growth factors: Testing-time and testing-effort factors. From the view point of actual software failure-occurrence mechanism, it is natural to consider that a software reliability growth process depends on not only testing-time factor but also other software reliability growth factors, i.e., testing-effort factors, such as test-execution time, testing-skill of test engineers, testing-coverage. From such reason, the two-dimensional software reliability measurement technologies enable us to conduct more feasible software reliability assessment than the one-dimensional (conventional) software reliability measurement approach, in which it is assumed that the software reliability growth process depends only on testing-time. We discuss two-types of two-dimensional software reliability growth modeling approaches for feasible software reliability assessment, and conduct goodness-of-fit comparisons of our models with existing one-dimensional software reliability growth models. Finally, as one of the examples for two-dimensional software reliability analysis, we show examples of the application of our two-dimensional software reliability growth model by using actual data.

A software reliability growth model for imperfect debugging

Software reliability assessment for an exponential-S-shaped reliability growth phenomenon

Over the past three decades, many software reliability growth models (SRGMs) were proposed and they are aimed at predicting and estimating software reliability. One common assumption of these conventional SRGMs is that detected faults will be removed immediately. In reality, this assumption may not be reasonable and may not always occur. Developers need time to identify the root causes of detected faults and then fix them. Besides, during debugging the fault correction rate may not be a constant and could be changed at some certain points as time proceeds. Consequently, in this paper, we will explore and study how to apply queueing model to investigate the fault correction process during software development. We propose an extended infinite server queueing model with multiple change-points to predict and assess software reliability. Experimental results based on real failure data show that proposed model can depicts the change of fault correction rates and predict the behavior of software development more accurately than traditional SRGMs.

Multiple Software Reliability Growth Models (SRGM) has been illustrated to inspire designers and software developers to develop quality software. These models implicitly assume that expenditure is constant throughout testing effort. Recently importance was given to the testing phase by the development team and it requires controlled screening closely to raise its overall operational efficiency. Software Reliability Growth Models exhibits a precise relation between the failure patterns with time. Considering this as a basic building block a SRGM which is a Non-Homogeneous Poisson Process in nature is i in this work. The various aspects of our proposed model are testing effort function and fault detection rate (FDR). Mainly in Software Reliability Growth Model it was assumed that fault removal process are perfect which is not true in reality as new faults may get introduced during removal or some faults may not get removed due to its complexity. Thus in this model the concept of imperfect debugging was considered and assists us to justify the software reliability more convincingly through the help of testing effort function.