Software reliability measurement with effect of change-point: modeling and application

In reality, the fault detection (or correction) phenomenon and software reliability estimation in the operational phase are different from that in the testing phase. The fault removal continues at a slower rate during the operational phase. In this paper, we will investigate some techniques for software reliability prediction and measurement in the operational phase. We first review how some software reliability growth models (SRGMs) based on non-homogeneous Poisson processes (NHPPs) can be readily derived based on a unified theory for NHPP models. Under this general framework, we can not only verify some conventional SRGMs but also derive some new SRGMs that can be used for software reliability measurement in the operational phase. That is, based on the unified theory, we can incorporate the concept of multiple changepoints into software reliability modeling. We can formularize and simulate the fault detection process in the operational phase. Some numerical illustrations based on real software failure data are also presented. The experimental results show that the proposed models can easily reflect the possible changes of fault detection process and offer quantitative analysis on software failure behavior in field operation.

This paper discusses two-dimensional software reliability measurement technologies, which describe a software reliability growth process depending on two-types of software reliability growth factors: Testing-time and testing-effort factors. From the view point of actual software failure-occurrence mechanism, it is natural to consider that a software reliability growth process depends on not only testing-time factor but also other software reliability growth factors, i.e., testing-effort factors, such as test-execution time, testing-skill of test engineers, testing-coverage. From such reason, the two-dimensional software reliability measurement technologies enable us to conduct more feasible software reliability assessment than the one-dimensional (conventional) software reliability measurement approach, in which it is assumed that the software reliability growth process depends only on testing-time. We discuss two-types of two-dimensional software reliability growth modeling approaches for feasible software reliability assessment, and conduct goodness-of-fit comparisons of our models with existing one-dimensional software reliability growth models. Finally, as one of the examples for two-dimensional software reliability analysis, we show examples of the application of our two-dimensional software reliability growth model by using actual data.

For several decades both practitioners as well as researchers have contributed immensely towards solving one of the prime concerns of software firms regarding how long to continue testing? Although, efficient testing ensures reliable software but costs a lot to developer in terms of higher testing and market opportunity cost. Hence it’s imperative to obtain optimal testing time of software. Product release marking the end to software testing has been the phenomenon commonly observed in both practice and theory but often leaves customer dissatisfied either in the form of delayed release (prolonged testing phase) or lower reliability (shortened testing phase). To address this issue, current practices in software industry involve a post release testing phase during which organization aims at treating remaining software faults and subsequently enhance product experience for customers. In this paper we present a generalized framework of optimal scheduling policy to determine optimal software release and testing stop time separately, to minimize overall testing cost under reliability and budget constraints. One of the biggest advantages of early release is capitalizing market opportunity while post release testing ensures adequate software reliability. We also propose that software testing & operational phases are governed by different distribution functions in our proposed cost model. Numerical illustration is provided at the end of the paper to validate the proposed cost model.

Software failure-occurrence or the fault-detection phenomenon is changed notably in an actual testing phase or an operational phase due to the changes of factors influencing the software reliability growth process. Taking into consideration of the effect of the change in software reliability growth modeling is expected to conduct more accurate software reliability assessment because it is said that such approach enables us to conduct more plausible software reliability assessment reflecting an actual testing-environment. We develop a framework for developing software hazard rate models with effect of change-point, and discuss change-point detection methods for applying our model to quantitative software reliability assessment. Additionally, we discuss an optimal software release problem for estimating optimal shipping time with the effect of the change as one of the application problems of our model in software project management. Finally, we show numerical examples for software reliability assessment and our optimal software release policy based on our change-point model by using actual data.

PurposeThe aim of this paper is to propose a software reliability growth model based on stochastic differential equations for the integration testing phase of distributed development environment.Design/methodology/approachA client/server system (CSS), which is a new development method, has come into existence as a result of the progress of networking technology by UNIX systems. On the other hand, the effective testing method for distributed development environment has only a few presented. The method of software reliability assessment considering the interaction among software components in a distributed one is discussed.FindingsConventional software reliability growth models for system testing phase in distributed development environment have included many unknown parameters. Especially, the effective estimation method in terms of these unknown parameters, which means the proportion of the total testing‐load for the software component, has never been presented. This software reliability growth model can be easily applied in distributed software development, because the model has a simple structure.Practical implicationsThis model is very useful for software developers in terms of practical reliability assessment in the actual distributed development environment.Originality/valueThe method of software reliability assessment considering the interaction among software components in distributed development environment is proposed. Additionally, several numerical examples for the actual data are presented.

Management technologies for improving software reliability are very important for software TQM (Total Quality Management). The quality characteristics of software reliability is that computer systems can continue to operate regularly without the occurrence of failures on software systems. In this chapter, we describe several recent developments in software reliability modeling and its applications as quantitative techniques for software quality/reliability measurement and assessment. That is, a quality engineering analysis of human factors affecting software reliability during the design-review phase, which is the upper stream of software development, and software reliability growth models based on stochastic differential equations and discrete calculus during the testing phase, which is the lower one, are discussed. And, we discuss quality-oriented software management analysis by applying the multivariate analysis method and the existing software reliability growth models to actual process monitoring data. Finally, we investigate an operational performability evaluation model for the software-based system, introducing the concept of systemability which is defined as the reliability characteristic subject to the uncertainty of the field environment.KeywordsHuman factor analysisDesign-review experimentOSS reliabilityStochastic differential equationDiscrete modelingDifference equationSoftware managementSoftware project assessmentSoftware performability modelingSystemability assessment

Quality, cost, and delivery are important software development management factors that software development managers control during the testing phase of the software development process. In particular, the software development management techniques based on software reliability growth models (SRGM) are often used. The majority of SRGMs are developed by treating the software failure-occurrence time or the fault-detection time intervals as random variables, and they assume that the stochastic characteristics of these quantities are the same throughout the testing phase. However, researchers often observe that stochastic characteristics of software failure-occurrence time intervals change because of a change in the testing environment. The chapter discusses the optimal software release problem based on the change-point model as an application. It describes the optimal software release problem based on the multi-attribute utility theory (MAUT). The MAUT is a utility theory on decision-making considering multiple constraints.

This study delivers a valuable adjoin to the literature of software reliability growth model (SRGM) and associated software release time problem. In context of SRGM, this study examines the incorporation of testing coverage and Fault Reduction Factor (FRF) simultaneously. Since the fault detection process and coverage rate generally get influenced by several attributes including testing tactics, change in resources etc. therefore the principle of change point analysis was introduced to deal with such changes during the testing phase. Models proposed in the study are developed using Non Homogeneous Poisson Process that can be used to evaluate the reliability of software system quantitatively. We have also discussed goodness-of-fit of the proposed models by statistical estimation on two real life fault datasets. To investigate the optimal time to end the testing of the software, a release planning problem is discussed. The proposed release planning problem takes into consideration the minimization of the development cost subject to the reliability requirement constraint. The developed cost model is modeled with the help of some established cost items such as costs of removing faults in testing phase as well as in operational phase and fixed cost per unit time. The study finally takes into consideration sensitivity analysis in optimal release time problem taking into account the impact of variations in the cost parameters.

In this paper we consider non-parametric estimation methods for software reliability assessment without specifying the fault distribution, where the underlying stochastic process to describe software fault-counts in the system testing is given by a non-homogeneous Poisson process. A comprehensive approach based on the kernel estimation is provided with several kernel functions and bandwidth estimations. Next, we develop interval estimation methods via the non-parametric bootstrap, and derive the confidence regions of several reliability measures such as the expected cumulative number of software faults, software intensity function, quantitative software reliability as well. The resulting data-driven methodology can give the useful probabilistic information on the software reliability prediction under the incomplete knowledge on fault distribution. In illustrative examples with a real software fault data, it is shown that the proposed methods provide useful software reliability measures under uncertainty from the view point of frequentist analysis.

In this paper, software reliability measurement is studied from the perspective of usage profile. Reliability parameters on each usage profile are expressed with unascertained rational numbers. Since failure process of each usage profile can be affected by multiple factors such as software attributes and users imports etc., GEP (gene expression programming) is introduced into software reliability modeling, to study comprehensive force among the factors. Finally, a software reliability GEP model based on usage profile is constructed. With the new model, users can confirm software reliability according to its usage, thus enabling software reliability to a more objective assessment. The new model is experimented to measure reliability of SYS in Musa data, and proved with a satisfactory performance. DOI: http://dx.doi.org/10.11591/telkomnika.v10i7.1572 Full Text: PDF

A software reliability growth model incorporating the amount of test effort expenditures during the software testing phase is described. The time-dependent behavior of test-effort expenditures is described by a Weibull curve due to the flexibility. Assuming that the error detection rate to the amount of test-effort spent during the testing phase is proportional to the current error content, the model is formulated by a nonhomogeneous Poisson process. Using this model, the method of data analysis for software reliability measurement is presented. Also, this model is applied to the prediction of additional test-effort expenditures to obtain the objective number of errors detected by software testing and the determination of the optimum time to stop software testing for release. >

Many software reliability growth models have been proposed in the past decade. Those models tacitly assume that testing-effort expenditures are constant throughout software testing. This paper develops realistic software reliability growth models incorporating the effect of testing-effort. The software error detection phenomenon in software testing is modeled by a nonhomogeneous Poisson process. The software reliability assessment measures and the estimation methods of parameters are investigated. Testing-effort expenditures are described by exponential and Rayleigh curves. Least-squares estimators and maximum likelihood estimators are used for the reliability growth parameters. The software reliability data analyses use actual data. The software reliability growth models with testing-effort can consider the relationship between the software reliability growth and the effect of testing-effort. Thus, the proposed models will enable us to evaluate software reliability more realistically.

Software testing involves verification and validation of the software to meet the requirements elucidated by customers in the earlier phases and to subsequently increase software reliability. Around half of the resources, such as manpower and CPU time are consumed and a major portion of the total cost of developing the software is incurred in testing phase, making it the most crucial and time-consuming phase of a software development lifecycle (SDLC). Also the fault detection process (FDP) and fault correction process (FCP) are the important processes in SDLC. A number of software reliability growth models (SRGM) have been proposed in the last four decades to capture the time lag between detected and corrected faults. But most of the models are discussed under static environment. The purpose of this paper is to allocate the resources in an optimal manner to minimize the cost during testing phase using FDP and FCP under dynamic environment. An elaborate optimization policy based on optimal control theory for resource allocation with the objective to minimize the cost is proposed. Further, genetic algorithm is applied to obtain the optimum value of detection and correction efforts which minimizes the cost. Numerical example is given in support of the above theoretical result. The experimental results help the project manager to identify the contribution of model parameters and their weight.

This investigation deals with a N-modules software reliability growth model (SRGM) wherein each module attains multiple types of faults of different severity levels. During the debugging phase, there may arise some new faults due to assumption of imperfect debugging. The reliability growth during the software testing phase governed by a non-homogeneous Poisson Process (NHPP). The proposed study is devoted to analyze the fault debugging phenomenon in all modules of software and to obtain expected number of faults observed, isolated and removed. The software reliability and optimum total cost of the software development during the testing phase are evaluated. The causes of fault occurrences in each module are different. The faults may occur with different failure rates and follow the nonlinear behavior with respect to the testing time. The NHPP model incorporates the testing-effort expenditure required for the faults debugging activities and governed by Weibull and logistic distributions which influence considerably not only the fault observation rate but also the time to isolate the faults and the time to remove the faults from all N-modules. In addition to predict the reliability of N-modular software, this study proposes optimal release policies for the SRGM by minimizing the expected cost subject to budget and testing-effort constraints, while ensuring a prescribed level of reliability and incorporating the effects of testing effort and efficiency. Some specific models are also deduced by setting the appropriate parameters so as to match with the existing models in the literature. Sensitivity results are presented through a numerical experiment to demonstrate the validity of the analytical formulations and the computational tractability of the model. An evolutionary algorithm, namely Particle Swarm Optimization (PSO) based on a dynamic penalty method, is employed to determine the optimal release time.

A large number of software reliability growth models (SRGMs) have been proposed during the past 30 years to estimate software reliability measures such as the number of remaining faults, software failure rate, and software reliability. Selection of an optimal SRGM for use in a particular case has been an area of interest for researchers in the field of software reliability. Tools and techniques for software reliability model selection found in the literature cannot be used with high confidence as they use a limited number of model selection criteria. For the first time, we developed a deterministic quantitative model based on a distance based approach (DBA) method, then applied it for evaluation, optimal selection, and ranking of SRGMs. DBA recognizes the need for relative importance of criteria for a given application, without which inter-criterion comparison could not be accomplished. It requires a set of model selection criteria, along with a set of SRGMs, and their level of criteria for optimal selection; and it successfully presents the results in terms of a merit value which is used to rank the SRGMs. We use two distinct, real data sets for demonstration of the DBA method. The result of this study will be a ranking of SRGMs based on the Euclidean composite distance of each alternative to the designated optimal SRGM.