A novel method for increasing the effectiveness and efficiency of software reliability testing

In recent years, smart grids have developed rapidly. As an important terminal of the smart grid, the reliability and safety of intelligent electric meters play an important role in the development of smart grids. However, in practical applications, intelligent electric meters have problems such as inconsistent software architecture, less penetration testing involved in software testing, and difficulty in finding vulnerabilities. In order to solve the above-mentioned problems, this paper analyzes the software architecture of intelligent electric meters and establishes a hierarchical system model. Then, actively analyzes the weaknesses, defects or vulnerabilities in the software to detect the design vulnerabilities and software defects in the intelligent electric meter, gives qualitative or quantitative test and verification results for its fault tolerance and ease of recovery, and establishes a complete software reliability and safety testing system, and develops a prototype system, which has good guiding significance and implementation value for the software reliability and safety testing of intelligent electric meters.

Software reliability testing refers to various software testing activities that are driven to achieve a quantitative reliability goal given a priori or lead to a quantitative reliability assessment for the software under test. In this paper we develop a modeling framework for the software reliability testing process, comprising a simplifying model and a generalized model. In both models the software testing action selection process and the defect removal mechanism are explicitly described. Both the discrete-time domain and the continuous-time domain are involved. The generalized model is more accurate or realistic than the simplifying model since the former avoids the assumption that defects are equally detectable and the assumption that defects are removed upon being detected. However simulation examples show that the simplifying model really captures some of essential features of the software testing process after a short initial testing stage. The modeling framework is practically realistic, mathematically rigorous, and quantitatively precise. It demonstrates that the relationship between software testing and delivered software reliability, which was poor understood, can well be formulated and quantified. Rigorous examinations show that several common assumptions adopted in software reliability modeling, including the independence assumption, the exponentiality assumption, and the NHPP assumption, are theoretically false in general. This paper sets a good starting point to further formalize and quantify the software testing process and its relation to delivered software reliability.

The software reliability failure data is the foundation of the software reliability’s quantitative evaluation based on the failure data, and it has an important influence on the accuracy of reliability evaluation. But there are always noises in the original software reliability failure data and make the reliability evaluation accuracy affected. This paper put forward the collecting method of reliability failure data and data preprocessing method including data cleaning and data analysis method, which based on the analysis of the importance and the source of failure data in the software reliability testing and the classification of software failure data. Finally through an example, it displayed the reduction of data noises and the promotion of data quality which produced by the preprocessing methods.

Software fails because it contains faults. If we could be certain that our attempts to fix a fault had been effective and that we had introduced no new faults, we would know that we had improved the software's reliability. Unfortunately, no single testing method can be trusted to give accurate results in every circumstance. There are at least two areas of uncertainty in testing. First, we cannot predict when a failure will occur as a result of choosing an input that cannot currently be processed correctly. Second, we do not know what effect fixing a fault will have on the software's reliability. Most of today's reliability models consider the software system to be a black box, using indirect measures of failure by making strong assumptions about the test cases. White-box testing measures, which are direct measures, are generally not used in software-reliability models. The assumption is that the more the program is covered, the more likely that the software is reliable. Hence we must develop new testing strategies. For example, we do not understand very well the notion of stress as it applies to software; what effect the dependence between versions of the same application has on reliability; the relationship between software reliability and white-box testing measures; and how to achieve the ultra-high reliability levels obtained through formal verification and fault-tolerance methods. These are just some of the areas for future research into improved testing strategies.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The traditional approach for generating software reliability test cases according to operational profile (OP) can be regarded as a simple random sampling. The stratified sampling is a complex but more accurate sampling approach which can make the sampling results more accurate and less variance by dividing the population into several subsets and randomly sampling from each subset respectively. Firstly, this paper introduced the traditional method which can be used to determine the number of test cases in the discrete-type software reliability demonstration testing. Secondly, the necessary condition for the minimum test case based on the stratified sampling was studied. Then a new approach was proposed and its principle was analyzed for determining the minimum test case of the discrete-type software by using the stratified sampling. Finally, the future work was discussed.

Combinatorial Testing (CT) has been proven to be effective in detecting and locating the interaction triggered failure in the last 20 years. But CT still suffers from many challenges, such as modeling for CT, sampling mechanisms for test generation, applicability and effectiveness. To overcome these issues of CT, adaptive combinatorial testing (ACT) is proposed in this paper, which improves the traditional CT with a well established adaptive testing method as the counter part of adaptive control and aims to make CT more flexible and practical. ACT can significantly enhance testing quality, software reliability and support testing strategy adjustment dynamically. To support further investigation, a preliminary form of concrete strategy for ACT is given as a heuristic guideline.

Over the last two decades, various software reliability growth models (SRGM) have been proposed, and there has been a gradual but marked shift in the balance between software reliability and software testing cost in recent years. Chiu and Huang (2008) provided a Software Reliability Growth Model from the Perspective of Learning Effects, which is able to reasonably describe the S-shaped and exponential-shaped types of behaviors simultaneously, and offers better performance when fitting different data with consideration of the learning effects. However, this earlier model assumes that the learning effects are constant. In contrast, this paper discusses a software reliability growth model with time-dependent learning effects.

By means of analysis of software's usage, it can be known that the software's actual usage is just like a stochastic process, whose series of usage events are extended according to some probability distribution. If the statistical characteristics of software reliability test suite is that of the implementation in practice when customers actually use the software product, that is to say, the test process is converge to the actual usage process. Thus the minimal test effort can be determined by measuring the approximate extent of statistical characteristics of test suite to that of the whole input domain. Based on the sampling probability, one determination method of the minimal software reliability test effort is proposed in this paper and the recommended value is presented. Compared to one existing method, this method is based on the same principle of the test suite's statistical characteristics, but it adopts one different mathematical method whose demonstration detail is given and finally the influence factors of the minimal test effort are discussed in detail.

Currently embedded software reliability testing of intellectual ammeter is through software comparison, which has poor scalability and can’t detect software’s fault tolerance ability after failure in running scene. In this regard, this paper presents a detection platform’s construction scheme for intellectual ammeter software reliability testing and designs testing communication protocol based on Ethernet master-slave mode. Experiments prove that the testing protocol possesses high usability and flexibility, and has value for intellectual ammeter software testing.

Failure Mode and Effect Analysis (FMEA) has been used quite extensively in aerospace industry for several years to establish hardware criticality. Determination of software reliability is equally important. There are several methods for determining software reliability. One methods for performing software reliability is the software FMEA. This paper discusses the application of principles to FMEA. A typical Tank pressurant is considered as an illustrative example. The tank pressurant system consists of three sets of solenoid valves (each set consists of a pair of valves), three pressure transducers, a vent valve, a heat exchanger and helium cold pressure bottles. Each set of solenoid valves consist of two solenoid valves connected in series. For the system to function properly, certain conditions have to be met. These are discussed in detail later in this paper. Based on the information provided from NASA, it is concluded that FMEA has to be capable of determining the software failure modes, effects and criticality as well as the hardware failure modes effects and criticality which includes the pressure transducers, solenoid valves, vent valve, heat exchanger and helium cold pressure bottles. The paper will discuss both the hardware and software FMEA application in detail.

SUMMARY & CONCLUSIONSIn the realm of reliability testing, one of the fundamental challenges lies in the identification and selection of the most critical failure modes and corresponding performance parameters. With a vast array of potential failure mechanisms and performance metrics, engineers often struggle to pinpoint those that truly reflect real-world degradation and field performance. Traditionally, this selection process is guided by expert judgment and established references such as the NSWC Mechanical Handbook, and other well-defined handbooks. However, practical constraints including limited time, budget, and test resources make exhaustive testing for all potential failure modes unfeasible.This paper introduces the Critical Failure Mode Prioritization (CFMP) methodology for reliability test design. Unlike traditional approaches that rely heavily on expert opinion and static reference standards, CFMP integrates real-world operating conditions and focuses on prioritizing the most significant and probable failure modes. By leveraging tools such as Parameter Diagram (P-Diagram), Design Failure Mode and Effects Analysis (DFMEA), actual field usage data, and historical field failure data, this methodology moves beyond subjective biases. It identifies and models failure modes based on their occurrences, Mean Time to Failure (MTTF), and percentage of contributions, thereby enhancing the relevance and accuracy of the testing process.The key distinction of the CFMP approach lies in its ability to handle products with multiple failure modes and mechanisms. It emphasizes identifying the critical failure mode or mechanism that could lead to system failure as early as possible. This method is more comprehensive, focusing on end-to-end processes and prioritizing sub-assembly and system-level testing from the end user's perspective, rather than primarily from an engineer’s point of view.This paper addresses the highly relevant topic of prioritizing different failure modes in reliability testing. It mentions relevant exemplary data, especially on the side of requirements.Moreover, the CFMP methodology is grounded in core reliability engineering principles. By incorporating physics-of-failure (POF) analysis to explore optimal failure mechanisms, degradation modeling to forecast failure progression, and preventive maintenance strategies for proactive decision-making, CFMP ensures that test conditions are aligned with actual usage profiles. This results in more accurate, relevant, and efficient testing, directly supporting the prediction of field performance and facilitating better-informed decisions.

Reliability testing of software is the key area of concern now-a-days; specially with the software which are safety critical and security prone systems. There is always a need of high software reliability. Most of the present models for evaluation of software reliability are based on statistical and probability approach. When we look into the use of these models for software reliability testing, we observe that there are possibilities of imprecision in the reliability estimation. For removing this imprecision, the Fuzzy logic and Fuzzy expert systems are used within various researches. The use of Fuzzy logic for reliability estimation can enhance the reliability of software even during the early stages of software development. Here we are giving a comparison analysis of few approaches and models proposed so far, for the estimation of software reliability and its improvement using Fuzzy logic.

A web service reliability test method for C/S architecture software based on log analysis is presented in this paper. In this method, the software usage model is constructed automatically to describe the real situation on the users' access to the web service by Markov chain. The test cases are generated according to Random Walk and applied to software reliability test. In the experiment process, MTBF (focusing on server crash) was chosen to be the software reliability evaluation index. Through the testing and analysis of a real web software, MTBF obtained by testing result is similar to that from the realistic log, and the web service reliability test method is validated.

This paper describes how software is in the heart of many safety critical systems in the railway sector. The development of systems that include software modules requires a correct evaluation of software Reliability, Availability, Maintainability and Safety (RAMS) to get a correct value of the overall system RAMS. In order to obtain appropriate software, the standards propose to perform a set of activities in the different phases of software development as well as tasks to control their correct accomplishment. They ensure the developed software is of adequate quality. However, it is necessary to go further and try to obtain a quantitative measure of RAMS for each software module as is usually done in hardware development. There are several techniques for the assurance of software reliability and safety that have been in use for years and must be analyzed to know their real potential: reliability growth models, artificial intelligence techniques, Markov chains, Software Fault Tree Analysis and Software Failure Mode and Effect Analysis among others. Two circumstances emphasize the strategic time the railway sector is living and the opportunity to adopt the most promising software techniques improve reliability and safety: (1) The development of high-performance railway networks that interconnect different countries and the liberalization and opening of the national markets demand new European global agreements. In this regard, the European Railway Agency has asked its Safety Unit to develop the new Common Safety Methods (CSM) and Common Safety Targets (CST) to be used in all European countries; (2) The IEC 61508-3 standard (from which some parts of CENELEC 50128 are derived) is now under revision, with the primary aim of ensuring the safety of the developed software by hardening the requirements and promoting the use of the most promising techniques.

This paper examines one of the common modes of structural failure in multichip BGAs, determines its locations within the package structure, relates it to the stresses generated in the reliability tests under which it occurs, and by Finite Element simulations, determines an explanation for the failure, and finally proposes a method to avoid this failure mechanism. Several designs of multichip BGA substrates were manufactured and production silicon assembled into them. These were all 14/spl times/22 mm 119 ball PBGA. These were subjected to a set of package reliability tests, until some units failed electrical test. The failed units were analysed and the physical location and shape of the failure was determined in many cases. From this information, the mechanical mode of failure for each unit was determined. In addition there was sufficient information in some of the analyses to provide definite suggestions as to the mechanism of failure. Meanwhile, Finite Element Analysis was performed using simplified representations of the multichip BGAs, in order to find the locations of highest stress, and the expected modes of failure. This data was matched to the failure modes found in the physical analysis. Some novel failure analysis techniques were used to expose the damage in the failed units. A particular failure mode occurred frequently in temperature cycle, and the sites of failure were located by failure analysis. The failure was due to open circuit in the copper tracks in the top layer of the substrate caused by cracking in the solder resist directly underneath the edge of the die attach fillet. Finite element analysis was carried out and the location of the actual failures was found to be a local zone of high tensile stress in the solder resist.