Quantifying the reliability of software: statistical testing based on a usage model

The paper concerns automatic generation of usage models from real-time software UML models. Firstly, we define the reasonably constrained real-time software UML artifacts, which include use case diagrams, timed sequence diagrams and the execution probability of each sequence diagram in its associated use case. Secondly, the paper presents a method that derives the software usage model from the constrained UML artifacts. The method elicits the messages associated with the objects under testing and their occurrence probabilities to generate the usage model of each use case. Timing constraints in sequence diagrams are considered during usage model generation. Then the usage models of use cases are integrated into the software usage model by utilizing the execution sequence relations between use cases. The usage models can be used to generate real-time software statistical test cases and facilitate real-time software statistical testing.

Statistical testing based on a Markov chain usage model, as a rigorous testing method, has been around for more than two decades. Through the comprehensive application of statistical science to the testing of software, it provides audit trails of evidence to support correctness arguments for a software-intensive system as well as a decision that the system is of requisite quality for its intended use. This paper reports a real-world case study in which we applied standard statistical testing practices to the phone flag copy testing problem at the site of our industrial collaborator, and presents our solution from problem formalization, usage modeling and model analysis to test case generation and analysis. Our results helped evaluate the coverage of a heuristically generated test suite, and shed light on what other test cases to craft/generate to improve test coverage and adequacy, as well as reliability estimates both at the arc (usage event) level and at the path (system) level.

CenUMs – concurrency enhanced usage models for statistical testing of complex systems with concurrent streams of use

Importance sampling (IS) is a change-of-measure technique for speeding up the simulation of rare events in stochastic systems. In this paper, we presented an approach uses Importance Sampling technique for efficient estimation of software reliability via Markov software usage models in statistical testing. By suitable changes of the probabilities of state transitions during test, an iterative method based on the Ali-Silvey distance is proposed for this choice, and an unbiased reliability estimator with zero variance is obtained. A learning algorithm for the computation of optimal transition probabilities of the Markov chain usage model is also presented and experimental results of this algorithm are reported.

Markov chain usage-based statistical testing has been an effective means in the economical production of high quality software that also provides credible evidence to support its dependability. Sequence-based specification is a rigorous specification method that derives a formal system model from informal functional requirements, which can be used as a formal method to construct a Markov chain usage model for statistical testing. This paper demonstrates a case study in which we apply these two methods in conjunction with supporting tools towards fully automated statistical testing of a GUI application. Preliminary results are promising.

A Polyhedron Approach to Calculate Probability Distributions for Markov Chain Usage Models

An approach to user modelling with discrete stochastic processes is presented, which aims at the dynamic individualization of user interfaces on the syntactic layer. The state of the art of syntactic user modelling is surveyed. The mathematical background of simple Markov Chains and the 'classic' Hidden Markov Model is presented. Furthermore, dynamic Bayesian Networks are introduced, which generalize these Markovian Models. A case study of the simulation experiments uses a multimodal user interface for supervisory control of advanced manufacturing cells. A corresponding simulation model is created and exploited to generate interaction cases, which are the empirical basis for the evaluation. Six topologies of dynamic Bayesian Networks are evaluated for 100 interaction cases and 50 replications each: (1) Markov Chain of order 1, (2) Hidden Markov Model, (3) autoregressive Hidden Markov Model, (4) factorial Hidden Markov Model, (5) simple hierarchical Hidden Markov Model, and (6) tree structured Hidden Markov Model. The dependent variable is the prediction accuracy for a single prediction lead. In a first step, a one-way analysis of variance in conjunction with Tukey's post-hoc test demonstrated a significant superiority of the simple hierarchical Hidden Markov Model. In a second step, an additional two-way analysis of variance also indicated a significantly better prediction accuracy of the simple hierarchical Hidden Markov Model compared to the Hidden Markov Model, but the number of interaction cases also had a significant effect. Hence, the modeller has to take both factors--model topology and number of interaction cases--into account when designing syntactic user models with stochastic processes.

A web service reliability test method for C/S architecture software based on log analysis is presented in this paper. In this method, the software usage model is constructed automatically to describe the real situation on the users' access to the web service by Markov chain. The test cases are generated according to Random Walk and applied to software reliability test. In the experiment process, MTBF (focusing on server crash) was chosen to be the software reliability evaluation index. Through the testing and analysis of a real web software, MTBF obtained by testing result is similar to that from the realistic log, and the web service reliability test method is validated.

This paper presents a new approach, model theory approach, to small and medium scale transaction processing system (TPS) development. A TPS of this paper is an information system designed to process day-to-day business event data at operational level of an organization. The paper is not concerned with data base construction but with transaction processing. The model theory approach is not a software engineering approach but a systems theory approach. In the approach a model of the target system, which is called a user model, is constructed in set theory using a formal system structure of a TPS. The user model is, then, compiled into an extended Prolog (extProlog) model. The extProlog is an extension of Prolog to meet requirements for management information system development. On compilation a standardized user interface (UI) called internal UI is attached. The extProlog model with the internal UI is, then, executed under control of another standardized UI called an external UI. Implementation is an integral part of the approach. Because the UIs are designed for the formalized (abstract) structure of a TPS, they can be standardized and are provided as black box components to system development. Because a systems developer is required to only build a user model in set theory based on a model theoretic structure in the approach, it is called a model theory approach. Advantages of this approach are that it provides a theoretical structure to information systems development so that systems development can be made an engineering discipline, and facilitates rapid systems development.

The main purpose of this paper is to show how model-based techniques are used to efficiently control the generation of less complex test suites. By directed adjusting specific probability values in the usage profile of a Markov chain usage model it is relatively easy to generate abstract test suites for different user classes and test purposes in an automated approach. A stepwise refinement process for hierarchical Markov chain usage models and choosing appropriate test generation, respectively selection strategies can reduce the complexity of the resulting test suite significantly. By using proper tools, like the TestUS Testplayer even less experienced test engineers will be able to efficiently generate abstract test cases and to graphically assess quality characteristics of different test suites.

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion™, an embedded system providing helicopter landing assistance.

Food, energy, and water are basic needs that are quite important in livelihood sustainability. In most situations, their interactions make trade‐offs between these key areas of need challenging to attain. For the Food‐Energy‐Water Nexus (FEWN), there exists a need for appropriate decision‐making tools that may help to surmount such challenges. Multiobjective Optimization Approaches (MOAs) have emerged as potential solutions within this domain. The major components explored in this review include the objective functions, decision variables, and optimization techniques within MOAs. Despite their potential, few studies comprehensively address the key concepts of these components, the role of stakeholder involvement, the limitations of existing research, and the emerging technological opportunities that could enhance MOAs in the FEWN. The Preferred Reporting Items for Systematic Reviews and Meta‐Analyses protocol, was combined with Meta‐ethnography to select, review, and analyze the literature. The data synthesis was done by meta‐ethnography, finding key terms to categorize objective functions, decision variables, and optimization techniques and developing key concepts related to MOAs in the FEWN. Key terms for categorizing objective functions are maximizing economic, environmental, and social benefits and food, energy, and water security. For the decision variables, the key terms include availability, accessibility, and sufficiency, while the optimization techniques are Mathematical Programming and Metaheuristics. Besides, emerging technologies create substantial opportunities to improve the performance of MOAs. Therefore, there is a need to develop MOAs that integrate economic, environmental, and social aspects to support the sustainability of the FEWN.

In this paper, we study the existence theorems of systems of variational inclusions problems. As consequences of our results, we study existence theorems of systems of generalized vector quasi-equilibrium problems, mathematical program with systems of variational inclusion constraints, bilevel problem with systems of constraints.

We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user mod- els into precision. Continuous-time MP behaves like time- calibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based ver- sions of MP produce values that are directly comparable.\nWe show that it is possible to re-create average precision using specific user models and this helps in providing an ex- planation of Average Precision (AP) in terms of user mod- els more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list.\nFinally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.

The systematic generation of test cases from statistical usage models has been investigated recently for specific application domains, such as wireless communications or automotive applications. For Markov chain usage models, the expected usage of a hardware/software system is represented by transitions between usage states and a usage profile, meaning probability values that are attached to the state transitions.In this paper, we explain how to calculate the profile probabilities for the Markov chain usage model from a set of linear usage constraints and by optimizing a convex polyhedron that represents the constrained solution space. Comparing the computed probability distributions of our polyhedron approach with the maximum entropy technique, which is the main technique used so far, illustrates that our results are more obvious to the intented constraint semantics. In order to demonstrate the applicability of our approach, workflow testing of a complex RIS/PACS system in the medical domain was carried through and has provided promising results.