A dimensionality model approach to testing and improving software robustness

Commercial off-the-shelf (COTS) software products are being included in ever more complex and critical systems. There are clearly advantages to considering the use of COTS in such systems, but given the rigorous needs of such critical systems or subsystems, there are concerns about the suitability of COTS software for such applications. The AIAA’s Software Systems Technical Committee and the AIAA’s Computer Systems Technical Committee have undertaken an effort to produce a guidebook that captures issues related to the consideration and use of COTS products in these large, complex systems, with a special emphasis on those risks and risk mitigation approaches that relate to mission critical systems. This guidebook identifies a set of characteristics of mission critical systems that makes the consideration, selection and validation processes of COTS products (hardware, software, subsystems, etc.) an emerging factor in systems definition, development and acceptance. The guidebook discusses a large number of risk areas related to using COTS software products in mission critical systems and identifies various mitigation approaches to the risks. It includes detailed processes for selecting and evaluating the products and considerations for using COTS software products within the overall context of the software development life cycle standardized in IEEE/EIA 12207. Mission critical system characteristics such as reliability, safety, availability, maintainability and certification tend to influence whether or not COTS software should be considered for a given application. Once the suitability of COTS software has been determined, then it is possible that additional requirements may be placed on the product and/or the product’s vendor for such mission critical applications. Further, it is possible that certain system requirements and expectations may need to be modified because of the inclusion of COTS software products in that system. As COTS products continue to be considered as candidates for inclusion within mission critical systems, there are likely to be additional concerns and factors to emerge that will influence how both acquirers and suppliers decide if and/or when to use COTS products. Ongoing monitoring of this technology area seems to be warranted.

The use of commercial-off-the-shelf (COTS) software has become more and more important in state-of-the-art and state-of-the-practice software and system development. The use of COTS software promises faster time-to-market, reduced development costs, increased productivity, and the possibility for companies to focus on their own core competencies. At the same time, COTS software increases risks, such as the economic instability of the COTS software vendor, unknown quality properties of the COTS software in use, and side-effects of the COTS software on the final product. Typically, COTS-based development, in parallel with the traditional development cycle (e.g. waterfall, spiral), consists of four phases. The first phase (COTS assessment and selection) is the most crucial phase in the COTS-based cycle, since long-term decisions about which COTS software to use in a software system are made there. A late recognition that the wrong COTS software was used can become extremely costly for a software organization. This paper presents a repeatable, cost-efficient and systematic method for performing measurement-based COTS assessment and selection. Moreover, cost/benefit analysis results of two industrial pilot projects, in which the method was successfully applied, are presented.

Over the last decade the United States Government has significantly increased its use of commercial-off-the-shelf (COTS) software as stand-alone solutions and as components in safety-critical systems. This increased use stems from the realization that pre-existing software products can be a means of lowering development costs, shortening development time, and keeping pace with the changing software market. The Federal government, particularly with regards to safety-critical systems, has found that COTS software is currently not plug-and-play, has significant tradeoffs, and usually contains a cradle-to-grave dependence on the software manufacturer. Unfortunately, there is currently no standard best commercial practice with regard to the acceptance of COTS software. Ad hoc attempts to apply standards for commercial software acceptance have been based upon subjective criteria and have proven to be imprecise and prone to error. Moreover, acceptability in safety-critical systems generally demands analysis of source code. Many software vendors, however, have chosen to keep all such source code proprietary due to liability and intellectual property concerns. When the source code is not available, there is very little that can be done to ensure the safety, reliability, and integrity of the software. This paper discusses the COTS Score, an approach that aids in determining the acceptability of COTS software. The process involves the application of predictive techniques used for financial credit scoring to the COTS domain. The methodology addresses the issue of acceptability by incorporating both functional and environmental software measures related to reliability, compatibility, certifiability, obsolescence, and life cycle including trade-off analyses. This approach satisfies NASA and ISO 9001 requirements to define an acceptance procedure for COTS software.

Commercial off-the-shelf (COTS) software has become a crucial component of many of today’s mission- and safety-critical products. As these products have proliferated internationally, global demand for software safety and performance quality has increased. Unfortunately, the current state-of-the-art in software assessment technology has not matured to the point where well-established and widely accepted schemes for evaluating COTS software quality have been presented. Without insight into the source code, COTS software users depend on a subjective assessment of attributes described by performance metrics, behavioral descriptions, and/or historical product data. Published techniques that attempt to formalize the above approach are often less than successful due to ill-defined and subjective metrics, and the naive assumption of independence between attributes. Recently, software standards have been seen as a means of improving the evaluation process by providing an acknowledged criterion for software metric comparison. Given the stated issues, is it possible to develop a standards-based dependency model of well-defined attributes that does not depend on naive assumptions? The answer to this question is yes. This paper describes a process that can be used to extract acyclic dependency models, if they exist, from various software product quality standards. Application of the extraction process will be used to reveal the attribute dependency model of the international software standard ISO/IEC 9126-1. Not only can this process be used for COTS software evaluation, but it can also be used by standards making bodies to clarify the reasoning, justifications and causal implications they employ when developing standards.

This paper presents development of a testbed to support collaboration among a set of distributed engineers to reverse, re-engineer, and fast manufacture (RRF) components for aging aircrafts. This testbed allows a geographically distributed team to work on a design task both synchronously and asynchronously. Commercial off-the-shelf (COTS) software tools and equipment that support the RRF process have been identified, evaluated, and tested. An integration framework has also been developed and employed to create an RRF testbed. This testbed constructed using COTS software and equipment supports three major engineering tasks: the reverse engineering that supports recovering of technical data from worn sample parts, re-engineering that alters design for better performance or lower cost, and fast prototyping that incorporates advanced manufacturing technologies to produce functional or physical prototype of the part in small quantity in a short turnaround time. Traditional reverse engineering projects are ad-hoc, usually takes relatively long time to generate solid models from physical parts. The developed testbed utilizes commercial software to accomplish reverse engineering, re-engineer and fast manufacture using a systematic approach. A number of examples obtained from logistics centers have been employed to illustrate and demonstrate the capabilities established in the RRF testbed.

The Software-as-a-Service (SaaS) model is a novel way of delivering software applications. In this paper, we present an analytical model to study the competition between the SaaS and the traditional COTS (Commercial off-the-shelf) software. The main research goal is to analyze the pricing strategy of the SaaS in a competitive setting. The model captures the most salient differences between the SaaS and COTS, including their distinct pricing structures, user initial setup costs, system customization levels, and delivery channels. We find that the two could coexist in a competitive market in the long run, and more importantly, we show how the SaaS could gradually take over the whole market even when its quality is inferior. Surprisingly, our analysis shows that the SaaS should raise (reduce) its prices when its software quality declines (increases) over time (in the relative sense).Keywordsthe SaaS business modelpricing strategycompetitionthe COTS software

This paper provides an overview of network architecture, network building block, and network application issues involved in the insertion of Fibre Channel (FC) Commercial Off-The-Shelf (COTS) technology into avionics systems. At the network architecture level, FC supports a simple and low cost Arbitrated Loop (FC-AL) topology which is extensible to fabric switch architectures. A COTS FC testbed is described which has been constructed to characterize FC-AL performance for defined domain-specific avionics scenarios. The testbed is also being used to prototype an avionics multimedia network via integration of FC COTS network controllers and software with streaming video encoders and decoders. Based on lessons learned from the testbed activity and other FC system architecture insertion studies, a set of FC based domain extended products suitable for insertion into avionics systems is identified. Finally future activities required to continue toward a fully capable FC Avionics Multimedia Network (AMN) are described.

Technology insertion into an existing system or subsystem requires careful planning and evaluation. Because the technology is new, there is often limited information upon which to base decisions. Further, decisions about technology insertion are typically made in the very early stages of a project when detailed information is sparse. Under these circumstances, the available information is typically linguistic, and fuzzy methods can be used to significant advantage in this environment. The implementation discussed in this paper employs an Excel spread sheet for user interface, and it employs COTS (Commercial Off the Shelf Technology) software for an inference engine. Heuristic based models are developed to evaluate risk in technology insertion. Risk elements include: budget risk, schedule risk, and performance risk. Results with an example problem are described to illustrate the fuzzy logic method, and conclusions are drawn regarding the advantages of this technique.

This paper presents a software tool for integrating a child-friendly computer system based on commercial off-the-shelf (COTS) components. The effective selection of COTS components, which meet a child’s requirements and expectations, is a non-trivial and challenging optimization problem. However, many published papers consider the functional requirements while ignoring usability requirements. The functional requirements are concerned with what the computer should be able to do, whereas the usability requirements are concerned with the extent to which the child is able to learn effectively and efficiently throughout the COTS based computer. In this paper, the authors propose an iterative five-task selection and integration of COTS process, including both hardware devices and software modules, to be automated. The core of the automated tool is employing Simulated Annealing (SA) to search the design space to match, select, and integrate COTS components with a maximal satisfaction while neither exceeding a given budget nor violating child and performance constraints. A Monte Carlo simulator was utilized to evaluate the goodness of the COTS based computer design. Computational results based on building a computer for a child handwriting e-learning application show feasibility of SPACots in finding a solution satisfying all constraints while reducing the cost by 58%.

This paper presents a software tool for integrating a child-friendly computer system based on commercial off-the-shelf (COTS) components. The effective selection of COTS components, which meet a child’s requirements and expectations, is a non-trivial and challenging optimization problem. However, many published papers consider the functional requirements while ignoring usability requirements. The functional requirements are concerned with what the computer should be able to do, whereas the usability requirements are concerned with the extent to which the child is able to learn effectively and efficiently throughout the COTS based computer. In this paper, the authors propose an iterative five-task selection and integration of COTS process, including both hardware devices and software modules, to be automated. The core of the automated tool is employing Simulated Annealing (SA) to search the design space to match, select, and integrate COTS components with a maximal satisfaction while neither exceeding a given budget nor violating child and performance constraints. A Monte Carlo simulator was utilized to evaluate the goodness of the COTS based computer design. Computational results based on building a computer for a child handwriting e-learning application show feasibility of SPACots in finding a solution satisfying all constraints while reducing the cost by 58%.

To date, most studies on the robustness of operating system software have focused on Unix based systems. The paper develops a methodology and architecture for performing intelligent black box analysis of software that runs on the Windows NT platform. The goals of the research are three fold: first, to develop intelligent robustness testing techniques for commercial Off-The-Shelf (COTS) software; second, to benchmark the robustness of NT software in handling anomalous events; and finally, to identify robustness gaps to permit fortification for fault tolerance. The random and intelligent data design library environment (RIDDLE) is a tool for analyzing operating system software, system utilities, desktop applications, component based software, and network services. RIDDLE was used to assess the robustness of native Windows NT system utilities as well as Win32 ports of the GNU utilities. Experimental results comparing the relative performance of the ported utilities versus the native utilities are presented.

Various reference models have been presented for purchasing software products and information processing systems. Moreover, such capability maturity models as SA-CMM, SPICE and BOOTSTRAP can be used to assess the maturity of software acquisition. However, these models do not specifically aim at supporting the acquisition of commercial off-the-shelf (COTS) software components. They are more intended for end-users to help making investments in software packages. Yet, the models can be used as a starting point to define a COTS software component acquisition (CSCA) process framework. This paper discusses experiences from an industriacase study, where requirements for a CSCA process were gathered, and a process framework developed for a telecommunication company based on the requirements. The existing reference and capability models for software acquisition were used to launch the work but the process framework was defined based mostly on the particular needs of the company, taking the key characteristics of COTS software components into account.

Mission-critical system designers may have to use a commercial off-the-shelf (COTS) approach to reduce costs and shorten development time, even though COTS software components may not specifically be designed for robust operation. Automated testing can assess component robustness without sacrificing the advantages of a COTS approach. This paper describes the Ballista methodology for scalable, portable, automated robustness testing of component interfaces. An object-oriented approach based on parameter data types rather than component functionality essentially eliminates the need for function-specific test scaffolding. A full-scale implementation that automatically tests the robustness of 233 operating system software components has been ported to ten POSIX systems. Between 42% and 63% of components tested had robustness problems, with a normalized failure rate ranging from 10% to 23% of tests conducted. Robustness testing could be used by developers to measure and improve robustness, or by consumers to compare the robustness of competing COTS component libraries.

The nested relational model extends the flat relational model by relaxing the first normal form assumption in order to allow the modeling of complex objects. Much of the previous work on the nested relational model has concentrated on defining the data structures and query language for the model. The work done on integrity constraints in nested relations has mainly focused on characterizing subclasses of nested relations and defining normal forms for nested relations with certain desirable properties. In this paper we define the semantics of nested relations, which may contain null values, in terms of integrity constraints, called null extended data dependencies , which extend functional dependencies and join dependencies encountered in flat relational database theory. We formalize incomplete information in nested relations by allowing only one unmarked generic null value , whose semantics we do not further specify. The motivation for the choice of a generic null is our desire to investigate only fundamental semantics which are common to all unmarked null types. This lead us to define a preorder on nested relations, which allows us to measure the relative information content of nested relations. We also define a procedure, called the extended chase procedure , for testing satisfaction of null extended data dependencies and for making inferences by using these null extended data dependencies. The extended chase procedure is shown to generalize the classical chase procedure, which is of major importance in flat relational database theory. As a consequence of our approach we are able to capture the novel notion of losslessness in nested relations, called herein null extended lossless decomposition . Finally, we show that the semantics of nested relations are a natural extension of the semantics of flat relations.

Commercial-off-the-shelf (COTS) many-core processors offer the performance needed for computational-intensive safety-critical real-time applications such as autonomous driving. However, these consumer-grade many-core processors are increasingly susceptible to faults because of their highly integrated design.In this paper, we present a fault-tolerance framework that eases the usage of COTS many-core processors for safety-critical applications. Our framework employs an adaptable software-based fault-tolerance mechanism that combines N Modular Redundancy (NMR) with a repair process and a rejuvenating round robin voting scheme. A Stochastic Activity Network (SAN) model of the fault-tolerance mechanism allows the framework to adapt the parameters of the mechanism such that a specified target availability is achieved with minimum overhead. Experiments on a cycle-accurate simulator empirically prove the correctness of the SAN model and evaluate the overhead of the framework.