A method for efficient measurement-based COTS assessment and selection method description and evaluation results

Commercial off-the-shelf (COTS) software products are being included in ever more complex and critical systems. There are clearly advantages to considering the use of COTS in such systems, but given the rigorous needs of such critical systems or subsystems, there are concerns about the suitability of COTS software for such applications. The AIAA’s Software Systems Technical Committee and the AIAA’s Computer Systems Technical Committee have undertaken an effort to produce a guidebook that captures issues related to the consideration and use of COTS products in these large, complex systems, with a special emphasis on those risks and risk mitigation approaches that relate to mission critical systems. This guidebook identifies a set of characteristics of mission critical systems that makes the consideration, selection and validation processes of COTS products (hardware, software, subsystems, etc.) an emerging factor in systems definition, development and acceptance. The guidebook discusses a large number of risk areas related to using COTS software products in mission critical systems and identifies various mitigation approaches to the risks. It includes detailed processes for selecting and evaluating the products and considerations for using COTS software products within the overall context of the software development life cycle standardized in IEEE/EIA 12207. Mission critical system characteristics such as reliability, safety, availability, maintainability and certification tend to influence whether or not COTS software should be considered for a given application. Once the suitability of COTS software has been determined, then it is possible that additional requirements may be placed on the product and/or the product’s vendor for such mission critical applications. Further, it is possible that certain system requirements and expectations may need to be modified because of the inclusion of COTS software products in that system. As COTS products continue to be considered as candidates for inclusion within mission critical systems, there are likely to be additional concerns and factors to emerge that will influence how both acquirers and suppliers decide if and/or when to use COTS products. Ongoing monitoring of this technology area seems to be warranted.

Agile methods are proposed nowadays as a way to support software systems procurement. Most of the existing proposals such as eXtreme programming or scrum seem to conceive software procurement as an exercise of software development. However, a great deal of software systems are commercial off-the-shelf (COTS)-based systems, in which the focus changes from bespoke software development to COTS selection and integration. Many proposals for COTS selection have been issued and therefore one may wonder how do they behave from the agile point of view. In this paper, we study the agile principles in the context of COTS selection and we analyze some of the most widespread existing methods. As a result, we identify some practices that would help in making COTS selection processes more agile.

Over the last decade the United States Government has significantly increased its use of commercial-off-the-shelf (COTS) software as stand-alone solutions and as components in safety-critical systems. This increased use stems from the realization that pre-existing software products can be a means of lowering development costs, shortening development time, and keeping pace with the changing software market. The Federal government, particularly with regards to safety-critical systems, has found that COTS software is currently not plug-and-play, has significant tradeoffs, and usually contains a cradle-to-grave dependence on the software manufacturer. Unfortunately, there is currently no standard best commercial practice with regard to the acceptance of COTS software. Ad hoc attempts to apply standards for commercial software acceptance have been based upon subjective criteria and have proven to be imprecise and prone to error. Moreover, acceptability in safety-critical systems generally demands analysis of source code. Many software vendors, however, have chosen to keep all such source code proprietary due to liability and intellectual property concerns. When the source code is not available, there is very little that can be done to ensure the safety, reliability, and integrity of the software. This paper discusses the COTS Score, an approach that aids in determining the acceptability of COTS software. The process involves the application of predictive techniques used for financial credit scoring to the COTS domain. The methodology addresses the issue of acceptability by incorporating both functional and environmental software measures related to reliability, compatibility, certifiability, obsolescence, and life cycle including trade-off analyses. This approach satisfies NASA and ISO 9001 requirements to define an acceptance procedure for COTS software.

In the process of selecting Commercial Off-The-Shelf (COTS) products, it is inevitable to encounter mismatches between system requirements and COTS products. These mismatches occur as a result of an excess or shortage of the COTS attributes. This paper proposes a decision support approach, called MiHOS (Mismatch Handling for COTS Selection), that aims at addressing COTS mismatches during and after the selection process. MiHOS can be integrated with existing COTS selection methods at two points: (1) For evaluating COTS candidates: MiHOS estimates the anticipated fitness of the candidates if their mismatches are resolved. This helps to base our COTS selection decisions on the fitness that the candidates will eventually have if selected. (2) Mismatch resolution after selecting a COTS product: MiHOS suggests alternative plans for resolving the most appropriate mismatches using suitable actions, such that the most important risk, technical, and resource constraints are met. A case-study is used to illustrate MiHOS and to discuss its added value.

Commercial off-the-shelf (COTS) software has become a crucial component of many of today’s mission- and safety-critical products. As these products have proliferated internationally, global demand for software safety and performance quality has increased. Unfortunately, the current state-of-the-art in software assessment technology has not matured to the point where well-established and widely accepted schemes for evaluating COTS software quality have been presented. Without insight into the source code, COTS software users depend on a subjective assessment of attributes described by performance metrics, behavioral descriptions, and/or historical product data. Published techniques that attempt to formalize the above approach are often less than successful due to ill-defined and subjective metrics, and the naive assumption of independence between attributes. Recently, software standards have been seen as a means of improving the evaluation process by providing an acknowledged criterion for software metric comparison. Given the stated issues, is it possible to develop a standards-based dependency model of well-defined attributes that does not depend on naive assumptions? The answer to this question is yes. This paper describes a process that can be used to extract acyclic dependency models, if they exist, from various software product quality standards. Application of the extraction process will be used to reveal the attribute dependency model of the international software standard ISO/IEC 9126-1. Not only can this process be used for COTS software evaluation, but it can also be used by standards making bodies to clarify the reasoning, justifications and causal implications they employ when developing standards.

This paper presents development of a testbed to support collaboration among a set of distributed engineers to reverse, re-engineer, and fast manufacture (RRF) components for aging aircrafts. This testbed allows a geographically distributed team to work on a design task both synchronously and asynchronously. Commercial off-the-shelf (COTS) software tools and equipment that support the RRF process have been identified, evaluated, and tested. An integration framework has also been developed and employed to create an RRF testbed. This testbed constructed using COTS software and equipment supports three major engineering tasks: the reverse engineering that supports recovering of technical data from worn sample parts, re-engineering that alters design for better performance or lower cost, and fast prototyping that incorporates advanced manufacturing technologies to produce functional or physical prototype of the part in small quantity in a short turnaround time. Traditional reverse engineering projects are ad-hoc, usually takes relatively long time to generate solid models from physical parts. The developed testbed utilizes commercial software to accomplish reverse engineering, re-engineer and fast manufacture using a systematic approach. A number of examples obtained from logistics centers have been employed to illustrate and demonstrate the capabilities established in the RRF testbed.

The Software-as-a-Service (SaaS) model is a novel way of delivering software applications. In this paper, we present an analytical model to study the competition between the SaaS and the traditional COTS (Commercial off-the-shelf) software. The main research goal is to analyze the pricing strategy of the SaaS in a competitive setting. The model captures the most salient differences between the SaaS and COTS, including their distinct pricing structures, user initial setup costs, system customization levels, and delivery channels. We find that the two could coexist in a competitive market in the long run, and more importantly, we show how the SaaS could gradually take over the whole market even when its quality is inferior. Surprisingly, our analysis shows that the SaaS should raise (reduce) its prices when its software quality declines (increases) over time (in the relative sense).Keywordsthe SaaS business modelpricing strategycompetitionthe COTS software

This paper provides an overview of network architecture, network building block, and network application issues involved in the insertion of Fibre Channel (FC) Commercial Off-The-Shelf (COTS) technology into avionics systems. At the network architecture level, FC supports a simple and low cost Arbitrated Loop (FC-AL) topology which is extensible to fabric switch architectures. A COTS FC testbed is described which has been constructed to characterize FC-AL performance for defined domain-specific avionics scenarios. The testbed is also being used to prototype an avionics multimedia network via integration of FC COTS network controllers and software with streaming video encoders and decoders. Based on lessons learned from the testbed activity and other FC system architecture insertion studies, a set of FC based domain extended products suitable for insertion into avionics systems is identified. Finally future activities required to continue toward a fully capable FC Avionics Multimedia Network (AMN) are described.

Technology insertion into an existing system or subsystem requires careful planning and evaluation. Because the technology is new, there is often limited information upon which to base decisions. Further, decisions about technology insertion are typically made in the very early stages of a project when detailed information is sparse. Under these circumstances, the available information is typically linguistic, and fuzzy methods can be used to significant advantage in this environment. The implementation discussed in this paper employs an Excel spread sheet for user interface, and it employs COTS (Commercial Off the Shelf Technology) software for an inference engine. Heuristic based models are developed to evaluate risk in technology insertion. Risk elements include: budget risk, schedule risk, and performance risk. Results with an example problem are described to illustrate the fuzzy logic method, and conclusions are drawn regarding the advantages of this technique.

Software robustness problems may hinder the use of Commercial Off-The-Shelf (COTS) software modules and legacy software modules in mission-critical and safety-critical applications. This research focuses on hardening COTS and legacy software modules against robustness failures triggered by exceptional inputs. An automated approach is presented that is capable of identifying the triggers of the robustness failures. A fault model-the dimensionality model-is used to guide analysis. An experiment is described which demonstrates the feasibility of automating the process of analyzing failure causes and hardening against certain data types in POSIX function calls, for example, NULL pointer values and scalar data types such as INT and FLOAT. The final goal of this research is to provide users a tool to harden COTS and legacy software modules automatically.

The use of commercial-off-the-shelf (COTS) products provides the Department of Defence with benefits such as reduced costs and time associated with the research and development of new military defence equipment. However, acquisition organizations face several challenges in the evaluation and selection process of the correct COTS products that comply with the requirements of a client such as the Department of Defence. The purpose of this research was to present a systematic customized framework for the evaluation and selection of COTS products for use as military defence equipment. The phases involved in the COTS evaluation and selection processes, challenges faced by the evaluators and mitigation to overcome the problem were identified through the reviewed literature and evidence collected from a case study. A framework for the selection of COTS products for use as military defence equipment is proposed with the aim to eliminate the lack of support by the acquisition organization to the client during the user requirements definition phase.

The use of Commercial Off‐The‐Shelf (COTS) products in the software development process requires the evaluation of existing COTS products, and then selecting the one that best fits system requirements. In this process, it is inevitable to encounter mismatches between COTS features and system requirements. Mismatches occur as a result of an excess or shortage of COTS capabilities. Many of these mismatches are resolved after selecting a COTS product. Existing COTS‐selection approaches fail to properly consider these mismatches. This article presents MiHOS (Mismatch Handling for COTS Selection), an approach that aims at addressing mismatches while considering limited resources. MiHOS can be integrated with existing COTS‐selection methods at two points: (i)When evaluating COTS candidates in order to estimate the anticipated fitness of the candidates if their mismatches are resolved. This helps to base our COTS‐selection decisions on the fitness that the COTS candidates will eventually have if selected. (ii) After selecting a COTS product in order to plan the resolution of the most appropriate mismatches using suitable actions, such that the most important risk, technical, and resource constraints are met. A case study from the e‐services domain is used to illustrate the method and to discuss its added value. Copyright © 2008 John Wiley & Sons, Ltd.

Various reference models have been presented for purchasing software products and information processing systems. Moreover, such capability maturity models as SA-CMM, SPICE and BOOTSTRAP can be used to assess the maturity of software acquisition. However, these models do not specifically aim at supporting the acquisition of commercial off-the-shelf (COTS) software components. They are more intended for end-users to help making investments in software packages. Yet, the models can be used as a starting point to define a COTS software component acquisition (CSCA) process framework. This paper discusses experiences from an industriacase study, where requirements for a CSCA process were gathered, and a process framework developed for a telecommunication company based on the requirements. The existing reference and capability models for software acquisition were used to launch the work but the process framework was defined based mostly on the particular needs of the company, taking the key characteristics of COTS software components into account.

Mission-critical system designers may have to use a commercial off-the-shelf (COTS) approach to reduce costs and shorten development time, even though COTS software components may not specifically be designed for robust operation. Automated testing can assess component robustness without sacrificing the advantages of a COTS approach. This paper describes the Ballista methodology for scalable, portable, automated robustness testing of component interfaces. An object-oriented approach based on parameter data types rather than component functionality essentially eliminates the need for function-specific test scaffolding. A full-scale implementation that automatically tests the robustness of 233 operating system software components has been ported to ten POSIX systems. Between 42% and 63% of components tested had robustness problems, with a normalized failure rate ranging from 10% to 23% of tests conducted. Robustness testing could be used by developers to measure and improve robustness, or by consumers to compare the robustness of competing COTS component libraries.

This paper defines a process of predictive approach to evaluate the maintainability of a Commercial Off-the-Shelf (COTS)-based system(CBS) by analyzing the complexity of the deployment of the system. The approach integrates architectural dependencies and the system’s concept of operations to derive a network-based representation of the software system. A greater understanding of the deployment complexity is gained by using a Design Structure Matrix (DSM) to determine the number of architectural dependencies on a COTS product, or in-degree, for each COTS product in the system. The arithmetic mean of the in-degree for all nodes in the system is then compared with the perceived effort to maintain the system. The resultant measure – the in-degree mean – is useful in evaluating the maintainability of the operational system while the system is being designed and throughout its lifetime. Architects can use the approach to assist in COTS product selection and to make product trades to optimize the maintainability of the system. Integrators can use the approach to optimize product deployment and to determine the upgrade strategy for deployment. Finally, maintenance engineers can use the approach to estimate the effort required to maintain the system and to identify areas in which extensive product expertise is required. Because the approach requires only basic information about the system, it can be applied early in the design process and used until the system is decommissioned.