Automated robustness testing of off-the-shelf software components

Various reference models have been presented for purchasing software products and information processing systems. Moreover, such capability maturity models as SA-CMM, SPICE and BOOTSTRAP can be used to assess the maturity of software acquisition. However, these models do not specifically aim at supporting the acquisition of commercial off-the-shelf (COTS) software components. They are more intended for end-users to help making investments in software packages. Yet, the models can be used as a starting point to define a COTS software component acquisition (CSCA) process framework. This paper discusses experiences from an industriacase study, where requirements for a CSCA process were gathered, and a process framework developed for a telecommunication company based on the requirements. The existing reference and capability models for software acquisition were used to launch the work but the process framework was defined based mostly on the particular needs of the company, taking the key characteristics of COTS software components into account.

Commercial off-the-shelf (COTS) software products are being included in ever more complex and critical systems. There are clearly advantages to considering the use of COTS in such systems, but given the rigorous needs of such critical systems or subsystems, there are concerns about the suitability of COTS software for such applications. The AIAA’s Software Systems Technical Committee and the AIAA’s Computer Systems Technical Committee have undertaken an effort to produce a guidebook that captures issues related to the consideration and use of COTS products in these large, complex systems, with a special emphasis on those risks and risk mitigation approaches that relate to mission critical systems. This guidebook identifies a set of characteristics of mission critical systems that makes the consideration, selection and validation processes of COTS products (hardware, software, subsystems, etc.) an emerging factor in systems definition, development and acceptance. The guidebook discusses a large number of risk areas related to using COTS software products in mission critical systems and identifies various mitigation approaches to the risks. It includes detailed processes for selecting and evaluating the products and considerations for using COTS software products within the overall context of the software development life cycle standardized in IEEE/EIA 12207. Mission critical system characteristics such as reliability, safety, availability, maintainability and certification tend to influence whether or not COTS software should be considered for a given application. Once the suitability of COTS software has been determined, then it is possible that additional requirements may be placed on the product and/or the product’s vendor for such mission critical applications. Further, it is possible that certain system requirements and expectations may need to be modified because of the inclusion of COTS software products in that system. As COTS products continue to be considered as candidates for inclusion within mission critical systems, there are likely to be additional concerns and factors to emerge that will influence how both acquirers and suppliers decide if and/or when to use COTS products. Ongoing monitoring of this technology area seems to be warranted.

Over the last decade the United States Government has significantly increased its use of commercial-off-the-shelf (COTS) software as stand-alone solutions and as components in safety-critical systems. This increased use stems from the realization that pre-existing software products can be a means of lowering development costs, shortening development time, and keeping pace with the changing software market. The Federal government, particularly with regards to safety-critical systems, has found that COTS software is currently not plug-and-play, has significant tradeoffs, and usually contains a cradle-to-grave dependence on the software manufacturer. Unfortunately, there is currently no standard best commercial practice with regard to the acceptance of COTS software. Ad hoc attempts to apply standards for commercial software acceptance have been based upon subjective criteria and have proven to be imprecise and prone to error. Moreover, acceptability in safety-critical systems generally demands analysis of source code. Many software vendors, however, have chosen to keep all such source code proprietary due to liability and intellectual property concerns. When the source code is not available, there is very little that can be done to ensure the safety, reliability, and integrity of the software. This paper discusses the COTS Score, an approach that aids in determining the acceptability of COTS software. The process involves the application of predictive techniques used for financial credit scoring to the COTS domain. The methodology addresses the issue of acceptability by incorporating both functional and environmental software measures related to reliability, compatibility, certifiability, obsolescence, and life cycle including trade-off analyses. This approach satisfies NASA and ISO 9001 requirements to define an acceptance procedure for COTS software.

This paper describes an approach to the design of a wrapper for self-managing COTS (commercial off-the-shelf) components. Each wrapper for COTS components encapsulates the properties of self-management — detection, reconfiguration, and repair. A COTS component deals with the application perspectives, whereas the wrapper handles the self-management perspectives, separately from the application perspectives. Each wrapper for self-managing COTS components is structured into several objects in support of detection, reconfiguration, and repair of the anomalous COTS components. The approach suggested in this paper is applied to the distributed elevator system consisting of multiple COTS components.

Safety-critical systems and in particular mixed-criticality systems require spatial and temporal separation for their hosted applications and functionalities. Additional constraints are using Commercial Off -- The -- Shelf (COTS) components, portability and determinism. These items are required for economic success for products with low piece numbers and long life-cycles like aircraft. Available embedded processors lack means for spatial separation of Input/Output (I/O) components like an Input/Output Memory Management Unit (IOMMU). The objective of this paper is to provide spatial separation for I/O in COTS mixed-criticality embedded real-time systems like avionics with minimum possible impact on performance (transfer time, transfer rate, Central Processing Unit (CPU) usage). The three main contributions of this paper are: (1) The presented Input/Output Memory Protection Unit (IOMPU) enables to upgrade spatial separation for I/O to a system by using COTS components and Non-Transparent Bridge (NTB) technology. In addition, the IOMPU concept is compatible with existing temporal separation solutions. (2) The paper shows a prototype implementation and a potential use case in context of hardware-based I/O virtualization. (3) The evaluation in this paper demonstrates that the IOMPU concept is practically applicable. The performance overhead (transfer time, transfer rate) is below 0.88%, which is almost negligible, particularly compared to state-of-the-art software-based solutions.

Budget and schedule savings are the driving factors for the adoption of commercial-off-the-shelf (COTS) software components by software development organizations. The reliance on COTS components has lead to component-based development (CBD) software systems and introduced changes to the software development process and hence software project management responsibilities and roles. This paper introduces a general framework discussion of essential management aspects for CBD, focusing on COTS. Particularly, stakeholders, requirements, component selection and architecture management issues are discussed from different angles. Some CBD management guidelines and best practices for these aspects are outlined in the conclusion. In addition, CBD management challenges are drawn along with some suggestions in the conclusion section.

This paper presents a software tool for integrating a child-friendly computer system based on commercial off-the-shelf (COTS) components. The effective selection of COTS components, which meet a child’s requirements and expectations, is a non-trivial and challenging optimization problem. However, many published papers consider the functional requirements while ignoring usability requirements. The functional requirements are concerned with what the computer should be able to do, whereas the usability requirements are concerned with the extent to which the child is able to learn effectively and efficiently throughout the COTS based computer. In this paper, the authors propose an iterative five-task selection and integration of COTS process, including both hardware devices and software modules, to be automated. The core of the automated tool is employing Simulated Annealing (SA) to search the design space to match, select, and integrate COTS components with a maximal satisfaction while neither exceeding a given budget nor violating child and performance constraints. A Monte Carlo simulator was utilized to evaluate the goodness of the COTS based computer design. Computational results based on building a computer for a child handwriting e-learning application show feasibility of SPACots in finding a solution satisfying all constraints while reducing the cost by 58%.

This paper presents a software tool for integrating a child-friendly computer system based on commercial off-the-shelf (COTS) components. The effective selection of COTS components, which meet a child’s requirements and expectations, is a non-trivial and challenging optimization problem. However, many published papers consider the functional requirements while ignoring usability requirements. The functional requirements are concerned with what the computer should be able to do, whereas the usability requirements are concerned with the extent to which the child is able to learn effectively and efficiently throughout the COTS based computer. In this paper, the authors propose an iterative five-task selection and integration of COTS process, including both hardware devices and software modules, to be automated. The core of the automated tool is employing Simulated Annealing (SA) to search the design space to match, select, and integrate COTS components with a maximal satisfaction while neither exceeding a given budget nor violating child and performance constraints. A Monte Carlo simulator was utilized to evaluate the goodness of the COTS based computer design. Computational results based on building a computer for a child handwriting e-learning application show feasibility of SPACots in finding a solution satisfying all constraints while reducing the cost by 58%.

The purpose of the project was to develop the processes for using commercial off-the-shelf (COTS) parts for WR production and to put in place a system for implementing the data management tools required to disseminate, store, track procurement, and qualify vendors. Much of the effort was devoted to determining if the use of COTS parts was possible. A basic question: How does the Nuclear Weapons Complex (NWC) begin to use COTS in the weapon Stockpile Life Extension Programs with high reliability, affordability, while managing risk at acceptable levels? In FY00, it was determined that a certain weapon refurbishment program could not be accomplished without the use of COTS components. The elements driving the use of COTS components included decreased cost, greater availability, and shorter delivery time. Key factors that required implementation included identifying the best suppliers and components, defining life cycles and predictions of obsolescence, testing the feasibility of using COTS components with a test contractor to ensure capability, as well as quality and reliability, and implementing the data management tools required to disseminate, store, track procurement, and qualify vendors. The primary effort of this project then was to concentrate on the risks involved in the use of COTS and address the issues of part and vendor selection, procurement and acceptance processes, and qualification of the parts via part and sample testing. The Enterprise Component Information System (eCIS) was used to manage the information generated by the COTS process. eCIS is a common interface for both the design and production of NWC components and systems integrating information between SNL National Laboratory (SNL) and the Kansas City Plant (KCP). The implementation of COTS components utilizes eCIS from part selection through qualification release. All part related data is linked across an unclassified network for access by both SNL and KCP personnel. The system includes not only NWC part information but also includes technical reference data for over 25 Million electronic and electromechanical commercial and military parts via a data subscription. With the capabilities added to the system through this project, eCIS provides decision support, parts list/BOM analysis, editing, tracking, workflows, reporting, and history/legacy information integrating manufacturer reference, company technical, company business, and design data.

: Small unmanned systems provide great utility to military applications due to their portable and expendable design. These systems are, however, costly to develop, produce, and maintain, making it desirable to integrate available commercial off the shelf (COTS) components. This research investigates the integration of COTS components through the development of a modular unified command and control (C2) architecture for heterogeneous and homogeneous vehicle teams to accomplish formation flocking and communication relay scenarios. A vehicle agnostic architecture was developed to be applied across different vehicle platforms, different vehicle combination, and different cooperative missions. COTS components consisting primarily of open source hardware and software were integrated and tested based on the positional accuracy, precision, and other qualitative measures. The resulting system successfully demonstrated formation flocking in three of four vehicle combinations, with the forth still demonstrating leader follower behaviors. The system achieved at best a mean positional error of 0.99m, a standard deviation of 0.44m, and a DRMS of 0.59m....

Component-Based Software Development (CBSD) provides a high efficient and low cost way to construct software systems by integrating reusable software components. Although CBSD has already become a widely accepted paradigm, it is still beyond possibility to assemble components easily from commercial-off-the-shelf (COTS) components into one application system. A variety of mismatches between components often impede the integration of COTS components, and component adaptation is becoming a key problem in Component-Based Software Engineering (CBSE). Aiming at this requirement, this paper presents a self-adaptive software architecture (model-based) approach for supporting seamless integration of COTS components. Specifically, we first propose a self-adaptive software architecture model, and then we discuss and exemplify how to eliminate the mismatches between heterogeneous COTS components based on this model. A simplified on-line shopping system is referred throughout the paper to illustrate our approach.

Radiation effects research is crucial as it defines risk to both human bodies and spacecraft. Employing radiation-hardened products is one way to mitigate radiation effects on in-orbit systems. However, radiation effects prohibit most of the state-of-the-art commercial off-the-shelf (COTS) technologies from use in space. Furthermore, radiation effects on software components are less studied compared to hardware components. In this work, we introduce a simulation tool that simulates and performs post-simulation analysis of the impact of radiation effects on schedulability of the software task sets that execute on COTS system-on-chip (SoC) platforms within in-orbit systems. In order to provide a meaningful verification environment, single-event effects (SEEs) are introduced as aleatory disturbances characterized by probability distribution of occurrence using their predefined models. The tool supports interoperability with several other tools as it uses the extensible markup language (XML) model files for input and output, i.e., for importing input task sets and radiation effects and exporting the simulation and analysis results. The proposed tool is extensively by running simulations using a use case of an in-orbit onboard monitoring system.

Dealing with Commercial off-the-shelf (COTS) com- ponents is a daily business for avionic system manufacturers. They are necessary ingredients for hardware designs, but are not built in accordance with the avionics consensus standard DO- 254 for Airborne Electronic Hardware (AEH) design. Especially for complex COTS hardware components used in safety critical AEH, like Microcontroller Units (MCUs), additional assurance activities have to be performed. All of them together shall form a convincing confident, that the hardware is safe in its intended operation environment. The focus of DO-254 is one approach called Design Assurance (DA). Its aim is to reduce design errors by adherence of prescribed process objectives for the entire design life cycle. The effort for certain COTS assurance activities could be reduced if it is possible to demonstrate, that the COTS design process is based on similar effective design process guide- lines to minimize desgin errors. In the last years, semiconductor manufacturers released safety MCUs in compliance to the ISO 26262 standard, dedicated for the development of functional safe automotive systems. These products are COTS components in the sense of avionics, but they are also developed according to a process that focuses on reduction of design errors. In this paper an evaluation is performed to figure out if the ISO 26262 prescribes a similar DA approach as the DO-254, in order to reduce the COTS assurance effort for coming avionic systems.

Traditionally, the Navy has developed systems which distribute time-sensitive data using specialized components such as dedicated point-to-point communications links. Today, however the Navy is moving away from specialized solutions and toward Commercial-Off-the-Shelf (COTS) components to both keep up with the rapid pace of technology changes and to reduce system costs. COTS components are designed to meet needs of applications in the general commercial marketplace, not the specialized needs of real-time applications. This paper examines the ability of COTS workstations to distribute and process time sensitive data in a shared media environment. Several metrics are defined and then used to evaluate the performance of a simulated time sensitive data distribution and processing mechanism running on COTS workstations from Hewlett Packard, the Digital Equipment Corporation, and Sun Microsystems. The data collected is analyzed. COTS real-time operating system features are then used to tune the performance of the client and server components of the mechanism.

This paper provides an overview of network architecture, network building block, and network application issues involved in the insertion of Fibre Channel (FC) Commercial Off-The-Shelf (COTS) technology into avionics systems. At the network architecture level, FC supports a simple and low cost Arbitrated Loop (FC-AL) topology which is extensible to fabric switch architectures. A COTS FC testbed is described which has been constructed to characterize FC-AL performance for defined domain-specific avionics scenarios. The testbed is also being used to prototype an avionics multimedia network via integration of FC COTS network controllers and software with streaming video encoders and decoders. Based on lessons learned from the testbed activity and other FC system architecture insertion studies, a set of FC based domain extended products suitable for insertion into avionics systems is identified. Finally future activities required to continue toward a fully capable FC Avionics Multimedia Network (AMN) are described.