Analysis and optimization techniques for real-time streaming image processing software on general purpose systems

Recent advances in mobile technologies have enabled a plethora of new applications. The hardware capabilities of mobile devices, however, are still insufficient for real-time stream data processing (e.g., real-time video stream). In order to process real-time streaming data, most existing applications offload the data and computation to a remote cloud service, such as Apache Storm or Apache Spark Streaming. Offloading streaming data, however, has high costs for users, e.g., significant service fees and battery consumption. To address these challenges, we design, implement and evaluate Mobile Storm, the first stream processing platform for mobile clouds, leveraging clusters of local mobile devices to process real-time stream data. In Mobile Storm, we model the workflow of a real-time stream processing job and decompose it into several tasks so that the job can be executed concurrently and in a distributed manner on multiple mobile devices. Mobile Storm was implemented on Android phones and evaluated extensively through a real-time HD video processing application. The result shows that Mobile Storm effectively processes HD Video Stream in a mobile cloud, which would be impossible on a single mobile device.

The ability to interpret spatiotemporal data streams in real-time is critical for a range of systems. However, processing vast amounts of spatiotemporal data out of several sources, such as online traffic, social platforms, sensor networks, and other sources, is a considerable challenge. The major goal of this study is to create a framework for processing and analyzing spatiotemporal data from multiple sources with irregular shapes so that researchers can focus on data analysis instead of worrying about the data sources' structure. We introduced a novel spatiotemporal data paradigm for true-real-time stream processing, which enables high-speed and low-latency real-time data processing, with these considerations in mind. A comparison of two state-of-the-art real-time process architectures was offered, as well as a full review of the various open-source technologies for real-time data stream processing, and their system topologies were also presented. Hence, this study proposed a brand-new framework that integrates Apache Kafka for spatiotemporal data ingestion, Apache flink for true real-time processing of spatiotemporal stream data, as well as machine learning for real-time predictions, and Apache Cassandra at the storage layer for distributed storage in real-time. The proposed framework was compared with others from the literature using the following features: Scalability (Sc), prediction tools (PT), data analytics (DA), multiple event types (MET), data storage (DS), Real-time (Rt), and performance evaluation (PE) stream processing (SP), and our proposed framework provided the ability to handle all of this task.

In this paper we introduce a new real-time parallel image processing system based on an IEEE1394-based PC-cluster. PC-clusters are becoming standard tools in the field of computer vision, especially for real-time multi-view image processing. To reduce the cost of system construction, we have employed IEEE1394 bus as a base network of the PC-cluster. IEEE1394 bus has the throughput of up to 400Mbps, which is large enough to transfer uncompressed video data, and supports not only asynchronous data transfer but isochronous one, which is inevitable for real-time data transfer. In addition, since we have several products of IEEE1394-based digital cameras, we can easily integrate real-time image processing system including camera systems. This paper presents an overview of our real-time parallel image processing system on an IEEE1394based PC-cluster, referring to basic features of IEEE1394. We also show some experimental results to evaluate the performance of the PC-cluster, comparing it with the performance of our previous Myrinet-based PC-cluster. Then we show a prototypical application of the system, which is real-time volume reconstruction from multi-view images by visual cone intersection method.

The first quarter of 2007 started as an active season for the real-time image processing community. The annual SPIE Conference on Real-Time Image Processing, which was held in San Jose, CA, highlighted recent developments in this growing branch of image processing. As one of the few regular events with its focus matching that of JRTIP, one witnessed the growth in applications of real-time image and video processing. The conference served as an effective instrument to foster the dissemination of recent results and to leverage the cohesion of the real time image processing community. The preparations for the next SPIE Conference on Real-Time Image Processing in 2008 are already progressing and its call-for-papers is included at the end of this issue. In 2007, JRTIP is going to seek involvement of more scientists in the field of real-time image and video processing. After 1 year of operation, the editors-in-chief have started to recruit additional associate editors to enlarge the expertise base of the editorial board, to solicit manuscript submissions from potential authors, and to make the reviewing process more efficient. Interested colleagues should contact the editors-in-chief expressing their Associate Editor candidature in support of one or more fields within the focus of JRTIP. In addition, they wish to encourage researchers to act as guest editors for special issues on any real-time image or video processing related topics of interest to them. It is possible to schedule two special issues in 2007, provided that a draft call-for-papers is sent to the editors-in-chief as soon as possible. The third issue of JRTIP presents four articles, two are survey papers and two are original research papers. These articles address different real-time image processing perspectives ranging from algorithmic to hardware issues. The first survey paper deals with the real-time filtering in 2D and 3D using ordered statistics that are supported by a rich set of experimental results derived from several algorithmic variations. Performance measurements are used to demonstrate the effect of noise suppression and preservation of fine image structures and to show the DSP implementation of denoising filters. The second survey paper introduces a non-invasive analysis of dynamic processes in living cells using microscopic computer vision techniques for understanding the interactions between cellular and molecular functional units and for providing novel insights into the cell biology. The third article is an original research paper covering a machine vision application involving vision-based position sensing for aerial refueling of unmanned aerial vehicles in real-time. A full image processing chain is investigated and tested in a MATLAB/Simulink soft real-time environment as well as in a Linux/RTAI hard real-time environment. The feasibility of the real-time implementation is shown using off-the-shelf commercial hardware. The fourth article is another original research paper describing a hardware architecture for real-time fractal coding for image compression. The introduced parallel hardware architecture contains units for domain M. F. Carlsohn (&) Ingenieurberatung fur Computer Vision und Bildkommunikation, Am Heiddamm 36g, 28355 Bremen, Germany e-mail: matthias.carlsohn@t-online.de

Safety-critical systems and in particular mixed-criticality systems require spatial and temporal separation for their hosted applications and functionalities. Additional constraints are using Commercial Off -- The -- Shelf (COTS) components, portability and determinism. These items are required for economic success for products with low piece numbers and long life-cycles like aircraft. Available embedded processors lack means for spatial separation of Input/Output (I/O) components like an Input/Output Memory Management Unit (IOMMU). The objective of this paper is to provide spatial separation for I/O in COTS mixed-criticality embedded real-time systems like avionics with minimum possible impact on performance (transfer time, transfer rate, Central Processing Unit (CPU) usage). The three main contributions of this paper are: (1) The presented Input/Output Memory Protection Unit (IOMPU) enables to upgrade spatial separation for I/O to a system by using COTS components and Non-Transparent Bridge (NTB) technology. In addition, the IOMPU concept is compatible with existing temporal separation solutions. (2) The paper shows a prototype implementation and a potential use case in context of hardware-based I/O virtualization. (3) The evaluation in this paper demonstrates that the IOMPU concept is practically applicable. The performance overhead (transfer time, transfer rate) is below 0.88%, which is almost negligible, particularly compared to state-of-the-art software-based solutions.

Processing a data stream in real time is a crucial issue for several applications, however processing a large amount of data from different sources, such as sensor networks, web traffic, social media, video streams and other sources, represents a huge challenge. The main problem is that the big data system is based on Hadoop technology, especially MapReduce for processing. This latter is a high scalability and fault tolerant framework. It also processes a large amount of data in batches and provides perception blast insight of older data, but it can only process a limited set of data. MapReduce is not appropriate for real time stream processing, and is very important to process data the moment they arrive at a fast response and a good decision making. Ergo the need for a new architecture that allows real-time data processing with high speed along with low latency. The major aim of the paper at hand is to give a clear survey of the different open sources technologies that exist for real-time data stream processing including their system architectures. We shall also provide a brand new architecture which is mainly based on previous comparisons of real-time processing powered with machine learning and storm technology.

Using commercial off the shelf (COTS) computing equipment in military aircraft applications may cause harmful electromagnetic interference (EMI) to other on-board systems, including communication and navigation equipment. An electromagnetic compatibility (EMC) design approach to successfully integrate COTS equipment into an existing military aircraft is described. The following system design and development steps are outlined: (1) requirements definition, (2) preliminary EMC analysis, (3) EMC control measure design, (4) COTS equipment EMI characterization tests, and (5) system level EMC verification.

With the increasing availability and scale of data from Web 2.0, the ability to efficiently and timely analyze huge amounts of data is important for industry success. A number of real-time stream processing platforms have been developed, such as Storm, S4, and Flume. A fundamental problem of these large scale decentralized stream processing systems is how to deploy the workload to each node so as to fully utilize the available resources and optimize the overall system performance. In this paper, we present StroMAX, a graph-partitioning based approach of workload scheduling for real-time stream processing systems. StroMAX uses two advanced generic schedulers to improve the performance of stream processing systems by reducing the inter-node communication cost while keeping the workload of nodes below a certain computational load threshold. The first scheduler analyzes the workload structure when a job is committed and uses the graph-partitioning result to determine the deployment of tasks. The second scheduler analyzes the statistical information of physical nodes, and dynamically reassigns the tasks during runtime to improve the overall performance. Besides, StroMAXcan be deployed to many other state-of-the-art real-time stream processing systems easily. We implemented StroMAX on Storm, a representative real-time stream processing system. Extensive experiments conducted with real-world workloads and datasets demonstrate the superiority of our approaches against the existing solutions.

The increase of complexity and miniaturizing level of Aerospace platforms make use of commercial off the shelf (COTS) components constitute a plausible alternative to the use of military or rad-tolerant components. In this work, giant magnetoresistance commercial sensors are studied to be used as COTS, the next missions to be launched in the framework of the Spanish National Space Program: OPTOS and SEOSAT. This technology of magnetic sensors is interesting due to their high operating range up to 2mT and the high temperature dynamic range from −50 up to 150°C. However, in contrast, it presents high hysteresis and nonlinearity, temperature dependence, and poor repeatability. To improve the hysteretic, nonlinear and nonrepetitive behavior, a method consisting of a combination of reset and biasing has been designed and implemented for the ±75μT linear region centered around 300–375μT biasing field.

: Small unmanned systems provide great utility to military applications due to their portable and expendable design. These systems are, however, costly to develop, produce, and maintain, making it desirable to integrate available commercial off the shelf (COTS) components. This research investigates the integration of COTS components through the development of a modular unified command and control (C2) architecture for heterogeneous and homogeneous vehicle teams to accomplish formation flocking and communication relay scenarios. A vehicle agnostic architecture was developed to be applied across different vehicle platforms, different vehicle combination, and different cooperative missions. COTS components consisting primarily of open source hardware and software were integrated and tested based on the positional accuracy, precision, and other qualitative measures. The resulting system successfully demonstrated formation flocking in three of four vehicle combinations, with the forth still demonstrating leader follower behaviors. The system achieved at best a mean positional error of 0.99m, a standard deviation of 0.44m, and a DRMS of 0.59m....

In the past, functional test requirements (FTR) or test requirement documents (TRD) and test program sets (TPS) were standalone items developed by individual engineers. In some cases one engineer would write the FTR/TRD and another would develop the TPS. Commercial ATLAS FTRs are prepared in ARINC 616 and 626 ATLAS. Military TRDs are written in IEEE ATLAS 716 versions. Previous test reuse attempts have not been successful because additional software, like browsers, is required to support these efforts. It was difficult to justify writing new software (for example browsers) to manage the application software. Today, commercial off the shelf (COTS) tools are in place to browse and view information from circuit diagrams to documents to source code. These tools can develop hierarchies to organize the information. These COTS tools are available throughout Boeing on many types of workstations and personal computers on every engineer's desk. This paper discusses how a reusable test library (RTL) is being developed using commercial off-the-shelf (COTS) tools such as Mosaic to address commercial and military test applications. It describes each of the tools and the process to develop TPSs using the reuse library. It defines the metrics and the benefits achieved.

Time is the best important resource in real-time video image processing system. Only few structure or algorithm can be used in real-time video image processing system because of time limitation. Establish a high speed data-access is a way to this problem. It transfer the video signal between output ports and kernel processor as less time delay. The mode, look-up-table, other analysis, adjustment or calculation rule will be applied first by medium or low data-access. We can choose different data-access according to data relation level. Data which has closer correlation will be treated through low-speed-access, which has loser correlation will be treated through medium-speed-access. Theory analysis and experiment showed that it is a high effective structure for real-time video image processing. The feature of video image and real-time processing system is presented in this paper. The basic theory of data-access is introduced also. The status and relation of high, medium, low data-access is discussed and the rules of data distribution in different access is analyzed. Some examples with datas-accesses are given in the paper (TV tracking and measuring, laser signal alarming system).

In this paper we explore the possibility of using external secure elements, namely smart cards, to enhance the off-line security of commercial off- the-shelf (COTS) Android smartphones. The idea of using smart cards with smartphones is not completely new, but they are usually employed in solutions that need to be approved for RESTRICTED classification and above, and leverage customized devices in dedicated infrastructures. This approach defeats in part the purpose of using COTS products, and may be unnecessary for situations where budget is limited and data is mission sensitive, but not necessary classified, as in the use-case motivating this work. Here we present our findings on how different types of smart cards as Mifare NFC cards and Java Cards in both microSD and Dual format may interact with standard Android devices to enhance their security when not on-line. We conclude that smart cards can be a cost-effective way to mitigate some weaknesses of smartphones which can constitute a risk in tactical military scenarios, but compatibility challenges and device ownership models can limit their effective security.

Big data challenged traditional storage and analysis systems in several new ways. In this paper we try to figure out how to overcome this challenges, why it's not possible to make it efficiently and describe three modern approaches to big data handling: NoSQL, MapReduce and real-time stream processing. The first section of the paper is the introduction. The second section discuss main issues of Big Data: volume, diversity, velocity, and value. The third section describes different approaches to solving the problem of Big Data. Traditionally one might use a relational DBMS. The paper propose some steps that allow to continue RDBMS using when it’s capacity becomes not enough. Another way is to use a NoSQL approach. The basic ideas of the NoSQL approach are: simplification, high throughput, and unlimited scaling out. Different kinds of NoSQL stores allow to use such systems in different applications of Big Data. MapReduce and it’s free implementation Hadoop may be used to provide scaling out Big Data analytics. Finally, several data management products support real time stream processing under Big Data. The paper briefly overviews these products. The final section of the paper is the conclusion.

Real-Time Image Processing is the key work in robot vision. With the limitation of the hardware technique, many algorithm-oriented firmware systems were designed in the past. But their architectures were not flexible enough to achieve a multi- algorithm development system. Because of the rapid development of microelectronics technique, many high performance DSP chips and high density FPGA chips have come to life, and this makes it possible to construct a more flexible architecture in real-time image processing system. In this paper, a Double DSP General Image Processing System (DDGIPS) is concerned. We try to construct a two-DSP-based FPGA-computational system with two TMS320C6201s. The TMS320C6x devices are fixed-point processors based on the advanced VLIW CPU, which has eight functional units, including two multipliers and six arithmetic logic units. These features make C6x a good candidate for a general purpose system. In our system, the two TMS320C6210s each has a local memory space, and they also have a shared system memory space which enable them to intercommunicate and exchange data efficiently. At the same time, they can be directly interconnected in star- shaped architecture. All of these are under the control of FPGA group. As the core of the system, FPGA plays a very important role: it takes charge of DPS control, DSP communication, memory space access arbitration and the communication between the system and the host machine. And taking advantage of reconfiguring FPGA, all of the interconnection between the two DSP or between DSP and FPGA can be changed. In this way, users can easily rebuild the real-time image processing system according to the data stream and the task of the application and gain great flexibility.