Stream and Event Processing Services for Real-time Linked Dataspaces

A Pervasive-computing-enriched smart environment, which contains hundreds of embedded devices coordinated by service management mechanisms, is capable of anticipating intensions of occupants and providing appropriate services accordingly. To acquire high-level contexts, such as human activities, usually involves analyzing and identifying causality and temporal ordering relationships among a bulk stream of sensor readings. However, there are relatively few works investigating this issue. We notice that Complex Event Processing (CEP) is useful for dealing with the issue mentioned above. In this work, we propose a platform for integrating CEP concepts with Per SAM, a service application model for pervasive environments. Applications and experiments are performed to verify the effectiveness of the proposed platform.

With the broader dissemination of digital technologies, visionary concepts like the Internet of Things also affect an increasing number of use cases with interfaces to humans, e.g. manufacturing environments with technical operators monitoring the processes. This leads to additional challenges, as besides the technical issues also human aspects have to be considered for a successful implementation of strategic initiatives like Industrie 4.0. From a technical perspective, complex event processing has proven itself in practice to be capable of integrating and analyzing huge amounts of heterogeneous data and establishing a basic level of situation awareness by detecting situations of interests. Whereas this reactive nature of complex event processing systems may be sufficient for machine-to-machine use cases, the new characteristic of application fields with humans remaining in the control loop leads to an increasing action distance and delayed reactions. Taking human aspects into consideration leads to new requirements, with transparency and comprehensibility of the processing of events being the most important ones. Improving the comprehensibility of complex event processing and extending its capabilities towards an effective support of human operators allows tackling technical and non-technical challenges at the same time. The main contribution of this thesis answers the question of how to evolve state-of-the-art complex event processing from its reactive nature towards a transparent and holistic situation management system. The goal is to improve the interaction among systems and humans in use cases with interfaces between both worlds. Realizing a holistic situation management requires three missing capabilities to be introduced by the contributions of this thesis: First, based on the achieved transparency, the retrospective analysis of situations is enabled by collecting information related to a situation's occurrence and development. Therefore, CEP engine-specific situation descriptions are transformed into a common model, allowing the automatic decomposition of the underlying patterns to derive partial patterns describing the intermediate states of processing. Second, by introducing the psychological model of situation awareness into complex event processing, human aspects of information processing are taken into consideration and introduced into the complex event processing paradigm. Based on this model, an extended situation life-cycle and transition method are derived. The introduced concepts and methods allow the implementation of the controlling function of situation management and enable the effective acquisition and maintenance of situation awareness for human operators to purposefully direct their attention towards upcoming situations. Finally, completing the set of capabilities for situation management, an approach is presented to support the generation and integration of prediction models for predictive situation management. Therefore, methods are introduced to automatically label and extract relevant data for the generation of prediction models and to enable the embedding of the resulting models for an automatic evaluation and execution. The contributions are introduced, applied and evaluated along a scenario from the manufacturing domain.

Complex event processing (CEP) evaluates queries over streams of event data to detect situations of interest. If the event data are produced by geographically distributed sources, CEP may exploit in-network processing that distributes the evaluation of a query among the nodes of a network. To this end, a query is modularized and individual query operators are assigned to nodes, especially those that act as data sources. Existing solutions for such operator placement, however, are limited in that they assume all query results to be gathered at one designated node, commonly referred to as a sink. Hence, existing techniques postulate a hierarchical structure of the network that generates and processes the event data. This largely neglects the optimisation potential that stems from truly decentralised query evaluation with potentially many sinks. To address this gap, in this paper, we propose Multi-Sink Evaluation (MuSE) graphs as a formal computational model to evaluate common CEP queries in a decentralised manner. We further prove the completeness of query evaluation under this model. Striving for distributed CEP that can scale to large volumes of high-frequency event streams, we show how to reason on the network costs induced by distributed query evaluation and prune inefficient query execution plans. As such, our work lays the foundation for distributed CEP that is both, sound and efficient.

Radio Frequency Identication (RFID) has been widely used in most applications. The RFID data generated are usually meaningless, real-time, and large-scale. Complex Event Processing (CEP) technology is used to extract meaningful information from the raw RFID data. This paper proposes a CEP model based on Open Service Gateway Initiative (OSGi) platform with the features of recongurable, customized and so on. Moreover, the open-source Esper technique for processing complex event and event stream is also introduced.

The great advance and affordability of technologies, communications and sensor technology has led to the generation of large amounts of data in the field of the Internet of Things and smart environments, as well as a great demand for smart applications and services adapted to the specific needs of each individual. This has entailed the need for systems capable of receiving, routing and processing large amounts of data to detect situations of interest with low latency, but despite the many existing works in recent years, studying highly scalable and low latency data processing systems is still necessary. In this area, the efficiency of complex event processing (CEP) technology is of particular significance and has been used in a variety of application scenarios. However, in most of these scenarios there is no performance evaluation to show how the system performs under various loads and therefore the developer is challenged to develop such CEP-based systems in new scenarios without knowing how the system will be able to handle different input data rates and address scalability and fault tolerance. This article aims to fill this gap by providing an evaluation of the various versions of one of the most reputable CEP engines—Esper CEP, as well as its integration with two renowned messaging brokers for data ingestion—RabbitMQ and Apache Kafka. For this purpose, we defined a benchmark with a series of event patterns with some of the most representative operators of the Esper CEP engine and we performed a series of tests with an increasing rate of input data to the system. We did this for three alternative software architectures: integrating open-source Esper and RabbitMQ, integrating one instance of Esper enterprise edition with Apache Kafka, and integrating two distributed instances of Esper enterprise edition with Apache Kafka. We measured the usage of CPU, RAM memory, latency and throughput time, looking for the data input rate with which the system overloads for each event pattern and we compared the results of the three proposed architectures. The results have shown a very low CPU consumption for all implementation options and input data rates; a balanced memory usage, quite similar among the three architectures, up to an input rate of 10,000 or 15,000 events per second, depending on the architecture and event pattern, and a quite efficient response time up to 10,000 or 15,000 events per second, depending on the architecture and event pattern. Based on a more exhaustive analysis of results, we have concluded that the different options offered by Esper for CEP provide very efficient solutions for real-time data processing, although each with its limitations in terms of brokers to be used for data integration, scalability, and fault tolerance; a number of suggestions have been drawn out for the developer to take as a basis for choosing which CEP engine and which messaging broker to use for the implementation depending on the of the system in question.

Cloud computing is the notion for delivering access to scalable on-demand computing resources and IT services. The resource management system in IaaS Clouds dynamically allocates the resources based on predefined customers' needs using Service Level Agreements (SLAs) between the Cloud provider and customers. One of the challenges of resource management is to continuously monitor resource utilization, manage, and adjust these resources in real-time fashion to meet the SLAs while not over provisioning resources. Though, there exist numerous Cloud monitoring solutions, they are often highly specialized and restrict the administrator in defining automation rules. Our work aims to develop a framework based on concepts from Complex Event Processing (CEP) and data stream processing where data from various primitive metrics streams are collected and treated as event streams (e.g., from CPU, memory, and disk sensors). By automatically detecting complex patterns and relationships among these primitive events, we can detect and understand more high-level situations from the information provided by sensor steams. In this paper, we describe our CEP-based resource monitoring framework and discuss a use case for implementing auto scaling facilities to the Proxmox platform.

There are two service selection strategies in dynamic service composition, i.e., global strategy and local strategy. The existed global service selection algorithms, due to insufficiently show users’ partiality and feature of the service, is unfavorable to encourage the service provider to optimize the service quality to some extent. In this paper, an ordinary utility function is used as a numerical scale of ordering local services, and then a multi-dimension Qos based local service selection model is proposed to provide important grounds to choose superior service and sift inferior service. Secondly, subjective weight mode, objective weight mode, and subject-objective weight mode are constructed to not only determine the weight coefficient of each Qos criterion, but also show users’ partiality and the objectivity of service quality. At last, this model is proved to be flexibility and effective based on our SEWSCP (Semantic Enable Web Service Composition Platform).

In such areas as transport, healthcare, business process management, financial services, complex event processing (CEP) systems are becoming in demand. Large volumes of streams of events are generated and collected by these systems. Real-time processing of event stream data is essential. Based on current and past events, a decision is made, and forecasting future events is an alternative goal. For the fastest reaction to non-standard behavior, it is necessary to process data streams in real time, it means, as quickly as possible. That's why it is important to find new technologies which can speed up data processing. In this article expensive queries in CEP is identified, effective optimizations to improve their performance significantly is proposed. It is developed and implemented algorithm, which should decrease processing time. The results are compared with state-of-the-art CEP system.

SPARQL was originally developed as a derivative of SQL to process queries over finite-length datasets encoded as RDF graphs. Processing of infinite data streams with SPARQL has been approached by using pre-processors dividing streams into finite-length windows based on either time or the number of incoming triples. Recent extensions to SPARQL can support interconnections of queries, enabling event processing applications to be constructed out of multiple incrementally processed collaborating SPARQL update rules. With more elaborate networks of queries it is possible to perform event processing on heterogeneous event formats without strict restrictions on the number of triples per event. Heterogeneous event support combined with the capability to synthesize new events enables the creation of layered event processing systems. In this paper we review the different types of complex event processing building blocks presented in literature and show their translations to SPARQL update rules through examples, supporting a modular and layered approach. The interconnected examples demonstrate the creation of an elaborate network of SPARQL update rules for solving event processing tasks.

The use of Complex Event Processing (CEP) and Stream Processing (SP) systems to process high-volume, high-velocity Big Data has renewed interest in procedures for managing these systems. In particular, self-management and adaptation of runtime platforms have been common research themes, as most of these systems run under dynamic conditions. Nevertheless, the research landscape in this area is still young and fragmented. Most research is performed in the context of specific systems, and it is difficult to generalize the results obtained to other contexts. To enable generic and reusable CEP/SP system management procedures and self-management policies, this research introduces the Attributed Graph Rewriting for Complex Event Processing Management ( AGeCEP ) formalism. AGeCEP represents queries in a language- and technology-agnostic fashion using attributed graphs. Query reconfiguration capabilities are expressed through standardized attributes, which are defined based on a novel classification of CEP query operators. By leveraging this representation, AGeCEP also proposes graph rewriting rules to define consistent reconfigurations of queries. To demonstrate AGeCEP feasibility, this research has used it to design an autonomic manager and to define a selected set of self-management policies. Finally, experiments demonstrate that AGeCEP can indeed be used to develop algorithms that can be integrated into diverse CEP systems.

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain.

There are complicated correlations among atomic Web services in a composite service. Considering the influence of such correlations on the quality of composite service is important to do service selection. Now, existing selection approaches don’t consider this issue. Using the idea of division, a dot pattern based composite service selection approach is presented in this paper. The main idea of this approach is that according to the correlation degree of atomic services, dividing the composite service into a series of composite service dots; then regarding the dot as selecting unit to do service selection based on the quality of corresponding dot patterns. Experimental results show that our selection approach can improve the quality of the selected composite service.

With the Internet of Things expansion, there has been an increase in the development of services and applications that use context-aware devices incorporated into smart environments. Those devices produce large quantities of data and demand faster and more precise defense mechanisms. Complex Event Processing (CEP) is an emerging and promising technology that allows real-time analysis of continuous data-steams. In this work, we proposed and developed a DDoS (Distributed Denial of Service) detection system and evaluated its performance running on a Raspberry Pi. The results show that the system can be executed on devices with reduced computational power and that CEP is a viable solution to improve the real-time performance on the detection of DDoS attacks on IoT environments.

Complex Event Processing has been a growing field for the last ten years. It has seen the development of a number of methods and tools to aid in the processing of event streams and clouds though it has also been troubled by the lack of a cohesive definition. This paper aims to layout the technologies surrounding CEP and to distinguish it from the closely related field of Event Stream Processing. It also aims to explore the work done to apply Data Mining Techniques to both of these fields. An outline of stream processing technologies is laid out including the Data Stream Mining techniques that have been adapted for CEP.

As RFID is being applied in more and more business applications, events to be processed by RFID middleware is becoming more and more complex. It is needed that some original events are processed to create a complex event in order to get complex information. This paper introduced the design of an RFID middleware based on complex event process (CEP) concept, and presented checking method that is based on limited auto-machine and Hash. Moveover, a detail design of the event processing method of our RFID middleware is presented. Experiment study has also been performed with experimental results given.