Constructing Event Processing Systems of Layered and Heterogeneous Events with SPARQL

Event processing (EP) is a data processing technology that conducts online processing of event information. In this survey, we summarize the latest cutting-edge work done on EP from both industrial and academic research community viewpoints. We divide the entire field of EP into three subareas: EP system architectures, EP use cases, and EP open research topics. Then we deep dive into the details of each subsection. We investigate the system architecture characteristics of novel EP platforms, such as Apache Storm, Apache Spark, and Apache Flink. We found significant advancements made on novel application areas, such as the Internet of Things; streaming machine learning (ML); and processing of complex data types such as text, video data streams, and graphs. Furthermore, there has been significant body of contributions made on event ordering, system scalability, development of EP languages and exploration of use of heterogeneous devices for EP, which we investigate in the latter half of this article. Through our study, we found key areas that require significant attention from the EP community, such as Streaming ML, EP system benchmarking, and graph stream processing.

SUMMARYRecently, event processing (EP) has gained considerable attention as an individual discipline in computer science. From a software engineering perspective, EP systems still lack the maturity of well‐established software architectures. For the development of industrial EP systems, generally accepted software architectures based on proven design patterns and principles are still missing. In this article, we introduce a catalog of design patterns that supports the development of event‐driven architectures (EDAs) and complex EP systems. The design principles originate from experiences reported in publications as well as from our own experiences in building EP systems with industrial and academic partners. We present several patterns on different layers of abstractions that define the overall structure as well as the building blocks for EP systems. Architectural patterns that determine the top‐level structure of an EDA can be distinguished from design patterns that specify the basic mechanisms of EP. The practical application of the catalog of patterns is described by the pattern‐based design of a sample EDA for a sensor‐based energy control system. Finally, we propose a coherent and general reference architecture for EP derived from the proposed patterns.Copyright © 2013 John Wiley & Sons, Ltd.

In such areas as transport, healthcare, business process management, financial services, complex event processing (CEP) systems are becoming in demand. Large volumes of streams of events are generated and collected by these systems. Real-time processing of event stream data is essential. Based on current and past events, a decision is made, and forecasting future events is an alternative goal. For the fastest reaction to non-standard behavior, it is necessary to process data streams in real time, it means, as quickly as possible. That's why it is important to find new technologies which can speed up data processing. In this article expensive queries in CEP is identified, effective optimizations to improve their performance significantly is proposed. It is developed and implemented algorithm, which should decrease processing time. The results are compared with state-of-the-art CEP system.

The use of Complex Event Processing (CEP) and Stream Processing (SP) systems to process high-volume, high-velocity Big Data has renewed interest in procedures for managing these systems. In particular, self-management and adaptation of runtime platforms have been common research themes, as most of these systems run under dynamic conditions. Nevertheless, the research landscape in this area is still young and fragmented. Most research is performed in the context of specific systems, and it is difficult to generalize the results obtained to other contexts. To enable generic and reusable CEP/SP system management procedures and self-management policies, this research introduces the Attributed Graph Rewriting for Complex Event Processing Management ( AGeCEP ) formalism. AGeCEP represents queries in a language- and technology-agnostic fashion using attributed graphs. Query reconfiguration capabilities are expressed through standardized attributes, which are defined based on a novel classification of CEP query operators. By leveraging this representation, AGeCEP also proposes graph rewriting rules to define consistent reconfigurations of queries. To demonstrate AGeCEP feasibility, this research has used it to design an autonomic manager and to define a selected set of self-management policies. Finally, experiments demonstrate that AGeCEP can indeed be used to develop algorithms that can be integrated into diverse CEP systems.

\n In distributed complex event processing, event streams are processed\n over a chain of subsequent operators. For large-scale applications\n like a logistic chain these operators may be hosted by different\n entities and thus are spread over different security domains.\n Current approaches for complex event processing cannot preserve the\n privacy of an operators incoming event streams. An adversary may\n infer the original input stream from its legally received event\n streams.\n\n In this paper we present a fine-grained access management for\n complex event processing. We show how to enforce privacy of events\n throughout the chain of dependent operators by specifying\n appropriate access policies and proposing an algorithm for policy\n consolidation. Furthermore, we introduce the calculation of\n obfuscation achieved in a correlation step. This allows us to ignore\n access requirements once a sufficient obfuscation level has been\n achieved, the proposed algorithms is capable to reduce the required\n overhead in the enforcement of access policies. We prove correctness\n and evaluate the cost in establishing policy consolidation.\n

It is always difficult to manipulate the production of huge amount of data which comes from multiple sources and to extract meaningful information to make appropriate decisions. When data comes from various input resources, to get required streams of events form this complex input network, the one of the strong functionality of Business Intelligence (BI) the Complex Event Processing (CEP) is the appropriate solution for the above mention problems. Real time processing, pattern matching, stream processing, big data management, sensor data processing and many more are the application areas of CEP. Health domain itself is a multi-dimension domain such as hospital supply chain, OPD management, disease diagnostic, In-patient, out-patient management, and emergency care etc. In this paper, the main focus is to discuss the application areas of Complex Event Processing (CEP) in health domain by using sensor device, such that how CEP manipulate health data set events coming from sensor devices such as blood pressure, heart rate, fall detection, sugar level, temperature or any other vital signs and how this systems respond to these events as quickly as possible. Different existing models and application using CEP are discussed and summarized according to different characteristics.

This study presents a novel method to develop kleene operators on a Binary Tree-Based Directed Graph (BTDG) to improve expressibility and performance of our developed complex event processing system. Complex Event Processing (CEP) systems are widely employed to notify opportunities and threats, which many of these situations are discovered through detection of multiple occurrences of the similar set of events. Many of CEP systems proposed kleene operators to detect these situations. The high expressibility of our implemented CEP system achieved by developing algorithms on kleene operator. These algorithms are designed on a BTDG as a unified basis to arrange input data and joining this input according to a given query. The performance is improved through directing events instantly to join with other events, dynamically switching to an optimal plan and late applying combination for kleene with specified size. The experiments shows kleene operator on BTDG brings rather high degree of expressiblity and performance in compare to its counterparts.

In recent years there is a huge increase in real-time data, which cannot be processed efficiently. Complex event processing has become a very important method to get meaningful information. However, supporting complex event detection in multiple sources environments is a challenging problem. To allow for inferring high level information from vast amounts of continuous arriving data. In this paper, we present a complex event processing system based on a novel distributed computing platform Storm, which goes further than distributing queries and achieves better scalability by parallelizing event detection, and also higher efficiency through the use of some optimizations. The experimental shows that the event processing system is effective and better scalability.

With the broader dissemination of digital technologies, visionary concepts like the Internet of Things also affect an increasing number of use cases with interfaces to humans, e.g. manufacturing environments with technical operators monitoring the processes. This leads to additional challenges, as besides the technical issues also human aspects have to be considered for a successful implementation of strategic initiatives like Industrie 4.0. From a technical perspective, complex event processing has proven itself in practice to be capable of integrating and analyzing huge amounts of heterogeneous data and establishing a basic level of situation awareness by detecting situations of interests. Whereas this reactive nature of complex event processing systems may be sufficient for machine-to-machine use cases, the new characteristic of application fields with humans remaining in the control loop leads to an increasing action distance and delayed reactions. Taking human aspects into consideration leads to new requirements, with transparency and comprehensibility of the processing of events being the most important ones. Improving the comprehensibility of complex event processing and extending its capabilities towards an effective support of human operators allows tackling technical and non-technical challenges at the same time. The main contribution of this thesis answers the question of how to evolve state-of-the-art complex event processing from its reactive nature towards a transparent and holistic situation management system. The goal is to improve the interaction among systems and humans in use cases with interfaces between both worlds. Realizing a holistic situation management requires three missing capabilities to be introduced by the contributions of this thesis: First, based on the achieved transparency, the retrospective analysis of situations is enabled by collecting information related to a situation's occurrence and development. Therefore, CEP engine-specific situation descriptions are transformed into a common model, allowing the automatic decomposition of the underlying patterns to derive partial patterns describing the intermediate states of processing. Second, by introducing the psychological model of situation awareness into complex event processing, human aspects of information processing are taken into consideration and introduced into the complex event processing paradigm. Based on this model, an extended situation life-cycle and transition method are derived. The introduced concepts and methods allow the implementation of the controlling function of situation management and enable the effective acquisition and maintenance of situation awareness for human operators to purposefully direct their attention towards upcoming situations. Finally, completing the set of capabilities for situation management, an approach is presented to support the generation and integration of prediction models for predictive situation management. Therefore, methods are introduced to automatically label and extract relevant data for the generation of prediction models and to enable the embedding of the resulting models for an automatic evaluation and execution. The contributions are introduced, applied and evaluated along a scenario from the manufacturing domain.

Complex Event Processing (CEP) systems aim at processing large flows of events to discover situations of interest. In CEP, the processing takes place according to user-defined rules, which specify the (causal) relations between the observed events and the phenomena to be detected. We claim that the complexity of writing such rules is a limiting factor for the diffusion of CEP. In this paper, we tackle this problem by introducing iCEP, a novel framework that learns, from historical traces, the hidden causality between the received events and the situations to detect, and uses them to automatically generate CEP rules. The paper introduces three main contributions. It provides a precise definition for the problem of automated CEP rules generation. It dicusses a general approach to this research challenge that builds on three fundamental pillars: decomposition into subproblems, modularity of solutions, and ad-hoc learning algorithms. It provides a concrete implementation of this approach, the iCEP framework, and evaluates its precision in a broad range of situations, using both synthetic benchmarks and real traces from a traffic monitoring scenario.

Information filtering is a crucial task in a world where data is generated steadily and at a high rate, helping users in distinguishing relevant from irrelevant content. This requires efficient processing of continuous streams of textual data. Event processing allows for real time processing of data streams. But up to now event processing has mainly been investigated in the context of business transaction-oriented domains like logistics or finance, but not explicitly in terms of text stream processing and information filtering. The growth of applications that analyze social media streams lets such an approach appear reasonable. Therefore we propose a common vocabulary represented by a text domain event model as well as a reference architecture for text stream processing and information filtering, in order to facilitate the implementation and the assessment of event processing applications for text streams. In addition we describe results from actual use cases that employ this architecture.

Event processing systems link event producers and consumers in a flexible manner, by supporting multiple communication patterns and powerful message transformations. Such systems support both loosely coupled communication patterns, such as one-to-one or one-to-many and tightly coupled (request-response) patterns. An important concern in these systems is the allocation of resources used to support event processing. These include traditional constrained resources such as CPU and network, as well as an increasing reliance on storage resources. For instance, transactional and reliable producers require events to be logged to stable storage. In this paper we consider the problem of utility driven allocation of these resources, taking into account resource capacity. In contrast to previous work, we (1) develop a unified utility model to deal with throughput requirements of one-way flows and latency requirements of request-response flows, (2) differentiate between classes of consumers for one-to-many flows and subject these classes to consumer admission control, (3) compute system optimization constraints using both resource bandwidth and latency and (4) develop a control architecture that combines periodic optimization with continuous rate control, both of which are utility aware. We describe an experimental evaluation of our approach on a prototype event processing server.

Complex Event Processing (CEP) systems extract interest situations from event flows based on event detection patterns. However, local event processing for distributed Internet of Things (IoT) has not been discussed yet. Besides, it is complex or impossible to discover such patterns in some applications of IoT. In this article, we design a complex event service to process event flows based on IoT resource models, which does not depend on existing patterns, and deals with both discrete events and continuous variables. To improve the CEP performance, local IoT resources are used for local event processing, and a lazy exchange method is designed to realize the collaborated event processing between network edges and a data center. Our evaluation shows that our solution is feasible and effective.

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain.

Complex event processing has become an important technology for big data and intelligent computing because it facilitates the creation of actionable, situational knowledge from potentially large amount events in soft realtime. Complex event processing can be instrumental for many mission-critical applications, such as business intelligence, algorithmic stock trading, and intrusion detection. Hence, the servers that carry out complex event processing must be made trustworthy. In this paper, we present a threat analysis on complex event processing systems and describe a set of mechanisms that can be used to control various threats. By exploiting the application semantics for typical event processing operations, we are able to design lightweight mechanisms that incur minimum runtime overhead appropriate for soft realtime computing.