Recent Advancements in Event Processing

With the broader dissemination of digital technologies, visionary concepts like the Internet of Things also affect an increasing number of use cases with interfaces to humans, e.g. manufacturing environments with technical operators monitoring the processes. This leads to additional challenges, as besides the technical issues also human aspects have to be considered for a successful implementation of strategic initiatives like Industrie 4.0. From a technical perspective, complex event processing has proven itself in practice to be capable of integrating and analyzing huge amounts of heterogeneous data and establishing a basic level of situation awareness by detecting situations of interests. Whereas this reactive nature of complex event processing systems may be sufficient for machine-to-machine use cases, the new characteristic of application fields with humans remaining in the control loop leads to an increasing action distance and delayed reactions. Taking human aspects into consideration leads to new requirements, with transparency and comprehensibility of the processing of events being the most important ones. Improving the comprehensibility of complex event processing and extending its capabilities towards an effective support of human operators allows tackling technical and non-technical challenges at the same time. The main contribution of this thesis answers the question of how to evolve state-of-the-art complex event processing from its reactive nature towards a transparent and holistic situation management system. The goal is to improve the interaction among systems and humans in use cases with interfaces between both worlds. Realizing a holistic situation management requires three missing capabilities to be introduced by the contributions of this thesis: First, based on the achieved transparency, the retrospective analysis of situations is enabled by collecting information related to a situation's occurrence and development. Therefore, CEP engine-specific situation descriptions are transformed into a common model, allowing the automatic decomposition of the underlying patterns to derive partial patterns describing the intermediate states of processing. Second, by introducing the psychological model of situation awareness into complex event processing, human aspects of information processing are taken into consideration and introduced into the complex event processing paradigm. Based on this model, an extended situation life-cycle and transition method are derived. The introduced concepts and methods allow the implementation of the controlling function of situation management and enable the effective acquisition and maintenance of situation awareness for human operators to purposefully direct their attention towards upcoming situations. Finally, completing the set of capabilities for situation management, an approach is presented to support the generation and integration of prediction models for predictive situation management. Therefore, methods are introduced to automatically label and extract relevant data for the generation of prediction models and to enable the embedding of the resulting models for an automatic evaluation and execution. The contributions are introduced, applied and evaluated along a scenario from the manufacturing domain.

SUMMARYRecently, event processing (EP) has gained considerable attention as an individual discipline in computer science. From a software engineering perspective, EP systems still lack the maturity of well‐established software architectures. For the development of industrial EP systems, generally accepted software architectures based on proven design patterns and principles are still missing. In this article, we introduce a catalog of design patterns that supports the development of event‐driven architectures (EDAs) and complex EP systems. The design principles originate from experiences reported in publications as well as from our own experiences in building EP systems with industrial and academic partners. We present several patterns on different layers of abstractions that define the overall structure as well as the building blocks for EP systems. Architectural patterns that determine the top‐level structure of an EDA can be distinguished from design patterns that specify the basic mechanisms of EP. The practical application of the catalog of patterns is described by the pattern‐based design of a sample EDA for a sensor‐based energy control system. Finally, we propose a coherent and general reference architecture for EP derived from the proposed patterns.Copyright © 2013 John Wiley & Sons, Ltd.

Complex Event Processing (CEP) systems extract interest situations from event flows based on event detection patterns. However, local event processing for distributed Internet of Things (IoT) has not been discussed yet. Besides, it is complex or impossible to discover such patterns in some applications of IoT. In this article, we design a complex event service to process event flows based on IoT resource models, which does not depend on existing patterns, and deals with both discrete events and continuous variables. To improve the CEP performance, local IoT resources are used for local event processing, and a lazy exchange method is designed to realize the collaborated event processing between network edges and a data center. Our evaluation shows that our solution is feasible and effective.

Fog computing is envisioned to enable profound applications in the Internet of Things (IoT). A key characteristic of such applications is the need to exchange vital information between distinct IoT devices in the form of event notifications, e. g., traffic conditions when performing traffic monitoring. Complex event processing (CEP) is a powerful paradigm to overcome the information gap from observing primary sensor data by IoT devices to delivering event notifications to the IoT application users. However, to perform CEP in a highly dynamic IoT environment, e. g., involving mobile and heterogeneous devices, require an extremely flexible design of a CEP system to adaptively meet the changing requirements and conditions in which the CEP system is executed. In this article, we show on the use case of CEP, “how to increase flexibility in a fog-cloud computing environment building on a methodology known as mechanism transitions”. In particular, we state and analyze two exemplary IoT use cases to show the potential of mechanism transitions. We identify and discuss possible promising mechanism transitions in the context of CEP. We perform an experimental study for operator placement and show how transitions help to adapt to conflicting performance objectives.

SPARQL was originally developed as a derivative of SQL to process queries over finite-length datasets encoded as RDF graphs. Processing of infinite data streams with SPARQL has been approached by using pre-processors dividing streams into finite-length windows based on either time or the number of incoming triples. Recent extensions to SPARQL can support interconnections of queries, enabling event processing applications to be constructed out of multiple incrementally processed collaborating SPARQL update rules. With more elaborate networks of queries it is possible to perform event processing on heterogeneous event formats without strict restrictions on the number of triples per event. Heterogeneous event support combined with the capability to synthesize new events enables the creation of layered event processing systems. In this paper we review the different types of complex event processing building blocks presented in literature and show their translations to SPARQL update rules through examples, supporting a modular and layered approach. The interconnected examples demonstrate the creation of an elaborate network of SPARQL update rules for solving event processing tasks.

Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams

With the increasing adoption of an event-based perspective in many organizations, the demands for automatic processing of events are becoming more sophisticated. Although complex event processing systems can process events in near real-time, these systems rely heavily upon human domain experts. This becomes an issue in application areas that are rich in specialized domain knowledge and background information, such as clinical environments. We utilize a framework of four techniques to enhance complex event processing with domain knowledge from ontologies to address this issue. We realize this framework in our novel approach of ontology-supported complex event processing, which stands in contrast to related approaches and emphasizes the strengths of current advances in the individual fields of complex event processing and ontologies. Experimental results from the implementation of our approach based on a state-of-the-art system show its feasibility and indicate the direction for future research.

\n In distributed complex event processing, event streams are processed\n over a chain of subsequent operators. For large-scale applications\n like a logistic chain these operators may be hosted by different\n entities and thus are spread over different security domains.\n Current approaches for complex event processing cannot preserve the\n privacy of an operators incoming event streams. An adversary may\n infer the original input stream from its legally received event\n streams.\n\n In this paper we present a fine-grained access management for\n complex event processing. We show how to enforce privacy of events\n throughout the chain of dependent operators by specifying\n appropriate access policies and proposing an algorithm for policy\n consolidation. Furthermore, we introduce the calculation of\n obfuscation achieved in a correlation step. This allows us to ignore\n access requirements once a sufficient obfuscation level has been\n achieved, the proposed algorithms is capable to reduce the required\n overhead in the enforcement of access policies. We prove correctness\n and evaluate the cost in establishing policy consolidation.\n

Complex Event Processing (CEP) systems aim at processing large flows of events to discover situations of interest. In CEP, the processing takes place according to user-defined rules, which specify the (causal) relations between the observed events and the phenomena to be detected. We claim that the complexity of writing such rules is a limiting factor for the diffusion of CEP. In this paper, we tackle this problem by introducing iCEP, a novel framework that learns, from historical traces, the hidden causality between the received events and the situations to detect, and uses them to automatically generate CEP rules. The paper introduces three main contributions. It provides a precise definition for the problem of automated CEP rules generation. It dicusses a general approach to this research challenge that builds on three fundamental pillars: decomposition into subproblems, modularity of solutions, and ad-hoc learning algorithms. It provides a concrete implementation of this approach, the iCEP framework, and evaluates its precision in a broad range of situations, using both synthetic benchmarks and real traces from a traffic monitoring scenario.

In such areas as transport, healthcare, business process management, financial services, complex event processing (CEP) systems are becoming in demand. Large volumes of streams of events are generated and collected by these systems. Real-time processing of event stream data is essential. Based on current and past events, a decision is made, and forecasting future events is an alternative goal. For the fastest reaction to non-standard behavior, it is necessary to process data streams in real time, it means, as quickly as possible. That's why it is important to find new technologies which can speed up data processing. In this article expensive queries in CEP is identified, effective optimizations to improve their performance significantly is proposed. It is developed and implemented algorithm, which should decrease processing time. The results are compared with state-of-the-art CEP system.

This study presents a novel method to develop kleene operators on a Binary Tree-Based Directed Graph (BTDG) to improve expressibility and performance of our developed complex event processing system. Complex Event Processing (CEP) systems are widely employed to notify opportunities and threats, which many of these situations are discovered through detection of multiple occurrences of the similar set of events. Many of CEP systems proposed kleene operators to detect these situations. The high expressibility of our implemented CEP system achieved by developing algorithms on kleene operator. These algorithms are designed on a BTDG as a unified basis to arrange input data and joining this input according to a given query. The performance is improved through directing events instantly to join with other events, dynamically switching to an optimal plan and late applying combination for kleene with specified size. The experiments shows kleene operator on BTDG brings rather high degree of expressiblity and performance in compare to its counterparts.

In recent years there is a huge increase in real-time data, which cannot be processed efficiently. Complex event processing has become a very important method to get meaningful information. However, supporting complex event detection in multiple sources environments is a challenging problem. To allow for inferring high level information from vast amounts of continuous arriving data. In this paper, we present a complex event processing system based on a novel distributed computing platform Storm, which goes further than distributing queries and achieves better scalability by parallelizing event detection, and also higher efficiency through the use of some optimizations. The experimental shows that the event processing system is effective and better scalability.

The Internet of Things (IoT) has grown rapidly to become the core of many areas of application, leading to the integration of sensors, with IoT devices. However, the number of attacks against these types of devices has grown as fast as the paradigm itself. Certain inherent characteristics of the paradigm, as well as the limited computational capabilities of the devices involved, make it difficult to deploy security measures. This is why it is necessary to design, implement and study new solutions in the field of cybersecurity. In this paper, we propose an architecture that is capable of generating Complex Event Processing (CEP) rules automatically by integrating them with machine learning technologies. While the former is used to automatically detect attack patterns in real time, the latter, through the use of the Principal Component Analysis (PCA) algorithm, allows the characterization of events and the recognition of anomalies. This combination makes it possible to achieve efficient CEP rules at the computational level, with the results showing that the CEP rules obtained using our approach substantially improve upon the performance of the standard CEP rules, which are rules that are not generated by our proposal but can be defined independently by an expert in the field. Our proposal has achieved an F1-score of 0.98 on average, a 76 percent improvement in throughput over standard CEP rules, and a reduction in the network overhead of 86 percent over standard simple events, which are the simple events that are generated when our proposal is not used.

Complex event processing has become an important technology for big data and intelligent computing because it facilitates the creation of actionable, situational knowledge from potentially large amount events in soft realtime. Complex event processing can be instrumental for many mission-critical applications, such as business intelligence, algorithmic stock trading, and intrusion detection. Hence, the servers that carry out complex event processing must be made trustworthy. In this paper, we present a threat analysis on complex event processing systems and describe a set of mechanisms that can be used to control various threats. By exploiting the application semantics for typical event processing operations, we are able to design lightweight mechanisms that incur minimum runtime overhead appropriate for soft realtime computing.

It is always difficult to manipulate the production of huge amount of data which comes from multiple sources and to extract meaningful information to make appropriate decisions. When data comes from various input resources, to get required streams of events form this complex input network, the one of the strong functionality of Business Intelligence (BI) the Complex Event Processing (CEP) is the appropriate solution for the above mention problems. Real time processing, pattern matching, stream processing, big data management, sensor data processing and many more are the application areas of CEP. Health domain itself is a multi-dimension domain such as hospital supply chain, OPD management, disease diagnostic, In-patient, out-patient management, and emergency care etc. In this paper, the main focus is to discuss the application areas of Complex Event Processing (CEP) in health domain by using sensor device, such that how CEP manipulate health data set events coming from sensor devices such as blood pressure, heart rate, fall detection, sugar level, temperature or any other vital signs and how this systems respond to these events as quickly as possible. Different existing models and application using CEP are discussed and summarized according to different characteristics.