An Unsupervised Rule Generation Approach for Online Complex Event Processing

Goal of Complex Event Processing (CEP) is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible. Traditional CEP frameworks do not have built-in security features. Any intruder may enter into the system and may send incorrect data to CEP engine. It causes to miss some patterns or it may lead to wrong prediction of patterns. Traditional CEP systems can be exploited through one or the other way. CEP system malfunctions when it is exploited. There is no built in procedure to check the health status of traditional CEP systems. Traditional CEP systems do not have procedures to take appropriate actions when system is affected or going to affect. When CEP system identifies some patterns, it will alert corresponding personnel about the occurrence of the event. Sometimes alert message may be undelivered or unnoticed by the corresponding people. Traditional CEP systems will not provide delivery assurance of alert message. These are few security challenges to existing CEP systems. In this paper Secure Complex Event Processing Framework is proposed to address above security challenges. Proposed framework is generic and it is pluggable into any existing CEP systems.

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain.

Owing to Big Data popularity, the metrics Variety, Volume and Velocity (V3), are gaining importance in large scale data intensive applications. Complex Event Processing (CEP) is an efficient solution for identifying events of interest on data streams arriving from geographically distributed heterogeneous sources in near-real time. CEP is capable of handling large variety of data with high computational velocity and delivers better solution compared to other existing techniques. However, handling large volume data streams still poses challenge in CEP systems. Hence, a middleware is required to manage huge streams of data on scalable distributed environment. This paper proposes Scalable Complex Event Processing (SCALACEP) framework for managing voluminous data streams. This paper addresses challenges such as state management and efficient rule distribution for design of distributed framework for CEP using the proposed SCALACEP. This paper also proposes a novel way of indexing CEP rules based on Geometric series. The proposed indexing is used for CEP Rule allocation and CEP rule pruning in the SCALACEP system. SCALACEP is evaluated and compared with other existing systems in the constrained virtualized environment and found to give better results in terms of throughput and reduction in Latency and Multicast.

Sequence Clustering-based Automated Rule Generation for Adaptive Complex Event Processing

Complex event processing (CEP) over event streams has become increasingly important for real-time applications ranging from health care, supply chain management to business intelligence. These monitoring applications submit complex event queries to track sequences of events that match a given pattern. As these systems mature the need for increasingly complex nested sequence query support arises, while the state-of-art CEP systems mostly support the execution of only flat sequence queries. In this paper, we introduce our nested CEP query language NEEL for expressing nested queries composed of sequence, negation, AND and OR operators. Thereafter, we also define its formal semantics. Subtle issues with negation and predicates within the nested sequence context are discussed. An E-Analytics system for processing nested CEP queries expressed in the NEEL language has been developed. Lastly, we demonstrate the utility of this technology by describing a case study of applying this technology to a real-world application in health care.

This paper describes a scalable complex event processing (CEP) system and rule allocation algorithm for large-scale M2M services and presents the results of its performance evaluations. Platforms providing M2M services deal with various data from massive event sources in the real world. To realize these services, a scalable system is required. Our previous system determines efficient allocation of CEP rules based on a single key attribute type. However, it cannot maintain scalability in M2M situation because CEP rules are defined by various attribute types and there is no single key attribute, i.e. common attribute type among CEP rules. The challenging issue in the present work is how to maintain scalability where no common attribute type usable for rule allocation. The key idea in this paper is to allocate CEP rules based on different attribute types for each CEP rule instead of single key attribute type. We propose a rule allocation algorithm that determines the efficient allocation of CEP rules when the system needs processing a large number of events, a large number of CEP rules, and various types of events. Evaluation results show that the system with the proposed algorithm is scalable with many CEP Servers. We thus conclude that the proposed algorithm contributes to the realization of large-scale M2M services.

In-network computing using programmable networking hardware is a strong trend in networking that promises to reduce latency and consumption of server resources through offloading to network elements (programmable switches and smart NICs). In particular, the data plane programming language P4 together with powerful P4 networking hardware has spawned projects offloading services into the network, e.g., consensus services or caching services. In this paper, we present a novel case for in-network computing, namely, Complex Event Processing (CEP). CEP processes streams of basic events, e.g., stemming from networked sensors, into meaningful complex events. Traditionally, CEP processing has been performed on servers or overlay networks. However, we argue in this paper that CEP is a good candidate for in-network computing along the communication path avoiding detouring streams to distant servers to minimize communication latency while also exploiting processing capabilities of novel networking hardware. We show that it is feasible to express CEP operations in P4 and also present a tool to compile CEP operations, formulated in our P4CEP rule specification language, to P4 code. Moreover, we identify challenges and problems that we have encountered to show future research directions for implementing full-fledged in-network CEP systems.

In this paper, we present a powerful logic-based approach for Complex Event Processing (CEP). The approach is founded on a logical transformation of event processing into logic programs. Many systems for event processing have ad-hoc semantics with unexpected behaviors. Further on, many formal approaches (with well defined semantics) cannot effectively be used in CEP, due to their inability to compute events in the data-driven fashion. Our approach enables both logic-based and data-driven complex event detection. Moreover, the backward chaining approach allows for very efficient reasoning of complex events and actions triggered by these events.

With the increasing adoption of an event-based perspective in many organizations, the demands for automatic processing of events are becoming more sophisticated. Although complex event processing systems can process events in near real-time, these systems rely heavily upon human domain experts. This becomes an issue in application areas that are rich in specialized domain knowledge and background information, such as clinical environments. We utilize a framework of four techniques to enhance complex event processing with domain knowledge from ontologies to address this issue. We realize this framework in our novel approach of ontology-supported complex event processing, which stands in contrast to related approaches and emphasizes the strengths of current advances in the individual fields of complex event processing and ontologies. Experimental results from the implementation of our approach based on a state-of-the-art system show its feasibility and indicate the direction for future research.

Complex event processing is a key technology for analyzing event patterns in real-time data streams. It has been widely applied in IoT monitoring, financial risk control, smart manufacturing and other time-sensitive fields. Query patterns serve as the core matching mechanism in complex event processing systems. These patterns are primarily defined through selection policies, predicate constraints and timestamps to establish complex event detection rules. The selection policy is particularly important as it determines the sequential relationships between events. This directly affects the system's event matching efficiency and accuracy. However, in practical applications, due to different business requirements for event sampling frequencies, event streams often contain numerous consecutive events of the same type representing identical states. While all these events participate in complex event matching, traditional processing methods generate large numbers of semantically redundant partial matches. Analysis shows that only a very small proportion of these intermediate results eventually form complete matches, creating serious computational redundancy issues. This redundancy not only significantly increases system resource consumption but also reduces overall processing efficiency due to ineffective computations. To address this critical problem, this paper innovatively proposes a cache-based optimized selection policy. The core concept of this policy is to effectively reduce redundant matching results by extracting representative events from continuous event sequences. Experimental results show that this policy can significantly decrease the redundant matching rate while alleviating system load, providing new insights for optimizing complex event processing systems.

Complex Event Processing (CEP) systems aim at processing large flows of events to discover situations of interest. In CEP, the processing takes place according to user-defined rules, which specify the (causal) relations between the observed events and the phenomena to be detected. We claim that the complexity of writing such rules is a limiting factor for the diffusion of CEP. In this paper, we tackle this problem by introducing iCEP, a novel framework that learns, from historical traces, the hidden causality between the received events and the situations to detect, and uses them to automatically generate CEP rules. The paper introduces three main contributions. It provides a precise definition for the problem of automated CEP rules generation. It dicusses a general approach to this research challenge that builds on three fundamental pillars: decomposition into subproblems, modularity of solutions, and ad-hoc learning algorithms. It provides a concrete implementation of this approach, the iCEP framework, and evaluates its precision in a broad range of situations, using both synthetic benchmarks and real traces from a traffic monitoring scenario.

With the broader dissemination of digital technologies, visionary concepts like the Internet of Things also affect an increasing number of use cases with interfaces to humans, e.g. manufacturing environments with technical operators monitoring the processes. This leads to additional challenges, as besides the technical issues also human aspects have to be considered for a successful implementation of strategic initiatives like Industrie 4.0. From a technical perspective, complex event processing has proven itself in practice to be capable of integrating and analyzing huge amounts of heterogeneous data and establishing a basic level of situation awareness by detecting situations of interests. Whereas this reactive nature of complex event processing systems may be sufficient for machine-to-machine use cases, the new characteristic of application fields with humans remaining in the control loop leads to an increasing action distance and delayed reactions. Taking human aspects into consideration leads to new requirements, with transparency and comprehensibility of the processing of events being the most important ones. Improving the comprehensibility of complex event processing and extending its capabilities towards an effective support of human operators allows tackling technical and non-technical challenges at the same time. The main contribution of this thesis answers the question of how to evolve state-of-the-art complex event processing from its reactive nature towards a transparent and holistic situation management system. The goal is to improve the interaction among systems and humans in use cases with interfaces between both worlds. Realizing a holistic situation management requires three missing capabilities to be introduced by the contributions of this thesis: First, based on the achieved transparency, the retrospective analysis of situations is enabled by collecting information related to a situation's occurrence and development. Therefore, CEP engine-specific situation descriptions are transformed into a common model, allowing the automatic decomposition of the underlying patterns to derive partial patterns describing the intermediate states of processing. Second, by introducing the psychological model of situation awareness into complex event processing, human aspects of information processing are taken into consideration and introduced into the complex event processing paradigm. Based on this model, an extended situation life-cycle and transition method are derived. The introduced concepts and methods allow the implementation of the controlling function of situation management and enable the effective acquisition and maintenance of situation awareness for human operators to purposefully direct their attention towards upcoming situations. Finally, completing the set of capabilities for situation management, an approach is presented to support the generation and integration of prediction models for predictive situation management. Therefore, methods are introduced to automatically label and extract relevant data for the generation of prediction models and to enable the embedding of the resulting models for an automatic evaluation and execution. The contributions are introduced, applied and evaluated along a scenario from the manufacturing domain.

Complex event processing has become an important technology for big data and intelligent computing because it facilitates the creation of actionable, situational knowledge from potentially large amount events in soft realtime. Complex event processing can be instrumental for many mission-critical applications, such as business intelligence, algorithmic stock trading, and intrusion detection. Hence, the servers that carry out complex event processing must be made trustworthy. In this paper, we present a threat analysis on complex event processing systems and describe a set of mechanisms that can be used to control various threats. By exploiting the application semantics for typical event processing operations, we are able to design lightweight mechanisms that incur minimum runtime overhead appropriate for soft realtime computing.

Enterprises have to be increasingly agile and responsive to address the challenges posed by the fast moving market. With the software architecture evolving into service-oriented architecture (SOA), and the adoption of radio frequency identification (RFID), event processing can fit well in enterprise information systems in terms of facilitation of event aggregation into high level actionable information, and event response to improve the responsiveness. To make it more applicable, the architecture of event processing in enterprise information systems is proposed; event meta model and context serve as the solid basis for event processing; the rules, operators and keys of complex event processing are defined. Especially, workflow model is firstly used to extract complex event pattern. We have implemented the event processing mechanism in enterprise information systems based on RFID, including the architecture, data structures, optimization strategies and algorithm. The performance evaluations show that the method is effective in terms of scalability and the capability of event processing. Complex event processing can improve operational performance and discover more actionable information, which is justified by application. Finally, lessons learned from the application are presented.

Complex Event Processing (CEP) systems extract interest situations from event flows based on event detection patterns. However, local event processing for distributed Internet of Things (IoT) has not been discussed yet. Besides, it is complex or impossible to discover such patterns in some applications of IoT. In this article, we design a complex event service to process event flows based on IoT resource models, which does not depend on existing patterns, and deals with both discrete events and continuous variables. To improve the CEP performance, local IoT resources are used for local event processing, and a lazy exchange method is designed to realize the collaborated event processing between network edges and a data center. Our evaluation shows that our solution is feasible and effective.