High-Performance Complex Event Processing

Goal of Complex Event Processing (CEP) is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible. Traditional CEP frameworks do not have built-in security features. Any intruder may enter into the system and may send incorrect data to CEP engine. It causes to miss some patterns or it may lead to wrong prediction of patterns. Traditional CEP systems can be exploited through one or the other way. CEP system malfunctions when it is exploited. There is no built in procedure to check the health status of traditional CEP systems. Traditional CEP systems do not have procedures to take appropriate actions when system is affected or going to affect. When CEP system identifies some patterns, it will alert corresponding personnel about the occurrence of the event. Sometimes alert message may be undelivered or unnoticed by the corresponding people. Traditional CEP systems will not provide delivery assurance of alert message. These are few security challenges to existing CEP systems. In this paper Secure Complex Event Processing Framework is proposed to address above security challenges. Proposed framework is generic and it is pluggable into any existing CEP systems.

Complex event processing is a key technology for analyzing event patterns in real-time data streams. It has been widely applied in IoT monitoring, financial risk control, smart manufacturing and other time-sensitive fields. Query patterns serve as the core matching mechanism in complex event processing systems. These patterns are primarily defined through selection policies, predicate constraints and timestamps to establish complex event detection rules. The selection policy is particularly important as it determines the sequential relationships between events. This directly affects the system's event matching efficiency and accuracy. However, in practical applications, due to different business requirements for event sampling frequencies, event streams often contain numerous consecutive events of the same type representing identical states. While all these events participate in complex event matching, traditional processing methods generate large numbers of semantically redundant partial matches. Analysis shows that only a very small proportion of these intermediate results eventually form complete matches, creating serious computational redundancy issues. This redundancy not only significantly increases system resource consumption but also reduces overall processing efficiency due to ineffective computations. To address this critical problem, this paper innovatively proposes a cache-based optimized selection policy. The core concept of this policy is to effectively reduce redundant matching results by extracting representative events from continuous event sequences. Experimental results show that this policy can significantly decrease the redundant matching rate while alleviating system load, providing new insights for optimizing complex event processing systems.

This study presents a novel method to develop kleene operators on a Binary Tree-Based Directed Graph (BTDG) to improve expressibility and performance of our developed complex event processing system. Complex Event Processing (CEP) systems are widely employed to notify opportunities and threats, which many of these situations are discovered through detection of multiple occurrences of the similar set of events. Many of CEP systems proposed kleene operators to detect these situations. The high expressibility of our implemented CEP system achieved by developing algorithms on kleene operator. These algorithms are designed on a BTDG as a unified basis to arrange input data and joining this input according to a given query. The performance is improved through directing events instantly to join with other events, dynamically switching to an optimal plan and late applying combination for kleene with specified size. The experiments shows kleene operator on BTDG brings rather high degree of expressiblity and performance in compare to its counterparts.

The nature of data in enterprises and on the Internet is changing. Data used to be stored in a database first and queried later. Today timely processing of new data, represented as events, is increasingly valuable. In many domains, complex event processing (CEP) systems detect patterns of events for decision making. Examples include processing of environmental sensor data, trades in financial markets and RSS web feeds. Unlike conventional database systems, most current CEP systems pay little attention to query optimisation. They do not rewrite queries to more efficient representations or make decisions about operator distribution, limiting their overall scalability. This paper describes the Next CEP system that was especially designed for query rewriting and distribution. Event patterns are specified in a high-level query language and, before being translated into event automata, are rewritten in a more efficient form. Automata are then distributed across a cluster of machines for detection scalability. We present algorithms for query rewriting and distributed placement. Our experiments on the Emulab test-bed show a significant improvement in system scalability due to rewriting and distribution. 1.

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain.

SUMMARYRecently, event processing (EP) has gained considerable attention as an individual discipline in computer science. From a software engineering perspective, EP systems still lack the maturity of well‐established software architectures. For the development of industrial EP systems, generally accepted software architectures based on proven design patterns and principles are still missing. In this article, we introduce a catalog of design patterns that supports the development of event‐driven architectures (EDAs) and complex EP systems. The design principles originate from experiences reported in publications as well as from our own experiences in building EP systems with industrial and academic partners. We present several patterns on different layers of abstractions that define the overall structure as well as the building blocks for EP systems. Architectural patterns that determine the top‐level structure of an EDA can be distinguished from design patterns that specify the basic mechanisms of EP. The practical application of the catalog of patterns is described by the pattern‐based design of a sample EDA for a sensor‐based energy control system. Finally, we propose a coherent and general reference architecture for EP derived from the proposed patterns.Copyright © 2013 John Wiley & Sons, Ltd.

We present in this paper logical tools for verification of Complex Event Processing (CEP) system at design time, and maintenance at runtime. Although the framework is general enough for most applications, we focus on its use in Business Process Management (BPM).

Maritime surveillance systems are of particular importance for the smooth and safe operation of maritime traffic. Such systems must efficiently analyze the continuous stream of incoming ship movement data to provide an up-to-date picture of the situation at sea. Complex Event Processing (CEP) is a software technology dedicated to the analysis of data streams in real time and seems to be promising for maritime surveillance as well. CEP is based on rules that usually have to be formulated manually by domain experts. Recently, several approaches of learning CEP rules have been proposed. However, all these approaches are limited to standard CEP rule languages which are not well suited for describing maritime situations. In this paper, we extend Bat4CEP, an innovative bat-inspired approach to automatic CEP rule learning, to meet the specific needs of the maritime domain. In particular, we extend the rule language of Bat4CEP to include temporal-spatial operators and domain-specific functions. This extended rule language makes it possible to express complex maritime facts in a simple and understandable form. Furthermore, it is shown how the proposed rule language extensions can be integrated into the Bat4CEP learning approach. The effectiveness of the approach is demonstrated through extensive experiments with real maritime data streams.

Complex event processing has become an important technology for big data and intelligent computing because it facilitates the creation of actionable, situational knowledge from potentially large amount events in soft realtime. Complex event processing can be instrumental for many mission-critical applications, such as business intelligence, algorithmic stock trading, and intrusion detection. Hence, the servers that carry out complex event processing must be made trustworthy. In this paper, we present a threat analysis on complex event processing systems and describe a set of mechanisms that can be used to control various threats. By exploiting the application semantics for typical event processing operations, we are able to design lightweight mechanisms that incur minimum runtime overhead appropriate for soft realtime computing.

Many modern applications, including online financial feeds, tag-based mass transit systems and RFID-based supply chain management systems transmit real-time data streams. There is a need for a special-purpose event stream processing technology to analyze this vast amount of sequential multi-dimensional data to enable online, operational decision making. Existing techniques such as traditional online analytical processing (OLAP) systems are not designed for real-time pattern-based operations, while state-of-the-art Complex Event Processing (CEP) systems designed for sequence detection do not support OLAP operations. Supporting complex pattern queries at different concept and pattern hierarchies must be devised by providing efficient computation and data sharing. In this dissertation, we propose a novel E-Cube model that combines CEP and OLAP techniques for multi-dimensional event pattern analysis at different abstraction levels. Further, we go beyond the linear sequence pattern queries targeted by ECube core system towards supporting an expressive composite pattern query language (composed of arbitrarily nested sequence, negation, recursion, AND and OR operators) to express powerful pattern matching requests.

Recent advances in mobile technologies have enabled a plethora of new applications. The hardware capabilities of mobile devices, however, are still insufficient for real-time stream data processing (e.g., real-time video stream). In order to process real-time streaming data, most existing applications offload the data and computation to a remote cloud service, such as Apache Storm or Apache Spark Streaming. Offloading streaming data, however, has high costs for users, e.g., significant service fees and battery consumption. To address these challenges, we design, implement and evaluate Mobile Storm, the first stream processing platform for mobile clouds, leveraging clusters of local mobile devices to process real-time stream data. In Mobile Storm, we model the workflow of a real-time stream processing job and decompose it into several tasks so that the job can be executed concurrently and in a distributed manner on multiple mobile devices. Mobile Storm was implemented on Android phones and evaluated extensively through a real-time HD video processing application. The result shows that Mobile Storm effectively processes HD Video Stream in a mobile cloud, which would be impossible on a single mobile device.

This paper presents an integrated approach to real-time processing and analysis of large-scale data streams using artificial intelligence methods, multi-agent systems, and multi-layered control architectures. Particular attention is devoted to combining technologies that are traditionally applied separately: unmanned aerial vehicle (UAV) control systems, situational centers, streaming video services, and distributed sensor networks. The proposed model accounts for the stochastic nature of the environment, the dynamic behaviour of data flows, and resource constraints, which are critical factors for real-time systems. The study develops a multi-level multi-agent architecture that includes central, regional, and edge nodes, as well as local agents such as drones, sensors, and client-side video devices. A mathematical formalization of agent interaction and reward functions is proposed to ensure a balance between service quality, latency, packet loss, and energy consumption. The use of multi-agent reinforcement learning (MARL) algorithms is introduced to support adaptive decision-making in real time, along with the development of a digital twin of the environment for predicting future system states using deep and generative models. The results demonstrate that the integration of edge computing, hierarchical decision-making, and local agent autonomy significantly enhances system resilience, provides stable performance under external disturbances, and enables effective infrastructure scaling. The findings form a scientific foundation for designing adaptive, robust, and self-learning next-generation digital ecosystems capable of intelligent processing of large data streams in real time

Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams

The use of Complex Event Processing (CEP) and Stream Processing (SP) systems to process high-volume, high-velocity Big Data has renewed interest in procedures for managing these systems. In particular, self-management and adaptation of runtime platforms have been common research themes, as most of these systems run under dynamic conditions. Nevertheless, the research landscape in this area is still young and fragmented. Most research is performed in the context of specific systems, and it is difficult to generalize the results obtained to other contexts. To enable generic and reusable CEP/SP system management procedures and self-management policies, this research introduces the Attributed Graph Rewriting for Complex Event Processing Management ( AGeCEP ) formalism. AGeCEP represents queries in a language- and technology-agnostic fashion using attributed graphs. Query reconfiguration capabilities are expressed through standardized attributes, which are defined based on a novel classification of CEP query operators. By leveraging this representation, AGeCEP also proposes graph rewriting rules to define consistent reconfigurations of queries. To demonstrate AGeCEP feasibility, this research has used it to design an autonomic manager and to define a selected set of self-management policies. Finally, experiments demonstrate that AGeCEP can indeed be used to develop algorithms that can be integrated into diverse CEP systems.

With the increasing adoption of an event-based perspective in many organizations, the demands for automatic processing of events are becoming more sophisticated. Although complex event processing systems can process events in near real-time, these systems rely heavily upon human domain experts. This becomes an issue in application areas that are rich in specialized domain knowledge and background information, such as clinical environments. We utilize a framework of four techniques to enhance complex event processing with domain knowledge from ontologies to address this issue. We realize this framework in our novel approach of ontology-supported complex event processing, which stands in contrast to related approaches and emphasizes the strengths of current advances in the individual fields of complex event processing and ontologies. Experimental results from the implementation of our approach based on a state-of-the-art system show its feasibility and indicate the direction for future research.