An event processing approach to text stream analysis

It is always difficult to manipulate the production of huge amount of data which comes from multiple sources and to extract meaningful information to make appropriate decisions. When data comes from various input resources, to get required streams of events form this complex input network, the one of the strong functionality of Business Intelligence (BI) the Complex Event Processing (CEP) is the appropriate solution for the above mention problems. Real time processing, pattern matching, stream processing, big data management, sensor data processing and many more are the application areas of CEP. Health domain itself is a multi-dimension domain such as hospital supply chain, OPD management, disease diagnostic, In-patient, out-patient management, and emergency care etc. In this paper, the main focus is to discuss the application areas of Complex Event Processing (CEP) in health domain by using sensor device, such that how CEP manipulate health data set events coming from sensor devices such as blood pressure, heart rate, fall detection, sugar level, temperature or any other vital signs and how this systems respond to these events as quickly as possible. Different existing models and application using CEP are discussed and summarized according to different characteristics.

Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams

Approach for Defining Rules in the Context of Complex Event Processing

Maritime surveillance systems are of particular importance for the smooth and safe operation of maritime traffic. Such systems must efficiently analyze the continuous stream of incoming ship movement data to provide an up-to-date picture of the situation at sea. Complex Event Processing (CEP) is a software technology dedicated to the analysis of data streams in real time and seems to be promising for maritime surveillance as well. CEP is based on rules that usually have to be formulated manually by domain experts. Recently, several approaches of learning CEP rules have been proposed. However, all these approaches are limited to standard CEP rule languages which are not well suited for describing maritime situations. In this paper, we extend Bat4CEP, an innovative bat-inspired approach to automatic CEP rule learning, to meet the specific needs of the maritime domain. In particular, we extend the rule language of Bat4CEP to include temporal-spatial operators and domain-specific functions. This extended rule language makes it possible to express complex maritime facts in a simple and understandable form. Furthermore, it is shown how the proposed rule language extensions can be integrated into the Bat4CEP learning approach. The effectiveness of the approach is demonstrated through extensive experiments with real maritime data streams.

SPARQL was originally developed as a derivative of SQL to process queries over finite-length datasets encoded as RDF graphs. Processing of infinite data streams with SPARQL has been approached by using pre-processors dividing streams into finite-length windows based on either time or the number of incoming triples. Recent extensions to SPARQL can support interconnections of queries, enabling event processing applications to be constructed out of multiple incrementally processed collaborating SPARQL update rules. With more elaborate networks of queries it is possible to perform event processing on heterogeneous event formats without strict restrictions on the number of triples per event. Heterogeneous event support combined with the capability to synthesize new events enables the creation of layered event processing systems. In this paper we review the different types of complex event processing building blocks presented in literature and show their translations to SPARQL update rules through examples, supporting a modular and layered approach. The interconnected examples demonstrate the creation of an elaborate network of SPARQL update rules for solving event processing tasks.

Complex Event Processing (CEP) is a branch of stream data mining, which is usually used to discover event patterns. Generally, a complex event processing engine uses nonprocedural declaration language and state machine to define event patterns. It generates synopses of universe data by sliding window model, therefore could identify event patterns which users care about from rapidly changing and potentially infinite event stream in a relatively short time. Among all the data streams, monitoring data flow is easy to describe by events, so CEP is suitable for supervision or decision control for business system. However, CEP does not support backtracking to eliminate invalid data, which limits CEP in supervision or decision control applications. The traditional complex event processing try to solve this problem by defining more complicated event patterns to avoid invalid results, the side effect of which would be heavier workload onto the engine. For this problem, we propose an optimized complex-event-processing model OCM. OCM will decompose complicated event patterns into small ones according to their relations, and distribute these patterns to more than one CEP engines to hierarchically filter invalid data. It can eliminate invalid data with better performance than the traditional approaches. We have also implemented the prototype and designed experiments to test the effectiveness of OCM. We have applied this model into Aviation Application in EU FP6 project Bridge.

Event processing (EP) is a data processing technology that conducts online processing of event information. In this survey, we summarize the latest cutting-edge work done on EP from both industrial and academic research community viewpoints. We divide the entire field of EP into three subareas: EP system architectures, EP use cases, and EP open research topics. Then we deep dive into the details of each subsection. We investigate the system architecture characteristics of novel EP platforms, such as Apache Storm, Apache Spark, and Apache Flink. We found significant advancements made on novel application areas, such as the Internet of Things; streaming machine learning (ML); and processing of complex data types such as text, video data streams, and graphs. Furthermore, there has been significant body of contributions made on event ordering, system scalability, development of EP languages and exploration of use of heterogeneous devices for EP, which we investigate in the latter half of this article. Through our study, we found key areas that require significant attention from the EP community, such as Streaming ML, EP system benchmarking, and graph stream processing.

Owing to Big Data popularity, the metrics Variety, Volume and Velocity (V3), are gaining importance in large scale data intensive applications. Complex Event Processing (CEP) is an efficient solution for identifying events of interest on data streams arriving from geographically distributed heterogeneous sources in near-real time. CEP is capable of handling large variety of data with high computational velocity and delivers better solution compared to other existing techniques. However, handling large volume data streams still poses challenge in CEP systems. Hence, a middleware is required to manage huge streams of data on scalable distributed environment. This paper proposes Scalable Complex Event Processing (SCALACEP) framework for managing voluminous data streams. This paper addresses challenges such as state management and efficient rule distribution for design of distributed framework for CEP using the proposed SCALACEP. This paper also proposes a novel way of indexing CEP rules based on Geometric series. The proposed indexing is used for CEP Rule allocation and CEP rule pruning in the SCALACEP system. SCALACEP is evaluated and compared with other existing systems in the constrained virtualized environment and found to give better results in terms of throughput and reduction in Latency and Multicast.

Complex event processing (CEP) evaluates queries over streams of event data to detect situations of interest. If the event data are produced by geographically distributed sources, CEP may exploit in-network processing that distributes the evaluation of a query among the nodes of a network. To this end, a query is modularized and individual query operators are assigned to nodes, especially those that act as data sources. Existing solutions for such operator placement, however, are limited in that they assume all query results to be gathered at one designated node, commonly referred to as a sink. Hence, existing techniques postulate a hierarchical structure of the network that generates and processes the event data. This largely neglects the optimisation potential that stems from truly decentralised query evaluation with potentially many sinks. To address this gap, in this paper, we propose Multi-Sink Evaluation (MuSE) graphs as a formal computational model to evaluate common CEP queries in a decentralised manner. We further prove the completeness of query evaluation under this model. Striving for distributed CEP that can scale to large volumes of high-frequency event streams, we show how to reason on the network costs induced by distributed query evaluation and prune inefficient query execution plans. As such, our work lays the foundation for distributed CEP that is both, sound and efficient.

In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

Geo-sciences involve large-scale parallel models, high resolution real time data from highly asynchronous and heterogeneous sensor networks and instruments, and complex analysis and visualization tools. Scientific workflows are an accepted approach to executing sequences of tasks on scientists’ behalf during scientific investigation. Many geo-science workflows have the need to interact with sensors that produce large continuous streams of data, but programming models provided by scientific workflows are not equipped to handle continuous data streams. This paper proposes a framework that utilizes scientific workflow infrastructure and the benefits of complex event processing to compensate for the impedance mismatch between scientific workflows and continuous data streams. Further we propose and formalize new workflow semantics that would allow the users to not only incorporate stream in scientific workflow, but also make use of the functionalities provided by the complex event processing systems effective within the scientific workflows.

This article proposes an approach to rely on the standard operators of relational algebra (including grouping and ag- gregation) for processing complex event without requiring window specifications. In this way the approach can pro- cess complex event queries of the kind encountered in appli- cations such as emergency management in metro networks. This article presents Temporal Stream Algebra (TSA) which combines the operators of relational algebra with an analy- sis of temporal relations at compile time. This analysis de- termines which relational algebra queries can be evaluated against data streams, i. e. the analysis is able to distinguish valid from invalid stream queries. Furthermore the analysis derives functions similar to the pass, propagation and keep invariants in Tucker's et al. \Exploiting Punctuation Seman- tics in Continuous Data Streams. These functions enable the incremental evaluation of TSA queries, the propagation of punctuations, and garbage collection. The evaluation of TSA queries combines bulk-wise and out-of-order processing which makes it tolerant to workload bursts as they typically occur in emergency management. The approach has been conceived for efficiently processing complex event queries on top of a relational database system. It has been deployed and tested on MonetDB.

Cloud computing and big data: Technologies and applications

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Enterprises have to be increasingly agile and responsive to address the challenges posed by the fast moving market. With the software architecture evolving into service-oriented architecture (SOA), and the adoption of radio frequency identification (RFID), event processing can fit well in enterprise information systems in terms of facilitation of event aggregation into high level actionable information, and event response to improve the responsiveness. To make it more applicable, the architecture of event processing in enterprise information systems is proposed; event meta model and context serve as the solid basis for event processing; the rules, operators and keys of complex event processing are defined. Especially, workflow model is firstly used to extract complex event pattern. We have implemented the event processing mechanism in enterprise information systems based on RFID, including the architecture, data structures, optimization strategies and algorithm. The performance evaluations show that the method is effective in terms of scalability and the capability of event processing. Complex event processing can improve operational performance and discover more actionable information, which is justified by application. Finally, lessons learned from the application are presented.