Streaming Tables: Native Support to Streaming Data in DBMSs

In recent years, data stream technology has been widely used in the processing of sensor data. Sonar sensors produce high-speed streaming data with special characteristics of huge volume continuously. Therefore, general data stream management systems with limited memory and processor resources leads to inefficiencies in processing this kind of sonar data stream. However, in distributed data stream management system (DDSMS), continuous query can be reduced into several steps and processed on different processing nodes efficiently and effectively, so DDSMS is often adopted to deal with sonar stream data. Load balancing is a key technology of DDSMS, existing researches mainly focus on using computing load mainly caused by processing queries on the node as the key factor of load balancing algorithm, ignoring the load of transmitting huge volume data mainly caused by moving data among nodes. In this paper, the transmission load of sonar data is computed and used in an improved load balancing algorithm, which is proposed to solve the problem of load scheduling for sonar stream data processing under the distributed environment. The simulation verifies the importance of considering transmission load and shows a better performance of improved load balancing algorithm.

데이터 스트림은 빠르게 연속적으로 발생하는 무제한의 데이터 튜플의 집합이다. 이러한 데이터 스트림에 대한 질의 처리 또한 연속적이고 신속해야 하며 엄격한 시공간적 제약이 요구된다. 대부분의 데이터 스트림 관리시스템(DSMS)에서는 시공간적 제약사항을 효과적으로 지키기 위해서 등록된 연속 질의들의 선택 조건(selection predicate)들을 그룹화하거나 색인처리 한다. 본 논문에서는 연속 질의들의 선택 조건들을 속성별로 그룹화한 새로운 구조체인 속성 선택체(Attribute Selection Construct)를 제안한다. 속성 선택체에는 해당 속성이 특정 질의조건에 사용되는지 여부, 부분적으로 미리 계산된 질의결과 정보, 그리고 해당 속성의 선택률 통계 등 효율적인 질의 처리를 위한 유용한 정보들이 포함된다. 또한, 대상 질의집합을 구현한 속성 선택체들 간의 처리 순서는 전체적인 질의성능에 많은 영향을 미칠 수 있기 때문에 효과적으로 속성 선택체 처리 순서를 결정할 수 있는 전략도 함께 제안된다. 마지막으로, 기존의 방법들이 포함된 다양한 실험을 통하여 제안된 방법론의 성능을 여러 각도에서 비교 검증한다. A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Query processing for such a data stream should also be continuous and rapid, which requires strict time and space constraints. In most DSMS(Data Stream Management System), the selection predicates of continuous queries are grouped or indexed to guarantee these constraints. This paper proposes a new scheme tailed an ASC(Attribute Selection Construct) that collectively evaluates selection predicates containing the same attribute in multiple continuous queries. An ASC contains valuable information, such as attribute usage status, partially pre calculated matching results and selectivity statistics for its multiple selection predicates. The processing order of those ASC's that are corresponding to the attributes of a base data stream can significantly influence the overall performance of multiple query evaluation. Consequently, a method of establishing an efficient evaluation order of multiple ASC's is also proposed. Finally, the performance of the proposed method is analyzed by a series of experiments to identify its various characteristics.

Because of increased stream data, managing stream data has become quite important. This paper describes our data stream management system, which employs an architecture combining a stream processing engine and DBMS. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Our proposed query language supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data. Users can also specify continuous queries that integrate streaming data and historical data stored in DBMS. Another contribution of this paper is feasibility validation of queries. Processing queries on streams with frequent inputs may cause the system to overflow its capacity. Specifically, the maximum writing rate to DBMS is a significant bottleneck when we try to store stream data into DBMS. Our system detects infeasible queries in advance.

Energy Management Systems (EMSs) are used to monitor energy consumption in buildings with the purpose of improving energy efficiency, by identifying savings opportunities and misuse situations. To achieve that, an EMS collects energy metering data streams from a network of energy meters in the building. Sensor data must be processed in (near) real-time, to support a timely decision making process. Currently, EMSs are using traditional DBMSs to process such data, introducing a persistence step that translates to an unacceptable latency on data evaluation. Moreover, sensor data monitoring queries are not elegantly supported by the SQL query language, thus hampering the ability of an EMS to process energy metering data in realtime. Data Stream Management Systems (DSMSs) are used to process data streams efficiently in several domains. Many sensor network monitoring applications have been implemented upon DSMSs resulting in significant improvements on performance and overall resource usage. This work validates the hypothesis proposed in [1] that, to process energy metering data streams in realtime, EMSs should be supported by DSMSs, instead of DBMSs. We introduce an EMS’s Data Processing Architecture supported by a DSMS that supports the implementation of an EMS capable of performing real-time data processing. We validate our solution through a benchmark evaluation against a DBMS based architecture. The results clearly show that the DSMSbased EMS outperformed the state of the art approach, both in data evaluation latency and query language expressibility— demonstrating its adequacy to process energy metering data streams in real-time.

In recent years, we have witnessed the emergence of new types of systems that deal with large volumes of streaming data.Examples include financial data analysis on feeds of stock tickers, sensorbased environmental monitoring, network track monitoring and click stream analysis to push customized advertisements or intrusion detection.Traditional database management systems (DBMS), which are very good at managing large volumes of stored data, are not suitable for this, as streaming needs low-latency processing on live data from push-based sources.Data Stream Management Systems (DSMS) are fast emerging to address this new type of data, but faces challenging issues, such as unpredictable data arrival rate.On bursty mode, processing need surpasses available system capacity affecting the Quality of Service (QoS) adversely.The system overloading is even more acute in XML data streams compared to relational streams due to its extra resource requirements for data preparation and result construction.The main focus of this paper is to find out suitable ways to process this high volume of data streams dealing with the spikes in data arrival gracefully, under limited or fixed system resources in the XML stream context.

Most data stream management systems are based on extensions of the relational data model and query languages, but rigorous analyses of the problems and limitations of this approach, and how to overcome them, are still wanting. In this article, we elucidate the interaction between stream-oriented extensions of the relational model and continuous query language constructs, and show that the resulting expressive power problems are even more serious for data streams than for databases. In particular, we study the loss of expressive power caused by the loss of blocking query operators, and characterize nonblocking queries as monotonic functions on the database. Thus we introduce the notion of NB -completeness to assure that a query language is as suitable for continuous queries as it is for traditional database queries. We show that neither RA nor SQL are NB -complete on unordered sets of tuples, and the problem is even more serious when the data model is extended to support order—a sine-qua-non in data stream applications. The new limitations of SQL, compounded with well-known problems in applications such as sequence queries and data mining, motivate our proposal of extending the language with user-defined aggregates (UDAs). These can be natively coded in SQL, according to simple syntactic rules that set nonblocking aggregates apart from blocking ones. We first prove that SQL with UDAs is Turing complete. We then prove that SQL with monotonic UDAs and union operators can express all monotonic set functions computable by a Turing machine ( NB -completeness) and finally extend this result to queries on sequences ordered by their timestamps. The proposed approach supports data stream models that are more sophisticated than append-only relations, along with data mining queries, and other complex applications.

The article provides the analysis of the current situation in the use of data stream management systems (DSMS) and discusses the reasons why this technology is not used to develop geographic information systems. DSMS, despite its novelty, has ceased to be a pure research project and is used in industrial applications. However, this technology is not used to design the GIS, although the necessity of processing and analyzing of spatio-temporal data streams arises in many practically important applications. The essence of the current problematic situation is the gap between new technological capabilities and the lack of a theoretical framework for the processing and analysis of spatio-temporal data streams in DSMS. Existing spatial analytics algorithms are designed for relational databases with precomputed spatial indexes and are not suitable for DSMS. The article shows that, to resolve the current problematic situation with the geoinformation systems development based on DSMS should do the following

There has been a rising need to handle and process streaming kind of data. It is continuous, unpredictable, time-varying in nature and could arrive in multiple rapid streams. Sensor data, web clickstreams, etc. are the examples of streaming data. One of the important issues about streaming data management systems is that it needs to be processed in real-time. That is, active rules can be defined over data streams for making the system reactive. These rules are triggered based on the events detected on the data stream, or events detected while summarizing the data or combination of both. In this paper, we study the challenges involved in monitoring events in a Data Stream Management System (DSMS) and how they differ from the same in active databases. We propose an architecture for event composition and detection in a DSMS, and then discuss an algorithm for detecting composite events defined on both the summarized data streams and the streaming data.KeywordsEvent AlgebraComposite EventEvent CompositionStreaming DataContinuous QueryThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Real-time monitoring systems are important for industry since they allow for avoiding unplanned system stops and keeping system availability high. The technical requirements for such systems include being both scalable and online, as the amount of generated data is increasing with time. Therefore, monitoring systems must integrate tools that can manage and analyze the data streams. The data stream management system is a stream processing tool that has the ability to manage and support operations on data streams in real-time. Several researchers have proposed and tested real-time monitoring systems which have the ability to search big data streams. In this paper, the research works that discuss the analysis of online data streams for fault detection in industry are reviewed. Based on the literature analysis, the industrial needs and challenges of monitoring big data streams are presented. Furthermore, feasible suggestions for improving the real-time monitoring system are proposed.

The continuous sliding-window query model is used widely in data stream management systems where the focus of a continuous query is limited to a set of the most recent tuples. In this paper, we show that an interesting and important class of queries over data streams cannot be answered using the sliding-window query model. Thus, we introduce a new model for continuous window queries, termed the predicate-window query model that limits the focus of a continuous query to the stream tuples that qualify a certain predicate. Predicate-window queries have some distinguishing characteristics, e.g., (1) The window predicate can be defined over any attribute in the stream tuple (ordered or unordered). (2) Stream tuples qualify and disqualify the window predicate in an out-of-order manner. In this paper, we discuss the applicability of the predicate-window query model. We will show how the existing sliding-window query models fail to answer some of the predicate-window queries. Finally, we discuss the challenges in supporting the predicate-window query model in data stream management systems.

In Continuous Data Analytics and in monitoring applications, hundreds of similar Aggregate Continuous Queries (ACQs) are registered at the Data Stream Management System (DSMS) to continuously monitor the infinite input stream of data tuples. Optimizing the processing of these ACQs is crucial in order for the DSMS to operate at the adequate required scalability. One optimization technique is to share the results of partial aggregation operations between different ACQs on the same data stream. However, finding the query execution plan that attains maximum reduction in total plan cost is computationally expensive. Weave Share, a multiple ACQs optimizer that computes query plans in a greedy fashion, was recently shown in experiments to achieve more than an order of magnitude improvement over the best existing alternatives. Maximizing the benefit of sharing, i.e., maximizing the cost-savings achieved by sharing partial aggregation results, is the goal of Weave Share. In this paper we prove that Weave Share approximates the optimal cost-savings to within a factor of 4 for a practical variant of the problem. To the best of our knowledge, this is the first theoretical guarantee provided for this problem. We also provide exact solutions for two natural special cases.

Traditional decision support systems (DSS) and executive information systems (EIS) gather and present information from several sources for business purposes. It is an information technology to help the knowledge worker (executive, manager, analyst) make faster and better decisions. So far, these data were stored statically and persistently in a database, typically in a data warehouse. Data warehouses collect masses of operational data, allowing analysts to extract information by issuing decision support queries on the otherwise discarded data. In a typical scenario, an organization stores a detailed record of its operations in a database, which is then analyzed to improve efficiency, detect sales opportunities, and so on. Performing complex analysis on these data is an essential component of these organizations’ businesses. Chaudhuri and Dayal (1997) present an excellent survey on decision-making and online analytical processing (OLAP) technologies for traditional database systems. ?n many applications however, it may not be possible to process queries within a database management system (DBMS). These applications involve data items that arrive online from multiple sources in a continuous, rapid and time-varying fashion (Babcock et. al., 2002). These data may or may not be stored in a database. As a result, a new class of data-intensive applications has recently attracted a lot of attention: applications in which the data is modeled not as persistent relations but rather as transient data streams. Examples include financial applications (streams of transactions or ticks), network monitoring (stream of packets), security, telecommunication data management (stream of calls or call packets), web applications (clickstreams), manufacturing, wireless sensor networks (measurements), RFID data, and others. In data streams we usually have “continuous” queries (Terry et. al., 1992; Babu & Widom, 2002) rather than “one-time.” The answer to a continuous query is produced over time, reflecting the stream data seen so far. Answers may be stored and updated as new data arrives or may be produced as data streams themselves. Continuous queries can be used for monitoring, alerting, security, personalization, etc. Data streams can be either transactional (i.e., log interactions between entities, such as credit card purchases, web clickstreams, phone calls), or measurement (i.e., monitor evolution of entity states, such as physical phenomena, road traffic, temperature, network). How to best model, express and evaluate complex queries over data streams is an open and difficult problem. This involves data modeling, rich querying capabilities to support real-time decision support and mining, and novel evaluation and optimization processing techniques. In addition, the kind of decision support over data streams is quite different from “traditional” decision-making: decisions are “tactical” rather than “strategic.” Research on data streams is currently among the most active areas in database research community. Flexible and efficient stream querying will be a crucial component of any future data management and decision support system (Abiteboul et al., 2005).

In many applications, the information required by the user cannot be found in just one source, but has to be retrieved from many varying sources. This is true not only of formatted data in database management systems, but also of textual documents and multimedia data, such as images and videos. We propose a mediator system that provides the end-user with a single query interface to an integrated view of multiple heterogeneous data sources. We exploit the capabilities of the MOMIS integration system and the MILOS multimedia data management system. Each multimedia source is managed by an instance of MILOS, in which a collection of multimedia records is made accessible by means of similarity searches employing the query-by-example paradigm. MOMIS provides an integrated virtual view of the underlying multimedia sources, thus offering unified multimedia access services. Two features are that MILOS is flexible--it is not tied to any particular similarity function--and the MOMIS's mediator query processor only exploits the ranks of the local answers.

Recently, there is an increasing number of applications that require support for real-time processing of rapidly arriving data streams. The high volume and fast arrival rate make it infeasible to install every data item of a stream into the database. In this study, we assume transactions running over data streams execute periodically to produce new results as temporal data to be installed into the database. We call these transactions continuous transactions. They resemble continuous queries in data stream management systems plus reading non-temporal data for creating and writing temporal data. Continuous transactions are continuous, standing and persistent with pre-declared data access pattern. Conventional real-time concurrency control protocols are not designed for systems with both continuous transactions and user transactions without prior knowledge of data access pattern. We need to ensure that user transactions can read fresh data and commit before their deadlines. On the other hand, we need to issue sufficient privilege for continuous transactions to keep temporal data fresh without violating database consistency and significantly affecting the performance of user transactions. Two new time-cognizant forced wait protocols are proposed tohandle the interplay between these two types of transactions with different characteristics. These protocols are evaluated through a series of simulation experiments. We have demonstrated that taking advantage of continuous transaction and temporal data semantics in transaction scheduling can significantly improve the performance of user transactions without jeopardizing the freshness of temporal data maintained by continuous transactions.

Nowadays, more and more organizations keep their valuable and sensitive data in Database Management Systems (DBMSs). The traditional database security mechanisms such as access control mechanisms, authentication, data encryption technologies do not offer a strong enough protection against the exploitation of vulnerabilities (e.g. intrusions) in DBMSs from insiders. Intrusion detection systems recently proposed in the literature focus on statistical approaches, which are not intuitive. Our research is the first ever effort to use process mining modeling low-level event logs for database intrusion detection. We have proposed a novel approach for visualizing database intrusion detection using process mining techniques. Our experiments showed that intrusion detection visualization will be able to help security officers who might not know deeply the complex system, identify the true positive detection and eliminate the false positive results.