The Metamorphosis (of RAM3S)

Real-time monitoring systems are important for industry since they allow for avoiding unplanned system stops and keeping system availability high. The technical requirements for such systems include being both scalable and online, as the amount of generated data is increasing with time. Therefore, monitoring systems must integrate tools that can manage and analyze the data streams. The data stream management system is a stream processing tool that has the ability to manage and support operations on data streams in real-time. Several researchers have proposed and tested real-time monitoring systems which have the ability to search big data streams. In this paper, the research works that discuss the analysis of online data streams for fault detection in industry are reviewed. Based on the literature analysis, the industrial needs and challenges of monitoring big data streams are presented. Furthermore, feasible suggestions for improving the real-time monitoring system are proposed.

In this work a new approach to the construction of models and logic circuits of algorithms and procedures for information technology processing and analysis of big data streams within the fractal paradigm is describes. In this case, data streams are defined as information objects whose physical nature can be arbitrary. The information object is investigated apart from any model or scheme, the logical scheme of intellectual technology is built in the form of: facts, regularities and reality. Fractal methods form the framework of the logical, algorithmic and content essence of the approach. The basic premise of the approach is as follows. First, the processing and analysis of data stream to determine whether it forms a fractal structure and construct a phase portrait of data stream as an information object. Second, to distinguish the areas of fractal percolation and aggregation in the multifractal structure of the stream, the phase portrait is used. Third, to estimate the spatial and temporal scales of fractal percolation and aggregation processes.

Immersive, interactive applications grouped under the concept of Immersipresence require on-line processing and mixing of multimedia data streams and structures. One critical issue seldom addressed is the integration of different solutions to technical challenges, developed independently in separate fields, into working systems, that operate under hard performance constraints. In order to realize the Immersipresence vision, a consistent, generic approach to system integration is needed, that is adapted to the constraints of research development. This paper introduces SAI, a new software architecture model for designing, analyzing and implementing applications performing distributed, asynchronous parallel processing of generic data streams. SAI provides a universal framework for the distributed implementation of algorithms and their easy integration into complex systems that exhibit desirable software engineering qualities such as efficiency, scalability, extensibility, reusability and interoperability. The SAI architectural style and its properties are described. The use of SAI and of its supporting open source middleware (MFSM) is illustrated with integrated, distributed interactive systems.

Many database applications require the analysis and processing of data streams. In such systems, huge amounts of data arrive rapidly and their values change over time. The variations on streams typically imply some fundamental changes of the underlying objects and possess significant domain meanings. In some data streams, successive events seem to recur in a certain time interval, but the data indeed evolves with tiny differences as time elapses. This feature is called pseudo periodicity, which poses a non-trivial challenge to variation management in data streams. This paper presents our research effort in online variation management over such streams, and the idea can be applied to the problem domain of medical applications, such as patient vital signal monitoring. We propose a new method named Pattern Growth Graph (PGG) to detect and manage variations over pseudo periodical streams. PGG adopts the wave-pattern to capture the major information of data evolution and represent them compactly. With the help of wave-pattern matching algorithm, PGG detects the stream variations in a single pass over the stream data. PGG only stores the different segments of the pattern for incoming stream, and hence it can substantially compress the data without losing important information. The statistical information of PGG helps to distinguish meaningful data changes from noise and to reconstruct the stream with acceptable accuracy. Extensive experiments on real datasets containing millions of data items demonstrate the feasibility and effectiveness of the proposed scheme.

Internet of Things (IoT) provides a promising opportunity to build powerful data analytics systems with real time event detection for smart health, and therefore wearable IoT has become a rising source of big data streams for smart health, for which security needs to be assured by detecting real-time event to avoid malicious activities, and meanwhile to control the information leakage of big sensing data streams. I refer to this as an information flow control problem. To address this problem, this paper proposes a static lattice model for information flow control over big sensing data streams. I initialize two static lattices, i.e., sensor lattice for wearable sensors and user lattice for users, and then static lattices aim to process the flow control model faster, because I am dealing with high volume and velocity of data streams. The experimental evaluation and results of the information flow model show that it can excellently handle the incoming big data streams with low latency and buffer requirement.

This paper presents an integrated approach to real-time processing and analysis of large-scale data streams using artificial intelligence methods, multi-agent systems, and multi-layered control architectures. Particular attention is devoted to combining technologies that are traditionally applied separately: unmanned aerial vehicle (UAV) control systems, situational centers, streaming video services, and distributed sensor networks. The proposed model accounts for the stochastic nature of the environment, the dynamic behaviour of data flows, and resource constraints, which are critical factors for real-time systems. The study develops a multi-level multi-agent architecture that includes central, regional, and edge nodes, as well as local agents such as drones, sensors, and client-side video devices. A mathematical formalization of agent interaction and reward functions is proposed to ensure a balance between service quality, latency, packet loss, and energy consumption. The use of multi-agent reinforcement learning (MARL) algorithms is introduced to support adaptive decision-making in real time, along with the development of a digital twin of the environment for predicting future system states using deep and generative models. The results demonstrate that the integration of edge computing, hierarchical decision-making, and local agent autonomy significantly enhances system resilience, provides stable performance under external disturbances, and enables effective infrastructure scaling. The findings form a scientific foundation for designing adaptive, robust, and self-learning next-generation digital ecosystems capable of intelligent processing of large data streams in real time

The querying and analysis of data streams has been a topic of much recent interest, motivated by applications from the fields of networking, web usage analysis, sensor instrumentation, telecommunications, and others. Many of these applications involve monitoring answers to continuous queries over data streams produced at physically distributed locations, and most previous approaches require streams to be transmitted to a single location for centralized processing. Unfortunately, the continual transmission of a large number of rapid data streams to a central location can be impractical or expensive. We study a useful class of queries that continuously report the k largest values obtained from distributed data streams (top-k monitoring queries), which are of particular interest because they can be used to reduce the overhead incurred while running other types of monitoring queries. We show that transmitting entire data streams is unnecessary to support these queries and present an alternative approach that reduces communication significantly. In our approach, arithmetic constraints are maintained at remote stream sources to ensure that the most recently provided top-k answer remains valid to within a user-specified error tolerance. Distributed communication is only necessary on occasion, when constraints are violated, and we show empirically through extensive simulation on real-world data that our approach reduces overall communication cost by an order of magnitude compared with alternatives that o er the same error guarantees.

Due to the explosive growth of the Internet and increasing service demands from all around the world, the cluster-based system that consists of one request-dispatching server and several request-handling servers has become a cost-effective way to serve the huge amount of service demands. Nowadays, Web servers have to handle more complex types of requests since requests from clients may be mixed with dynamic Web pages, database processing, or multimedia stream data. Therefore, a Web cluster should be designed with intelligent request dispatching policies for supporting various types of service requests.

This paper presents the design and the implementation of an architecture for the analysis of data streams in distributed environments. In particular, data stream analysis has been carried out for the computation of items and item sets that exceed a frequency threshold. The mining approach is hybrid, that is, frequent items are calculated with a single pass, using a sketch algorithm, while frequent item sets are calculated by a further multi-pass analysis. The architecture combines parallel and distributed processing to keep the pace with the rate of distributed data streams. In order to keep computation close to data, miners are distributed among the domains where data streams are generated. The paper also reports the experimental results obtained with a prototype of the architecture, tested on a Grid composed of two domains handling two different data streams.

The article provides the analysis of the current situation in the use of data stream management systems (DSMS) and discusses the reasons why this technology is not used to develop geographic information systems. DSMS, despite its novelty, has ceased to be a pure research project and is used in industrial applications. However, this technology is not used to design the GIS, although the necessity of processing and analyzing of spatio-temporal data streams arises in many practically important applications. The essence of the current problematic situation is the gap between new technological capabilities and the lack of a theoretical framework for the processing and analysis of spatio-temporal data streams in DSMS. Existing spatial analytics algorithms are designed for relational databases with precomputed spatial indexes and are not suitable for DSMS. The article shows that, to resolve the current problematic situation with the geoinformation systems development based on DSMS should do the following

Recent advances and widespread usage of online web services and social media platforms, coupled with ubiquitous low cost devices, mobile technologies, and increasing capacity of lower cost storage, has led to a proliferation of Big data, ranging from, news, e-commerce clickstreams, and online business transactions to continuous event logs and social media expressions. These large amounts of online data, often referred to as data streams, because they get generated at extremely high throughputs or velocity, can make conventional and classical data analytics methodologies obsolete. For these reasons, the issues of management and analysis of data streams have been researched extensively in recent years. The special case of social media Big Data brings additional challenges, particularly because of the unstructured nature of the data, specifically free text. One classical approach to mine text data has been Topic Modeling. Topic Models are statistical models that can be used for discovering the abstract ``topics'' that may occur in a corpus of documents. Topic models have emerged as a powerful technique in machine learning and data science, providing a great balance between simplicity and complexity. They also provide sophisticated insight without the need for real natural language understanding. However they have not been designed to cope with the type of text data that is abundant on social media platforms, but rather for traditional medium size corpora consisting of longer documents, adhering to a specific language and typically spanning a stable set of topics. Unlike traditional document corpora, social media messages tend to be very short, sparse, noisy, and do not adhere to a standard vocabulary, linguistic patterns, or stable topic distributions. They are also generated at high velocity that impose high demands on topic modeling; and their evolving or dynamic nature, makes any set of results from topic modeling quickly become stale in the face of changes in the textual content and topics discussed within social media streams. In this dissertation, we propose an integrated topic modeling framework built on top of an existing stream-clustering framework called Stream-Dashboard, which can extract, isolate, and track topics over any given time period. In this new framework, Stream Dashboard first clusters the data stream points into homogeneous groups. Then data from each group is ushered to the topic modeling framework which extracts finer topics from the group. The proposed framework tracks the evolution of the clusters over time to detect milestones corresponding to changes in topic evolution,

A big data exploration approach to exploit in-vehicle data for smart road maintenance

Stream processing platforms allow applications to analyse incoming data continuously. Several use cases exist that make use of these capabilities, ranging from monitoring of physical infrastructures to pre selecting video surveillance feeds for human inspection. It is difficult to predict how much computing resources are needed for these stream processing platforms, because the volume and velocity of input data may vary over time. The open source Apache Storm software provides a framework for developers to build processing applications that use the computing resources of all machines within an established cluster. Because of the varying processing needs of such applications, the platform should be able to automatically grow and shrink as needed. Unfortunately, the current Storm platform does not provide this capability. In this paper we describe the design and implementation of a tool that monitors several aspects of the Storm platform, the applications running on top of it, and external systems such as queues and databases. Based on this information, the tool decides whether extra servers are needed or machines may be decommissioned from the cluster, and then acts upon it by actually creating new virtual machines or shutting them down.

Nowadays, a huge quantity of data are produced by means of multiple data sources. The existing tools and techniques are not capable of handling such voluminous data produced from a variety of sources. This continuous and varied generation of data requires advanced technologies for processing and storage, which seems to be a big challenge for data scientists. Some research studies are well defined in the area of streaming in big data. Streaming data are the real-time data or data in motion such as stock market data, sensor data, GPS data and twitter data. In stream processing, the data are not stored in databases instead it is processed and analyzed on the fly to get the value as soon as they are generated. There are a number of streaming frameworks proposed till date for big data applications that are used to pile up, evaluate and process the data that are generated and captured continuously. In this chapter, we provide an in-depth summary of various big data streaming approaches like Apache Storm, Apache Hive and Apache Samza. We also presented a comparative study regarding these streaming platforms.

The real time analysis of massive data streams is of utmost importance in data intensive applications that need to detect as fast as possible and as efficiently as possible (in terms of computation and memory space) any correlation between its inputs or any deviance from some expected nominal behavior. The IoT infrastructure can be used for monitoring any events or changes in structural conditions that can compromise safety and increase risk. It is thus a recurrent and crucial issue to determine whether huge data streams, received at monitored devices, are correlated or not as it may reveal the presence of attacks. We propose a metric, called codeviation, that allows to evaluate the correlation between distributed massive streams. This metric is inspired from classical metric in statistics and probability theory, and as such enables to understand how observed quantities change together, and in which proportion. We then propose to estimate the codeviation in the data stream model. In this model, functions are estimated on a huge sequence of data items, in an online fashion, and with a very small amount of memory with respect to both the size of the input stream and the values domain from which data items are drawn. We then generalize our approach by presenting a new metric, the Sketch-\(\star \) metric, which allows us to define a distance between updatable summaries of large data streams. An important feature of the Sketch-\(\star \) metric is that, given a measure on the entire initial data streams, the Sketch-\(\star \) metric preserves the axioms of the latter measure on the sketch. We finally present results obtained during extensive experiments conducted on both synthetic traces and real data sets allowing us to validate the robustness and accuracy of our metrics.