Real-time big data processing for anomaly detection: A Survey

In the digital era, data is one of the most important assets since it conceals valuable information. Developers of data-intensive systems have new challenges at each level of streaming, storing, and processing large quantities of data in a variety of forms and speeds. Obtaining useful information at the proper time and place is also crucial. Since the value of information is inversely proportional to time, real-time data processing and analytics are receiving more attention. Due to the importance of real-time data processing and analytics, this study focuses on real-time data processing concepts and terminology, popular technologies used in real-time data processing and analytics, popular NoSQL storage technologies used in real-time data processing, and real-time data processing application areas. The purpose of this paper is to provide researchers of real-time analysis and developers of data-intensive systems with a comparative perspective on real-time data processing by highlighting the key characteristics of real-time data processing technologies, NoSQL storage technologies, their application domains, and selected examples from previous studies.

Real-time big data processing is a crucial need of meteorology today. As a matter of fact, the weather is a determining factor in decision making in different areas such as air or sea transport, so acquiring knowledge of the environment in real time is a consequential question. A promising approach for big data treatment in real time is based on the concept of complex events processing CEP). The CEP aims to identify significant events in a cloud of simple and individually detectable events in real time. The meteorological variables, constituting simple event's spaces, offer significant flow of data to be processed using CEP technologies. This real-time data processing is done by phase of their integration with weather systems. This paper discusses the big data's meteorological aspects, and shows the architectural model for integration of weather data and its processing in real time.

In recent times, a massive amount of smart devices or objects are connected that enhances the scale of the digital world. These smart objects are referred as “things” or physical devices that have the potential to sense the real-world physical objects, collect the data, and network with others. The objects are connected through the internet, which crafts the terminology of Internet of Things (IoT). IoT has been developed and become the center of consideration due to the novelty of embedded device and a rapid enhancement in its number. This increase is resulting in the creative applications of smart environments. Smart transportation is a central stake for the quality of life of citizens in smart environment. Smart transportation involves the use of devices and sensors in the control system of vehicle; for example navigation system of cars, traffic signal management system, number recognition system and speed monitoring system. In this research article, we propose architecture for smart transportation system using Big Data analytics, in order to achieve real time processing and facilitate a friendly communication in the environment of IoT based smart transportation. The proposed architecture is a 3-phase scheme which is responsible for the organization and management of Big Data, real-time processing of Big Data and service management. The proposed architecture is a generic solution for the smart transportation planning using real time Big Data processing. The proposed scheme is realized using Spark over single node Hadoop setup with various input libraries. A huge amount of data from different authentic and reliable sources is measured to validate the proposed architecture. In addition, the effectiveness of proposed scheme also highlighted with regard to throughput.

In this era of information explosion, the dissemination of new media has emerged as a significant force influencing societal discourse and commercial decisions. With the rapid advancement of internet technology, the user base of new media platforms has expanded swiftly, leading to a gradual upheaval of traditional communication models. Against this backdrop, data analysis, as an innovative technological tool, is profoundly impacting the methods of new media dissemination with its powerful information processing and insight capabilities. Through the real-time collection and processing of big data, data analysis not only provides precise user profiles for new media but also offers robust support for content production and the optimization of dissemination paths. Particularly in areas such as enhancing user engagement and evaluating dissemination effectiveness, data analysis demonstrates its irreplaceable value. However, with the explosive growth of data, addressing issues related to data privacy, security, and algorithmic bias has become a critical research topic. This paper explores the practical applications of data analysis in new media dissemination, aiming to reveal future development trends and challenges in this field, and provide theoretical support and practical references for professionals in the new media industry.

Major advances in telecommunications and the Internet of Things have given rise to numerous smart city scenarios in which smart services are provided. What was once a dream for the future has now become reality. However, the need to provide these smart services quickly, efficiently, in an interoperable manner and in real time is a cutting-edge technological challenge. Although some software architectures offer solutions in this area, these are often limited in terms of reusability and maintenance by independent modules —involving the need for system downtime when maintaining or evolving, as well as by a lack of standards in terms of the interoperability of their interface. In this paper, we propose a fully reusable microservice architecture, standardized through the use of the Web of things paradigm, and with high efficiency in real-time data processing, supported by complex event processing techniques. To illustrate the proposal, we present a fully reusable implementation of the microservices necessary for the deployment of the architecture in the field of air quality monitoring and alerting in smart ports. The performance evaluation of this architecture shows excellent results.

The problems of design, computer simulation, manufacture of the device sample and its characteristics investigation of the acoustooptical scanner based on surface acoustic waves for 2D data processing in real time are discussed. The method is based on waveguide Bragg and radiation acoustooptic interactions with a synthesized optical aperture using a few optical beams. The total information field is divided into subareas, each of which is irradiated using an individual optical beam controlled by the corresponding waveguide acoustooptic Bragg cell and interaction cell. During the work elaboration of unit contraction, computer simulation, fabrication sample on based Y-cut lithium niobate and its characteristics investigated. This unit makes it possible to process in real time 2600times650 scanning points (information bits) representing two-dimensional arrays

Recent advances in BazookaSPECT: Real-time data processing and the development of a gamma-ray microscope

In light of the prevailing approach in which data from side-scan sonar (SSS) from Autonomous Underwater Vehicles (AUVs) are primarily processed and visualized post mission, failing to meet the requirements in terms of timeliness for on-the-fly image acquisition, this paper introduces a novel method for real-time processing and superior imaging of navigation strip data from SSS aboard AUVs. Initially, a comprehensive description of the real-time processing sequence is provided, encompassing the integration of multi-source navigation data using Kalman filtering, and high-pass filtering of attitude and heading data to exclude anomalies, as well as the use of bidirectional filtering techniques within and between pings, ensuring real-time quality control of raw data. In addition, this study adopts the semantic segmentation Unet network for automatic real-time tracking of seafloor lines, devises a real-time correction strategy for radial distortion based on historical echo data, and utilizes the alternating direction multiplier method for real-time noise reduction in strip images. With the combined application of these four pivotal techniques, we adeptly address the primary challenges in real-time navigation data processing. In conclusion, marine tests conducted in Bohai Bay substantiate the efficacy of the methodologies delineated in this research, offering a fresh paradigm for real-time processing and superior visualization of SSS navigation strip data on AUVs.

This paper presents an integrated approach to real-time processing and analysis of large-scale data streams using artificial intelligence methods, multi-agent systems, and multi-layered control architectures. Particular attention is devoted to combining technologies that are traditionally applied separately: unmanned aerial vehicle (UAV) control systems, situational centers, streaming video services, and distributed sensor networks. The proposed model accounts for the stochastic nature of the environment, the dynamic behaviour of data flows, and resource constraints, which are critical factors for real-time systems. The study develops a multi-level multi-agent architecture that includes central, regional, and edge nodes, as well as local agents such as drones, sensors, and client-side video devices. A mathematical formalization of agent interaction and reward functions is proposed to ensure a balance between service quality, latency, packet loss, and energy consumption. The use of multi-agent reinforcement learning (MARL) algorithms is introduced to support adaptive decision-making in real time, along with the development of a digital twin of the environment for predicting future system states using deep and generative models. The results demonstrate that the integration of edge computing, hierarchical decision-making, and local agent autonomy significantly enhances system resilience, provides stable performance under external disturbances, and enables effective infrastructure scaling. The findings form a scientific foundation for designing adaptive, robust, and self-learning next-generation digital ecosystems capable of intelligent processing of large data streams in real time

Tampering detection has been increasingly attracting attention in the field of digital forensics. As a popular nonlinear smoothing filter, median filtering is often used as a post-processing operation after image forgeries such as copy-paste forgery (including copy-move and image splicing), which is of particular interest to researchers. To implement the blind detection of median filtering, this paper proposes a novel approach based on a frequency-domain feature coined the annular accumulated points (AAP). Experimental results obtained on widely used databases, which consists of various real-world photos, show that the proposed method achieves outstanding performance in distinguishing median-filtered images from original images or images that have undergone other types of manipulations, especially in the scenarios of low resolution and JPEG compression with a low quality factor. Moreover, our approach remains reliable even when the feature dimension decreases to 5, which is significant to save the computing time required for classification, demonstrating its great advantage to be applied in real-time processing of big multimedia data.

Since 2020, the epidemic of Corona Virus Disease 2019 has swept the world, which has had a great impact on people's life. Especially for college campus life. This paper presents an emergency management system for college students. We use Java Web, big data processing and other technologies to develop, implement and test the system. The system integrates daily management and emergency management, which can meet the basic needs of teachers and students on campus. We hope to provide convenience for students and teachers in learning and management.

In a medical emergency situation, real-time patient data sharing may improve the survivability of a patient. In this paper, we explore how Digital Twin (DT) technology can be used for real-time data storage and processing in emergency healthcare. We investigated various enabling technologies, including cloud platforms, data transmission formats, and storage file formats, to develop a feasible DT storage solution for emergency healthcare. Through our analysis, we found Amazon AWS to be the most suitable cloud platform due to its sophisticated real-time data processing and analytical tools. Additionally, we determine that the MQTT protocol is suitable for real-time medical data transmission, and FHIR is the most appropriate medical file storage format for emergency healthcare situations. We propose a cloud-based DT storage solution, in which real-time medical data are transmitted to AWS IoT Core, processed by Kinesis Data Analytics, and stored securely in AWS HealthLake. Despite the feasibility of the proposed solution, challenges such as insufficient access control, lack of encryption, and vendor conformity must be addressed for successful practical implementation. Future work may involve Hyperledger Fabric technology and HTTPS protocol to enhance security, while the maturation of DT technology is expected to resolve vendor conformity issues. By addressing these challenges, our proposed DT storage solution has the potential to improve data accessibility and decision-making in emergency healthcare settings.

With the development of various technologies such as high-speed Internet and SNS dissemination, there have been many fields that require processing of big data generated in real time. Accordingly, real-time streaming data processing technology has been developed, and representative platforms include Apache Storm, Apache Spark, and Hadoop. These processing technologies provide scalability to configure distributed systems using multiple servers because they vary in performance, such as throughput and processing speed, depending on the server environment, but the more the number of servers, the more difficult it is to manage. To solve this problem, a problem can be solved by using a docker, a kind of virtualization system that provides ease of expansion. However, there is a place to maintain a native environment without using Docker due to the problem that performance may be reduced, which is a disadvantage of all virtualization systems. In this paper, we build Apache Storm and Apache Spark, which are real-time data processing systems in Docker and Native environments and conduct performance measurements through experiments processing JSON-format data to verify how much performance decreases in Docker environments.

In present day technology there is an immense need of developing suitable data communication interfaces for real time embedded systems.Field Programmable Gate Array (FPGA) offers various resources, which can be programmed for building up an efficient embedded system.In recent years the SOC (System on Chip) design eg, in media processing [1] is becoming more and more important in real time embedded applications as SOCs require low power, low area but are still capable of implementing various complex functionalities.In order to achieve SOC architecture, which can run a real time application, we need to develop high-speed data interfaces of the system with the external world through its various I/O ports.The DMA controller, which sends the data from I/O to memory and vice-versa without intervention of the processor, thus plays a vital role in these systems in order to achieve faster I/O data transfer.This paper proposes a technique to implement a DMA controller core on Spartan 3A FPGA hardware, which serves as an essential component for developing a real time data acquisition and processing system.

Processing a data stream in real time is a crucial issue for several applications, however processing a large amount of data from different sources, such as sensor networks, web traffic, social media, video streams and other sources, represents a huge challenge. The main problem is that the big data system is based on Hadoop technology, especially MapReduce for processing. This latter is a high scalability and fault tolerant framework. It also processes a large amount of data in batches and provides perception blast insight of older data, but it can only process a limited set of data. MapReduce is not appropriate for real time stream processing, and is very important to process data the moment they arrive at a fast response and a good decision making. Ergo the need for a new architecture that allows real-time data processing with high speed along with low latency. The major aim of the paper at hand is to give a clear survey of the different open sources technologies that exist for real-time data stream processing including their system architectures. We shall also provide a brand new architecture which is mainly based on previous comparisons of real-time processing powered with machine learning and storm technology.