A Stream Data Processing Framework for Location-Based Service Using NoSQL Technology

Recently, many researches on positioning technology using LBS (Location-Based Services) have been conducted with the development of wearable devices. In addition, the data used in these devices is helping to perform LBS using Big data technology. And the existing method of finding a specific location is not suitable for collecting and processing all the data [1] [2] [3] [4]. Therefore, in order to process all streaming data in real time, it is necessary to use Big data processing technology. In this paper, we use NoSQL technology to solve this problem. We propose a framework for improving the performance of LBS using NoSQL.

Streaming data is widely generated in our lives. This has promoted a lot of research on streaming data mining, such as streaming data clustering and filtering. In our work, we present a problem about data stream processing, namely, streaming data sorting. There are some important characteristics of streaming data. Firstly, streaming data comes in the form of streams. It is usually assumed that streaming data is infinite, so it cannot be stored completely in memory. Secondly, we must process the streaming data in real time, otherwise we may lose the opportunity to deal with it forever. Based on these characteristics, we propose a dynamic algorithm that can make full use of memory and minimize error to solve the problem of streaming data sorting, which is further combined with the BFR algorithm to sort a particular type of streaming data. Some experiments are conducted to confirm the effectiveness of the proposed algorithms.

The increase in personal wearable device use enables the continuous measurement of users’ health and anthropometric features. Many digitally collected health and behavioral data is already captured. These streams can include physical activity, calorie burn, heart rate, and sleep patterns. Recently, remote health and wellness devices are equipped with options to synchronize the captured data to different real-time aggregation platforms. As of this year, a number of software platforms are available on the market that enable real-time data aggregation of the collected personal wellness and health monitoring devices. All of them provide application programming interfaces (APIs) to manipulate the synchronized data streams. To further foster the aggregation and distillation of personal wellness and health monitoring devices, additional filtering and processing methods are needed to sift through the combined data streams and provide necessary feedback without information overload. Last but not least, these generated systems need to be user friendly with a design that will make the communities adherent and loyal. Anecdotally, this implies that the designed system has taken both inputs and critical insights from the future community of interest to design a system that ultimately benefits those who use it the most. To empirically test that hypothesis, and provide a design pattern of what a successful software ecosystem is for patient engagement, the company’s want to understand the community's software usage habits. A few sample questions of interest are assessed and the percentage distribution on each is calculated. Outcomes and aggregate data will be shared with the scientific community within 30 days of receiving the request. It is possible that aggregated data will also be made publicly available and freely shared.The growing use of personal wearable devices has enabled the continuous tracking of users’ health and anthropometric data, including metrics such as physical activity, calorie expenditure, heart rate, and sleep patterns. Modern wellness technologies now often support real-time synchronization of this data to various aggregation platforms, with many offering APIs for flexible data manipulation. As of this year, numerous software platforms exist that facilitate this real-time data collection and integration. However, to effectively manage and interpret these extensive data streams, advanced filtering and processing methods are necessary to avoid overwhelming users with excessive information. Crucially, these systems must also be user-friendly and tailored to the needs and behaviors of their intended communities. This means incorporating direct feedback and insights from target users during the design phase to ensure the system fosters long-term engagement and loyalty. To validate these approaches and identify effective design patterns for software ecosystems that support patient engagement, companies are conducting empirical studies on community software usage habits. Sample questions are posed, and the resulting data is analyzed and shared—potentially made publicly available—to benefit the broader scientific and developer communities.

A Big Data Stream Computing (BDSC) Platform handles real-time data from various applications such as risk management, marketing management and business intelligence. Now a days Internet of Things (IoT) deployment is increasing massively in all the areas. These IoTs engender real-time data for analysis. Existing BDSC is inefficient to handle Real-data stream from IoTs because the data stream from IoTs is unstructured and has inconstant velocity. So, it is challenging to handle such real-time data stream. This work proposes a framework that handles real-time data stream through device control techniques to improve the performance. The frame work includes three layers. First layer deals with Big Data platforms that handles real data streams based on area of importance. Second layer is performance layer which deals with performance issues such as low response time, and energy efficiency. The third layer is meant for Applying developed method on existing BDSC platform. The experimental results have been shown a performance improvement 20%-30% for real time data stream from IoT application.

Taking a byte out of big data

This paper presents an integrated approach to real-time processing and analysis of large-scale data streams using artificial intelligence methods, multi-agent systems, and multi-layered control architectures. Particular attention is devoted to combining technologies that are traditionally applied separately: unmanned aerial vehicle (UAV) control systems, situational centers, streaming video services, and distributed sensor networks. The proposed model accounts for the stochastic nature of the environment, the dynamic behaviour of data flows, and resource constraints, which are critical factors for real-time systems. The study develops a multi-level multi-agent architecture that includes central, regional, and edge nodes, as well as local agents such as drones, sensors, and client-side video devices. A mathematical formalization of agent interaction and reward functions is proposed to ensure a balance between service quality, latency, packet loss, and energy consumption. The use of multi-agent reinforcement learning (MARL) algorithms is introduced to support adaptive decision-making in real time, along with the development of a digital twin of the environment for predicting future system states using deep and generative models. The results demonstrate that the integration of edge computing, hierarchical decision-making, and local agent autonomy significantly enhances system resilience, provides stable performance under external disturbances, and enables effective infrastructure scaling. The findings form a scientific foundation for designing adaptive, robust, and self-learning next-generation digital ecosystems capable of intelligent processing of large data streams in real time

Open hole logging is challenging in Tengiz reservoir where lost circulation events and abundance of natural fractures lead to wireline tool conveyance issues. In order to address conveyance issues, several alternative methods were reviewed and logging while drilling in washdown mode was selected as the most appropriate technique for given conditions. Alternative solutions such as hole finders, pipe conveyed logging, through bit logging were rejected due to inability to provide required results within the specified time constraints or due to well control risks. Conventional logging while drilling method was declared not feasible due to excessive shocks and vibration under managed pressure drilling technique that is currently being used in Tengiz for drilling through fractured reservoir with significant fluid losses. Therefore, logging while drilling equipment, together with the newly developed procedures on preparing hole conditions, pumping rates and logging speeds were utilized to log reservoir section in washdown mode. Prior to commencing of logging in a fractured zone with significant losses, extensive trial was conducted on two stable wells to develop and refine logging procedures. In order to assure quality log data acquisition, full suite of wireline open hole logs was also acquired from these two wells. Further comparison and calibration of logging while drilling data vs. wireline log data allowed to achieve required accuracy and repeatability. Procedures developed on stable wells then applied to wells with significant losses considering specific pump rates to achieve constant stream of log data to surface. It is critical for Tengiz operations to establish and maintain continuous log telemetry and stream data in real time for efficient rig time utilization. The challenge was to sustain positive pipe pressure in a well with total losses to enable mud pulse telemetry. This issue was successfully resolved by increasing pump rates and use of flow restrictor sub as a part of logging tool string. The novel approach of the described logging method enables log data acquisition on a well that experienced total losses and had significant obstructions for conveyance of wireline tools. Method allows to efficiently collect critical log data required for well completion and reservoir characterization.

The article provides a comprehensive overview of incremental learning in the context of big data processing. The basic concepts, modern approaches, and key aspects of incremental learning are considered. The advantages of this approach for processing large amounts of data are analyzed, including the efficient use of computing resources, the ability to process streaming data in real time, and adaptability to changes in data. The main limitations and challenges, such as the problem of "catastrophic forgetting", the difficulty of balancing new and old knowledge, dependence on the order of data arrival, and potential loss of accuracy, are investigated. An analysis of specific problems is presented, including the handling of conceptual drift, unbalanced classes, and missing features. Applications of incremental learning in various fields, including data analytics, robotics, autonomous driving, and activity recognition, are discussed. We suggest directions for future research to address the identified problems and improve the effectiveness of incremental learning in the context of big data.

Cloud computing and big data: Technologies and applications

Chapter 32 - StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time

The use of large and diverse data in real time (called Big Data) affects many business processes and models. The tourism industry, characterized by manifold sub-sectors and players, provides a variety of starting points for Big Data usage. Examples are the optimization of transport offers using transaction data or a comprehensive analysis of destination trends based on social media posts. Big Data is a trending topic, however, the general discourse centres around potential ideas but fewer practical solutions. Based on a systematic literature analysis of initially 148 peer-reviewed journal articles, this article evaluates the current state of Big Data research within tourism. For this purpose, research articles centering around tourism-related Big Data were investigated according to the actual state of implementation of an IT solution, whether they truly grasp or represent Big Data in technological terms, and which added value they create for the tourism industry and research community. One key finding is that traditional data analysis is often wrongfully subsumed under the Big Data label. Further, the scientific literature predominantly discusses ideas or theoretical considerations, fewer tangible Big Data implementations, and fails to address and/or meet all requirements to be classified as Big Data. Only a minority of the presented solutions processes data in real time, whereas many rely on only one data source or structured data. Furthermore, most articles revolve around post-trip data analyses and are set to a destination context. In contrast, other tourism sectors as well as data interpretation and usage in pre-trip and on-trip phases are less represented. Lastly, this literature analysis provides an overview of true Big Data solutions already in operation and enables researchers to validly classify their own research activities in order to plan initiatives more specifically.

This article explores the impact of the introduction of Big Data into the internal processes and decision making of football teams. Among the main conclusions of this exploratory conceptual study, we could mention: the boom of data analysis in sports; the development of new technological devices that give rise to new and meaningful sources of information, being able to process, interpret, present, and share these data in real time; and the more and more constant implementation of scientific methods in the decision-making process, with the former being directly linked to the achievement of satisfactory results.

The object of the study is the adaptive machine learning systems that are able to process large amounts of rapidly changing streaming data in real time. The problem of maintaining prediction accuracy and computational efficiency in the presence of concept drift is treated. Concept drift refers to the overweighting of static models when stationary models are tried, and the nature of the underlying distributions changes. The adaptive architecture includes revision divergence-oriented concept drift detection, incremental model updating via hyper-dimensional statistical clustering of segments. Results from experiments using simulated and real-world datasets demonstrate that the adaptive architecture maintains predictive accuracy above 0.83 across abrupt, gradual, recurrent, and continuous drift scenarios. Compared with non-adaptive models, adaptation latency is reduced by approximately 2.6×, while unnecessary retraining operations are decreased by up to 40%. These results are possible due to the fact that proposed framework is able to retrain solutions if, and only if, distributional changes are determined to be statistically significant and meaningful to the model. This leads to the avoidance of processors being given redundant computations and providing a steady-state model during non-drift conditions. A principal contribution is that feature engineering is accomplished in a drift-aware manner, thresholding is made adaptive to the distributions indicated, and update mechanisms are employed which efficiently utilize resources in a unified high-performance streaming pipeline. The architecture performs well under abrupt, gradual, recurrent, and continuous drift and effective for real-time applications which include smart-city analytics, cyber security monitoring, roadways system works, and IoT for industrial systems

The main objective of online anomaly detection is to identify abnormal/unusual behavior such as network intrusions, malware infections, over utilized system resources due to design defects etc from real time data stream. Terrabytes of performance data generated in cloud data centers is a well accepted example of such data stream in real time. In this paper, we propose an online anomaly detection framework using non-parametric statistical technique in cloud data center. In order to determine the accuracy of the proposed work, we experiments it to data collected from RUBis cloud testbed and Yahoo Cloud Serving Benchmark (YCSB). Our experimental results shows the greater accuracy in terms of True Positive Rate (TPR), False Positive Rate (FPR), True Negative Rate (TNR) and False Negative Rate (FNR).

Anomaly detection plays a critical role in large-scale industrial systems, where real-time streaming data is generated at high volumes. This research journal presents an in-depth study on developing an efficient anomaly detection algorithm specifically designed for such industrial systems. The goal is to identify abnormal patterns and deviations from normal behavior, enabling proactive maintenance, improved operational efficiency, and reduced downtime. The proposed algorithm leverages machine learning and stream processing techniques to handle the challenges associated with real- time streaming data analysis. The research covers algorithm design, implementation, evaluation, and performance analysis using large-scale datasets from industrial domains. Keywords: anomaly detection, real-time streaming data, large-scale industrial systems, machine learning, stream processing