Design and Implementation of a High-Performance Stream-Oriented Big Data Processing System

In the big data processing systems, the amount of data is increasing. At the same time, the real-time requirement of data processing and analysis is higher and higher. Therefore, it is required that the big data processing and analysis systems have better performance. Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. However, job scheduling is a difficult NP-hard problem. There are many factors that need to be considered for job scheduling. For example, jobs have dependencies among stages, therefore we should not allocate resources to tasks that are not ready. Sometimes, there are constraints between jobs. These are a challenge to the scheduling performance of big data processing and analysis systems. In this paper, we try to solve the problem by translating it into Boolean Satisfiability Problem (SAT) which is an exact method. SAT-based scheduling algorithm is not a new approach, but in the past it mainly used to solve the static scheduling problems. For dynamic scheduling system, it requires all problems to be solved within a limited time, which is a challenge for SAT encoding. In this paper, we refer to the previous SAT solution to the Job Shop Scheduling Problem, and adjust the algorithm to meet the requirements of the big data processing system. At the same time, we optimized the coding approach and reduced the number of clauses. Thus, the efficiency of the problem solved is improved to meet the performance requirements. The experimental results show that the number of clauses is reduced by more than 30%, and the processing time of the SAT solver to get the solution can be reduced by more than 50%. To demonstrate its effectiveness, we have also implemented our new job scheduler in Apache Hadoop YARN, and validated its effectiveness

This study is concerned with the design and implementation of a real-time processing system of full resolution quick-look image of HJ-1 environmental satellite C SAR based on high-performance clusters. The system processes the first quick-look SAR image on December 9, 2012. The results show that the design and implementation of the quick-look processing system satisfies the real-time SAR image processing performance requirements at full resolution. Moreover, this system is the first real-time business system of full-resolution quick-look spaceborne SAR images in China.

With the rapid development of Internet of Things (IoT) technologies, the increasing volume and diversity of sources of geospatial big data have created challenges in storing, managing, and processing data. In addition to the general characteristics of big data, the unique properties of spatial data make the handling of geospatial big data even more complicated. To facilitate users implementing geospatial big data applications in a MapReduce framework, several big data processing systems have extended the original Hadoop to support spatial properties. Most of those platforms, however, have included spatial functionalities by embedding them as a form of plug-in. Although offering a convenient way to add new features to an existing system, the plug-in has several limitations. In particular, while executing spatial and nonspatial operations by alternating between the existing system and the plug-in, additional read and write overheads have to be added to the workflow, significantly reducing performance efficiency. To address this issue, we have developed Marmot, a high-performance, geospatial big data processing system based on MapReduce. Marmot extends Hadoop at a low level to support seamless integration between spatial and nonspatial operations of a solid framework, allowing improved performance of geoprocessing workflow. This paper explains the overall architecture and data model of Marmot as well as the main algorithm for automatic construction of MapReduce jobs from a given spatial analysis task. To illustrate how Marmot transforms a sequence of operators for spatial analysis to map and reduce functions in a way to achieve better performance, this paper presents an example of spatial analysis retrieving the number of subway stations per city in Korea. This paper also experimentally demonstrates that Marmot generally outperforms SpatialHadoop, one of the top plug-in based spatial big data frameworks, particularly in dealing with complex and time-intensive queries involving spatial index.

In order to meet the tomato color sorting system with the requirements of multi-channel data acquisition and real-time processing, [ ]firstly, a multi-channel color digital acquisition and processing technique program based on FPGA were proposed . Then, a multi-channel data acquisition and real-time parallel processing system was built. The RGB color sensors were slected as detection components and the FPGA chip EP1C3T144C8 was slected as a core of data acquisition and processing. Meanwhile, ports of signal acquisition and the solenoid valve control were adopted a two-way analog switch HC4051 for saving I/O ports of FPGA. Thirdly, under the modular design and module reuse principles, we developed all modules with VHDL Language in QuartusII development environment. Finally, RTL and time sequence simulation for the system were done. The results showed that the connections between the modules accord with control logics. [²»´o¶O accorded ¹ýE¥E½] The performance of system time sequencev are good, which can meet with the requirements of real-time processing of tomato color sorting. On these bases, we also conducted a tomato sorting test verification. The results showed that the average accuracy of sorting is 98.70% and coefficient of variation is 0.01. This results can meet requirements of production.

Time is the best important resource in real-time video image processing system. Only few structure or algorithm can be used in real-time video image processing system because of time limitation. Establish a high speed data-access is a way to this problem. It transfer the video signal between output ports and kernel processor as less time delay. The mode, look-up-table, other analysis, adjustment or calculation rule will be applied first by medium or low data-access. We can choose different data-access according to data relation level. Data which has closer correlation will be treated through low-speed-access, which has loser correlation will be treated through medium-speed-access. Theory analysis and experiment showed that it is a high effective structure for real-time video image processing. The feature of video image and real-time processing system is presented in this paper. The basic theory of data-access is introduced also. The status and relation of high, medium, low data-access is discussed and the rules of data distribution in different access is analyzed. Some examples with datas-accesses are given in the paper (TV tracking and measuring, laser signal alarming system).

As people's water consumption become large and large in china, people begin to expect high requirements of water quality. However, some waterworks' monitoring system cannot automatically detect and process water quality parameters and system operation status. Therefore, staffs have to manual meter reading. The paper first discusses the design of the whole system, including the server side and the client side. It then goes on to discuss, in the server side, how to realize the communication between server and OPC client by the OPC communication technology. Finally, the paper focuses on the client side that the user interface is designed to export of automatic report and to draw the graph. Running results show that the software can detect and process parameters in real-time, and the interface is friendly, with good reliability, and can meet the demands of customers.

With the increasing availability and scale of data from Web 2.0, the ability to efficiently and timely analyze huge amounts of data is important for industry success. A number of real-time stream processing platforms have been developed, such as Storm, S4, and Flume. A fundamental problem of these large scale decentralized stream processing systems is how to deploy the workload to each node so as to fully utilize the available resources and optimize the overall system performance. In this paper, we present StroMAX, a graph-partitioning based approach of workload scheduling for real-time stream processing systems. StroMAX uses two advanced generic schedulers to improve the performance of stream processing systems by reducing the inter-node communication cost while keeping the workload of nodes below a certain computational load threshold. The first scheduler analyzes the workload structure when a job is committed and uses the graph-partitioning result to determine the deployment of tasks. The second scheduler analyzes the statistical information of physical nodes, and dynamically reassigns the tasks during runtime to improve the overall performance. Besides, StroMAXcan be deployed to many other state-of-the-art real-time stream processing systems easily. We implemented StroMAX on Storm, a representative real-time stream processing system. Extensive experiments conducted with real-world workloads and datasets demonstrate the superiority of our approaches against the existing solutions.

A significant challenge in modern data-driven organizations is the development of scalable architectures capable of processing real-time data streams efficiently. This article presents a comprehensive framework for building and optimizing real-time data processing systems, focusing on integrating contemporary technologies such as Apache Kafka, Apache Flink, and Spark Streaming. The article systematically analyzes microservices architectures and event-driven design patterns, and it examines various approaches to data ingestion, state management, and dynamic load balancing in distributed environments. The methodology combines an extensive literature review

The exponential growth of real-time data from financial transactions, IoT devices, social media, and industrial applications has intensified the need for high-speed, intelligent, and fault-tolerant streaming architectures. Traditional batch-processing and micro-batch systems, such as Apache Kafka with Spark Streaming, struggle with high latency, static resource allocation, and reactive fault recovery mechanisms, making them inadequate for modern data-driven enterprises. To address these challenges, we propose the Real-Time Adaptive Streaming Model (RTASM)—an AI-driven, ultra-low-latency streaming framework that integrates Apache Kafka, Hadoop, and AI-powered dynamic optimization. RTASM introduces several groundbreaking innovations, including AI-Optimized Workload Balancing, Predictive Caching & Query Optimization, Self-Healing Disaster Recovery, and 5G-Ready Edge Computing. By leveraging reinforcement learning algorithms, anomaly detection, and predictive load balancing, RTASM reduces event-to-insight latency to sub-5ms, enhances data quality to 99.99% accuracy, and minimizes failover downtime by 40%. These advancements enable real-time analytics for mission-critical applications such as financial fraud detection, autonomous vehicle telemetry, cybersecurity threat intelligence, and smart manufacturing. Comparative analysis with traditional Kafka + Spark Streaming architectures highlights RTASM’s superior performance, lower latency, higher data quality, and proactive disaster recovery mechanisms. Our evaluation demonstrates that RTASM outperforms traditional models by optimizing parallel processing, dynamically adjusting computational resources, and preventing failures before they occur. Future research directions include quantum-assisted processing, neural network-driven query optimizations, and integration with 6G-enabled streaming architectures. By transforming real-time data analytics into an adaptive, intelligent, and self-healing system, RTASM sets a new benchmark for zero-latency, high-performance data processing, ensuring that enterprises can make instant, data-driven decisions with unprecedented efficiency and reliability

Recent advances in mobile technologies have enabled a plethora of new applications. The hardware capabilities of mobile devices, however, are still insufficient for real-time stream data processing (e.g., real-time video stream). In order to process real-time streaming data, most existing applications offload the data and computation to a remote cloud service, such as Apache Storm or Apache Spark Streaming. Offloading streaming data, however, has high costs for users, e.g., significant service fees and battery consumption. To address these challenges, we design, implement and evaluate Mobile Storm, the first stream processing platform for mobile clouds, leveraging clusters of local mobile devices to process real-time stream data. In Mobile Storm, we model the workflow of a real-time stream processing job and decompose it into several tasks so that the job can be executed concurrently and in a distributed manner on multiple mobile devices. Mobile Storm was implemented on Android phones and evaluated extensively through a real-time HD video processing application. The result shows that Mobile Storm effectively processes HD Video Stream in a mobile cloud, which would be impossible on a single mobile device.

Research and discuss the target detection algorithms and flows in remote sensing image. Different from other simulation detection algorithms, we added span-tile determination steps in an actual implementation. Based on the analysis of remote sensing image real-time processing system features, design and implement a high-performance real-time target detection signal processing system. The system consists of an amount of dedicated floating point DSPs formed a processing element (PE) and high-speed serial switching architecture which established the communication links between PEs. As a novel contribution of this paper, a pipelined parallel method and the implementation of a RapidIO based remote sensing image real-time processing system depicted in detail. Experiments carried out on actual satellite remote sensing image data set confirmed the processing and detection performance of the system.

A real-time operating system (RTOS) provides a platform for the design and implementation of a wide range of applications in real-time systems, embedded systems, and mission-critical systems. This paper presents a formal design model for a general RTOS known as RTOS+ that enables a specific target RTOS to be rigorously and efficiently derived in real-world applications. The methodology of a denotational mathematics, Real-Time Process Algebra (RTPA), is described for formally modeling and refining architectures, static behaviors, and dynamic behaviors of RTOS+. The conceptual model of the RTOS+ system is introduced as the initial requirements for the system. The architectural model of RTOS+ is created using RTPA architectural modeling methodologies and refined by a set of Unified Data Models (UDMs). The static behaviors of RTOS+ are specified and refined by a set of Unified Process Models (UPMs). The dynamic behaviors of the RTOS+ system are specified and refined by the real-time process scheduler and system dispatcher. This work is presented in two papers; the conceptual and architectural models of RTOS+ is described in this paper, while the static and dynamic behavioral models of RTOS+ will be elaborated in a forthcoming paper.

A new design method for the real-time image processing system based on field programmable gate array (FPGA) and digital signal processing (DSP) structure is presented in the paper. In the practical system, FPGA is used to be logic unit for sensor interface protocol and complete image preprocessing and display of PAL video; DSP is applied as the main processing unit which is extendable for better performance; Modules are connected by user-configurable I/O interface, external memory interface (EMIF) and host-post interface (HPI), which configure the system to be a loosely coupled and cascaded multi-DSP system. Finally, the design approach shows good performance in the real-time image processing system with the capability of real-time processing, extendibility and multi-interface, which is significant for practical application.

To clearly observe the upper urinary tract with a thin flexible endoscope, we developed a novel real-time image processing system. The dark cladding portion in an image guide was electrically painted out by the light which was emitted from each core pixels to obtain bright image. Thirty patients were examined using a thin flexible endoscope with this system at retrograde pyelography or puncture of a renal or prostatic cyst. The endoscope which was 0.75mm in diameter had 3,000 pixels of image fiber. The objectives were clearly seen without the dilation of the ureteral orifice. Finer and more colorful image could be obtained with this system. However, visualization was not clear in 4 patients because of poor manipulation capability at the endoscope tip. There were no complications, such as perforation of the ureter. We concluded that a thin endoscope with the real-time processing system would be useful for the diagnosis or treatment of the upper urinary tract or a cystic disease.

High-performance analytical data processing systems often run on servers with large amounts of main memory. A common operation in such environments is combining data from two or more sources using some algorithm. The focus of this paper is on studying hash-based and sort-based equi-join algorithms when the data sets being joined fully reside in main memory. We only consider a single node setting, which is an important building block for larger high-performance distributed data processing systems. A critical contribution of this work is in pointing out that in addition to query response time, one must also consider the memory footprint of each join algorithm, as it impacts the number of concurrent queries that can be serviced. Memory footprint becomes an important deployment consideration when running analytical data processing services on hardware that is shared by other concurrent services. We also consider the impact of particular physical properties of the input and the output of each join algorithm. This information is essential for optimizing complex query pipelines with multiple joins. Our key contribution is in characterizing the properties of hash-based and sort-based equi-join algorithms, thereby allowing system implementers and query optimizers to make a more informed choice about which join algorithm to use.