• Home
  • Search
  • Dynamic Multi-objective Resource Optimization in Big Data Clusters
  • https://doi.org/10.37082/ijirmps.v9.i4.232618Copy DOI Icon

Dynamic Multi-objective Resource Optimization in Big Data Clusters

  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Big data systems handle vast volumes of structured and unstructured information using distributed computing frameworks like Apache Hadoop, Spark, and Flink. These platforms rely heavily on computational resources, and CPU utilization becomes a critical factor in determining the performance and efficiency of data processing operations. Effective CPU usage directly impacts throughput, latency, and the ability to meet service-level objectives. In large-scale clusters, workloads vary from CPU-intensive tasks such as real-time analytics, data mining, and machine learning to more I/O-driven operations like data ingestion or storage management. Poor CPU distribution can lead to overloaded nodes while others remain underutilized, ultimately reducing system efficiency. Traditional scheduling mechanisms in big data platforms are often rule-based and may not respond well to dynamic or unpredictable workloads, which contributes to suboptimal CPU usage. When CPU utilization remains consistently high, it can trigger job slowdowns, task failures, and increased power consumption, negatively affecting overall performance. Conversely, low CPU utilization indicates underused resources, leading to waste in computational capacity and energy. Additionally, container technologies and control groups (cgroups) support fine-tuned CPU allocation and isolation, ensuring fair usage across concurrent users or tasks in multi-tenant environments. In more advanced implementations, machine learning algorithms are used to forecast CPU needs and schedule jobs more efficiently by learning from historical patterns and usage behavior. Reinforcement learning approaches have also shown potential in achieving balanced CPU usage by adapting policies based on feedback and environment changes. These intelligent mechanisms allow big data systems to optimize CPU allocation continuously, adapting to fluctuating demand without manual intervention. Efficient CPU management contributes significantly to faster job execution, reduced operational costs, and higher cluster reliability. As data volumes and complexity continue to grow, maintaining optimal CPU utilization remains a priority for organizations aiming to derive timely insights while maximizing infrastructure value. Basic CPU utilization approaches are exhibiting performance limitations. This paper addresses performance limitations related to CPU utilization.

Similar Papers
  • Book Chapter
  • Citations3

Experiences and Lessons in Practice Using TPCx-BB Benchmarks

  • Dec 30, 2017
  • Kebing Wang +3
  • Conference Article
  • Citations3

PRINCIPIA: Opportunistic CPU and CPU-shares Allocation for Containerized Virtualization in Mobile Edge Computing

  • May 08, 2023
  • Andres F Ocampo +4
  • Research Article
  • Citations101

Big Sensor Data Applications in Urban Environments

  • Mar 02, 2016
  • Big Data Research
  • Li-Minn Ang +1
  • Research Article
  • Citations16

Modelling and analysis of big data platform group adoption behaviour based on social network analysis

  • Mar 29, 2021
  • Technology in Society
  • Zhimei Lei +2
  • Book Chapter
  • Citations6

Framework for Building a Big Data Platform for Publishing Industry

  • Jan 01, 2015
  • Aravind Kumaresan
  • Research Article
  • Citations3

Power-efficient and high-performance block I/O framework for mobile virtualization systems

  • Jul 08, 2016
  • The Journal of Supercomputing
  • Kihong Lee +3
  • Conference Article
  • Citations3

Power-efficient and high-performance block I/O framework for mobile virtualization systems

  • Jan 08, 2015
  • Kihong Lee +2
  • Conference Article
  • Citations3

Visualizing Multi-process CPU Utilization using CUSP

  • Mar 05, 2022
  • Mark W Maimone
  • Conference Article
  • Citations3

Research on the Big Data Platform and Its Key Technologies for the Railway Locomotive System

  • May 28, 2020
  • Li Xin +2
  • Research Article
  • Citations32

Mobile Internet big data platform in China Unicom

  • Feb 01, 2014
  • Tsinghua Science and Technology
  • Wenliang Huang +5
  • Conference Article
  • Citations35

A performance study of big data analytics platforms

  • Dec 01, 2017
  • Pouria Pirzadeh +2
  • Conference Article

Big data clustering considering chaotic correlation dimension feature extraction

  • Jul 01, 2020
  • Shanshan Liu
  • Conference Article

Personalized Film and Television Recommendation System Based on Big Data Platform

  • Sep 01, 2021
  • Qian Zhu +1
  • Book Chapter

Data on Computer-Aided Health in Inner Mongolia Autonomous Region Based on Big Data

  • Jan 01, 2021
  • Ningping Yuan
  • Research Article
  • Citations1

Design Approach to Big data Systems in Developing and Maintaining the Information Security Systems

  • May 01, 2016
  • IOSR Journal of Computer Engineering
  • Dr.Mani Sarma Vittapu
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.