10
- https://doi.org/10.1002/cpe.7905
Improving prediction of computational job execution times with machine learning
- Sep 12, 2023
- Concurrency and Computation: Practice and Experience
- Bartosz Balis +4 more
SummaryPredicting resource consumption and run time of computational workloads is crucial for efficient resource allocation, or cost and energy optimization. In this paper, we evaluate various machine learning techniques to predict the execution time of computational jobs. For experiments we use datasets from two application areas: scientific workflow management and data processing in the ALICE experiment at CERN. We apply a two‐stage prediction method and evaluate its performance. Other evaluated aspects include: (1) comparing performance of global (per‐workflow) versus specialized (per‐job) models; (2) impact of prediction granularity in the first stage of the two‐stage method; (3) using various feature sets, feature selection, and feature importance analysis; (4) applying symbolic regression in addition to classical regressors. Our results provide new valuable insights on using machine learning techniques to predict the runtime behavior of computational jobs.
- # Feature Importance Analysis
- # Scientific Workflow Management
- # ALICE Experiment
- # Prediction Of Execution Times
- # Execution Time Of Jobs
- # Impact Of Granularity
- # Scientific Data Processing
- # Energy Optimization
- # Behavior Of Jobs
- # Feature Selection