• Home
  • Search
  • Job failure prediction in Hadoop based on log file analysis
  • Cite Icon7
  • https://doi.org/10.1080/1206212x.2020.1732081Copy DOI Icon

Job failure prediction in Hadoop based on log file analysis

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Hadoop is a popular framework based on MapReduce programming model to allow for distributed processing of large datasets across clusters with various number of computer nodes. Just like any dynamic computational environment, Hadoop has some problems and one of which is unsuccessful execution of MapReduce jobs. Job failures can cause significant resource wasting, performance deterioration, and user dissatisfaction. Therefore, a proactive and predictive management approach could be very useful in Hadoop systems. In this paper, we try to predict the futurity of MapReduce jobs in OpenCloud Hadoop cluster by using its log files. OpenCloud is a research cluster managed by CMU’s Parallel Data Lab which uses Hadoop to process big data. We first tried to study the log files and analyze the relationship between the jobs, resources, and workload characteristics and the failures in order to discover the effective features for the prediction process. After recognizing the job failure patterns, some popular machine learning algorithms are deployed to predict the success/failure status of the jobs before they start to execute. Eventually, we compared the learning methods and showed that the C5.0 algorithm had the best results with an accuracy of 91.37%, a recall of 74.43%, and a precision of 80.31%.

Similar Papers
  • Research Article
  • Citations98

General test result checking with log file analysis

  • Jul 01, 2003
  • IEEE Transactions on Software Engineering
  • J.H Andrews +1
  • Research Article
  • Citations26

Studying E-Journal User Behavior Using Log Files: The Experience of SuperJournal

  • Aug 01, 2000
  • Library and Information Science Research
  • Liangzhi Yu +1
  • Conference Article
  • Citations11

Analysing Access Control Specifications

  • May 01, 2009
  • Christian W Probst +1
  • Research Article
  • Citations9

Analysis of Hadoop log file in an environment for dynamic detection of threats using machine learning

  • Oct 26, 2022
  • Measurement: Sensors
  • K Bapayya Naidu +7
  • Research Article
  • Citations8

Single isocenter SRS using CAVMAT offers improved robustness to commissioning and treatment delivery uncertainty compared to VMAT.

  • Jun 24, 2021
  • Journal of Applied Clinical Medical Physics
  • Edward T Cullom +6
  • Conference Article
  • Citations6

An Improved Scheme of Single Sign-On Protocol Based on Dynamic Double Password

  • Jul 01, 2009
  • Yang Jian
  • Research Article
  • Citations15

An Efficient Log File Analysis Algorithm Using Binary-based Data Structure

  • May 01, 2014
  • Procedia - Social and Behavioral Sciences
  • Sallam Osman Fageeri +1
  • Research Article
  • Citations45

Assessing the quality of proton PBS treatment delivery using machine log files: comprehensive analysis of clinical treatments delivered at PSI Gantry 2

  • Jan 15, 2016
  • Physics in Medicine & Biology
  • D Scandurra +6
  • Research Article
  • Citations5

RETRACTED: Federate learning on Web browsing data with statically and machine learning technique

  • Aug 22, 2022
  • International Journal of Pervasive Computing and Communications
  • Ratnmala Nivrutti Bhimanpallewar +5
  • Research Article
  • Citations2

Evaluating index Morganagus: Indexing and providing access to library‐related serials using harvest

  • Jan 01, 1997
  • New Review of Information Networking
  • Eric Lease Morgan
  • PDF
  • Research Article
  • Citations81

Towards effective and efficient patient-specific quality assurance for spot scanning proton therapy.

  • Apr 10, 2015
  • Cancers
  • X Zhu +10
  • Research Article
  • Citations3

Hardware Implementation of Associative Memories Based on Multiple-Valued Sparse Clustered Networks

  • Mar 01, 2016
  • IEEE Journal on Emerging and Selected Topics in Circuits and Systems
  • Naoya Onizawa +3
  • Research Article
  • Citations23

Determining 1D fast-ion velocity distribution functions from ion cyclotron emission data using deep neural networks

  • May 01, 2021
  • Review of Scientific Instruments
  • B S Schmidt +8
  • PDF
  • Research Article
  • Citations55

Log Files as a Tool for Improving Internet Dictionaries

  • Mar 08, 2017
  • HERMES - Journal of Language and Communication in Business
  • Henning Bergenholtz +1
  • Conference Article
  • Citations7

Benchmarking a site with realistic workload

  • Jan 01, 2002
  • Ballocca +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.