Open Access
3- https://doi.org/10.14778/2536222.2536224
Optimization strategies for A/B testing on HADOOP
- Aug 1, 2013
- Proceedings of the VLDB Endowment
- Andrii Cherniak +2 more
In this work, we present a set of techniques that considerably improve the performance of executing concurrent MapReduce jobs. Our proposed solution relies on proper resource allocation for concurrent Hive jobs based on data dependency, inter-query optimization and modeling of Hadoop cluster load. To the best of our knowledge, this is the first work towards Hive/MapReduce job optimization which takes Hadoop cluster load into consideration. We perform an experimental study that demonstrates 233% reduction in execution time for concurrent vs sequential execution schema. We report up to 40% extra reduction in execution time for concurrent job execution after resource usage optimization. The results reported in this paper were obtained in a pilot project to assess the feasibility of migrating A/B testing from Teradata + SAS analytics infrastructure to Hadoop. This work was performed on eBay production Hadoop cluster.
- # Reduction In Execution Time
- # Resource Usage Optimization
- # Extra Reduction
- # Hadoop Cluster
- # Concurrent Execution
- # Reduction In Time
- # Data Dependency
- # Modeling Of Load
- # Proper Allocation
- # Execution Time
