gSched : a resource aware Hadoop scheduler for heterogeneous cloud computing environments

Abstract

MapReduce has become a major programming model for data-intensive applications in cloud computing environments. Hadoop, an open source implementation of MapReduce, has been adopted by an increasingly wideuser community. However, Hadoop suffers from task scheduling performance degradation in heterogeneouscontexts because of its homogeneous design focus. This paper presents gSched, a resource-aware Hadoopscheduler that takes into account both the heterogeneity of computing resources and provisioning charges in taskallocation in cloud computing environments. gSched is initially evaluated in an experimental Hadoop clusterand demonstrates enhanced performance compared with the default Hadoop scheduler. Further evaluationsare conducted on the Amazon EC2 cloud that demonstrates the effectiveness of gSched in task allocation in het-erogeneous cloud computing environments.