Dynamic cluster scheduling for ALICE

Abstract

With the ever-increasing growth in data processing, simulation and other computation-heavy operations, more organisations are resorting to the use of computer clusters in order to keep up with their computational demands. A Large Ion Collider Experiment (ALICE), one of the experiments at the European Organisation for Nuclear Research (CERN), clearly exemplifies this demand for computational power, with hundreds of computing centres devoted to process and analyse petabytes of data generated yearly by the Large Hadron Collider (LHC). A number of problem domains often employ the use of specialised distributed software frameworks which assume total control of cluster resources, making efficient resource utilisation difficult when running the frameworks concurrently. In this work, meta-scheduling approaches are investigated in order to mitigate the resource allocation problem present when running multiple frameworks concurrently; in par- ticular, Apache Mesos, a meta-scheduler for distributed systems, is employed and extended to improve resource allocation without resorting to static partitioning. Furthermore, this research investigates the use of network topology-aware scheduling for a number of application profiles; a novel method for automatic dynamic network topology discovery using Layer 2 devices is presented and evaluated. The results show that employing topology-aware meta-scheduling strategies can drastically improve performance for certain application profiles, demonstrating significant speed-up, while also improving resource utilisation.