Two-level Dynamic Load Balancing for High Performance Scientific Applications

Scientific applications are often complex, irregular, and computationally intensive. To accommodate their ever-increasing computational demands, high performance computing (HPC) systems have become larger and more complex, offering increased hardware parallelism at multiple levels (e.g., nodes, cores per node, threads per core). Scientific applications need to exploit all multilevel hardware parallelism to harness the available computational power. The performance of applications executing on such HPC systems may adversely be affected by load imbalance at multiple levels, caused by problem, algorithmic, and systemic characteristics. Existing dynamic load balancing methods do not simultaneously address load imbalance at multiple software parallelism levels. This work investigates the joint impact of load imbalance on the performance of three scientific applications at the thread and process levels. We jointly apply and evaluate selected dynamic loop self-scheduling (DLS) techniques to both levels. This approach is generic and applicable to any multiprocess-multithreaded computationally intensive application. We conduct an exhaustive set of experiments to assess and compare the combination of six DLS techniques at the thread level and eleven at the process level. The results show that improved overall application performance, by up to 21%, can only be achieved by jointly addressing load imbalance at both software parallelism levels. We offer insights into the performance of the selected DLS techniques and discuss the interplay of dynamic load balancing at the thread and process levels.