The impact of information availability and workload characteristics on the performance of job co-allocation in multi-clusters

• Main Authors: Jones, W.M., Ligon, W.B., Shrivastava, N.
• Format: Conference Proceeding
• Language: English
• Subjects: Availability; clusters; Computer networks; Concurrent computing; Degradation; multiple computational; multisite scheduling; network contention; Network topology; Parallel architectures; parallel job scheduling; Processor scheduling; Runtime; System performance; Throughput; workload effects
• ISSN: 1521-9097; 2690-5965
• DOI: 10.1109/ICPADS.2006.105

In this paper, we utilize a bandwidth-centric job communication model that captures the interaction and impact of simultaneously co-allocating jobs across multiple clusters. We make use of a parallel job model that seeks to capture both local and global communication access patterns. By doing so, we are able to explore scheduling strategies that attempt to improve average job turnaround time by selectively mapping jobs across cluster boundaries in a process known as job co-allocation. In this research, we focus on scheduling strategies that make use of available information such as network link utilization, per-processor bandwidths, and job communication topology in order to make intelligent decisions regarding application partition sizes and job placement. We provide results that help to establish the relationship between the quantity of information available a priori to the scheduler and its ability to improve overall system performance. Additionally, we demonstrate the dramatic impact that salient workload characteristics can have on the effectiveness of co-allocation