A containerized task clustering for scheduling workflows to utilize processors and containers on clouds

• Published in: The Journal of supercomputing 2021-11, Vol.77 (11), p.12879-12923
• Main Authors: Kanemitsu, Hidehiro, Kanai, Kenji, Katto, Jiro, Nakazato, Hidenori
• Format: Article
• Language: English
• Subjects: Algorithms; Allocations; Cloud computing; Clustering; Compilers; Computer Science; Containers; Interpreters; Network latency; Parallel processing; Priority scheduling; Processor Architectures; Programming Languages; Resource utilization; Scheduling; Task scheduling; Workflow
• ISSN: 0920-8542; 1573-0484
• DOI: 10.1007/s11227-021-03789-2

Recent advancements of virtualization technologies for parallel processing involve scheduling containerized tasks in a workflow. Since a container can include multiple tasks, it can be reused or shared among applications. If every task in a workflow uses its dedicated container without sharing among any tasks, each container image must be downloaded for each task. As a result, many computational resources are required to process and the communication latency related to container image downloading can become a bottleneck for the makespan. In task scheduling algorithms for workflows, this characteristic produces a new challenging issue that how effectively shares containers among tasks to avoid redundant container image download processes and redundant task allocations. One of the fundamental problems is that no policy has been established for simultaneously satisfying effective container sharing, maintaining the degree of task parallelism, and effective computational resource utilization. In this paper, we propose a clustering-based containerized task scheduling algorithm for clouds, namely, shareable functional task clustering for utilizing virtualized resources (SF-CUV). The objective of SF-CUV is to minimize the makespan with less computational resources and containers than other algorithms by clustering tasks and sharing each container among tasks. SF-CUV consists of two phases: (i)task clustering and pre-virtual CPU (vCPU) allocation phase to derive an accurate scheduling priority, and (ii)task ordering and actual task reallocation phase. Experimental results obtained via simulation and in a real environment show that SF-CUV can utilize both vCPUs and containers with a shorter makespan compared with other approaches.