Resource Selection in Large-Scale Distributed System Using Dynamic Task Sharing

• Main Authors: Dae Gun Kim, Zhong Yuan Li, Sung Soo Moon, Hee Yong Youn
• Format: Conference Proceeding
• Language: English
• Subjects: Data intensive computing; Distributed databases; Dynamic scheduling; Estimation; large scale distributed systems; Peer to peer computing; Processor scheduling; Resource management; resource selection; Servers; task sharing
• DOI: 10.1109/GreenCom-CPSCom.2010.53

Large-scale distributed system provides an attractive scalable infrastructure for network applications. In such environment there exist large sets of heterogeneous and geographically distributed resources. These resources can be aggregated as a virtual computing platform for executing large-scale scientific applications. Among numerous optional resources, selecting appropriate resources for applications is challenging and affected by many factors. The loosely coupled nature of large-scale distributed environment makes data access unpredictable and instability. Slow allocation process may offset the benefit obtained by running a job on a fast node. Besides, the operation condition of a resource provider changes rapidly. The status of job execution and computing capability of a resource provider need to be considered dynamically. In this paper we present an approach of dynamic task-sharing based on the record of previous data download and current execution status of resource providers to select the appropriate one or more providers to execute a job together. The proposed approach can also avoids single point failure and server overload problem.