Scheduling multiple virtual environments in cloud federations for distributed calculations

• Published in: Future generation computer systems 2017-09, Vol.74, p.90-103
• Main Authors: Rubio-Montero, A.J., Huedo, E., Mayo-García, R.
• Format: Article
• Language: English
• Subjects: Application compatibility; Cloud middleware; Cloud scheduling; Pilot jobs; Resource management
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2016.03.021

In the pool of cloud providers that are currently available there is a lack of standardised APIs and brokering tools to effectively distribute high throughput calculations among them. Moreover, the current middleware tools are not able to straightforwardly provision the ephemeral and specific environments that certain codes and simulations require. These facts prevent the massive portability of legacy applications to cloud environments. Such an issue can be overcome by effectively scheduling the distributed calculations using the basic capacities offered by cloud federations. In this work, a framework achieving such a goal is presented: a pilot system (GWpilot) that has been improved with cloud computing capabilities (GWcloud). This framework profits from the expertise acquired in grid federations and provides interesting features that make it more efficient, flexible and useable than other approaches. Thus, decentralisation, middleware independence, dynamic brokering, on-demand provisioning of specific virtual images, compatibility with legacy applications, and the efficient accomplishment of short tasks, among other features, are achieved. Not only this, the new framework is multi-user and multi-application, dynamically instantiating virtual machines depending on the available and demanded resources, i.e. allowing users to consolidate their own resource provisioning. Results presented in this work demonstrate these features by efficiently executing several legacy applications with very different requirements on the FedCloud infrastructure at the same time. •Dynamic, configurable, and efficient IaaS brokering across multiple providers for HTC.•Decentralisation and on-demand provisioning of customised environments in cloud.•Users deploy VMs with arbitrary configurations and perform post-configuration tasks.•Support of multiple environments and fair-sharing among users and applications.•Compatibility with legacy applications and grid services with middleware independence.