SPO: A Secure and Performance-aware Optimization for MapReduce Scheduling

• Published in: Journal of network and computer applications 2021-02, Vol.176, p.102944, Article 102944
• Main Authors: Maleki, Neda, Rahmani, Amir Masoud, Conti, Mauro
• Format: Article
• Language: English
• Subjects: Bigdata; Hadoop; Heterogeneity; Makespan; MapReduce; Optimization model; Scheduling; Security
• ISSN: 1084-8045; 1095-8592
• DOI: 10.1016/j.jnca.2020.102944

MapReduce is a common framework that effectively processes multi-petabyte data in a distributed manner. Therefore, MapReduce is widely used in heterogeneous environments, such as cloud, to provide performance adequate for system needs. Despite the MapReduce benefits, tweaking the system configuration to achieve the maximum performance is still challenging and needs deep expertise. Besides, some new MapReduce security issues, which has not been well-addressed yet, are recently raised. In this paper, we present a performance-aware and secure framework, named SPO, to minimize the makespan of the tasks while considering task security constraints. Inspired by the HEFT algorithm, first, we introduce SPO, which proposes a two-stage static scheduler in Map and Reduce phases, respectively, to minimize makespan while considering network traffic. Plus, SPO∗ introduces a mathematical optimization model of the proposed scheduler aiming to estimate the system performance while considering security constraints with an error of less than 2%. The experimental results demonstrate that SPO outperforms Hadoop-stock in terms of makespan and network traffic by 29% and 31%, respectively, for the tasks running in heterogeneous environments. •SPO is an asynchronous two-stage framework for heterogeneous environments•SPO addresses Map and Reduce tasks scheduling while considering makespan and security•SPO proposes an optimization model for makespan minimization with security constraint•SPO applies a flexible partition binder for Reduce tasks to reduce the Shuffle time