Scalable feature subset selection for big data using parallel hybrid evolutionary algorithm based wrapper under apache spark environment

Extant sequential wrapper-based feature subset selection (FSS) algorithms are not scalable and yield poor performance when applied to big datasets. Hence, to circumvent these challenges, we propose parallel and distributed hybrid evolutionary algorithms (EAs) based wrappers under Apache Spark. We pr...

Full description

Saved in:
Bibliographic Details
Published in:Cluster computing 2023-06, Vol.26 (3), p.1949-1983
Main Authors: Vivek, Yelleti, Ravi, Vadlamani, Krishna, P. Radha
Format: Article
Language:English
Subjects:
Ablation
Adaptive algorithms
Big Data
Computer Communication Networks
Computer Science
Data mining
Datasets
Design
Evolutionary algorithms
Evolutionary computation
Genetic algorithms
Iterative methods
Operating Systems
Operators (mathematics)
Optimization
Parallel processing
Permutations
Processor Architectures
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Extant sequential wrapper-based feature subset selection (FSS) algorithms are not scalable and yield poor performance when applied to big datasets. Hence, to circumvent these challenges, we propose parallel and distributed hybrid evolutionary algorithms (EAs) based wrappers under Apache Spark. We propose two hybrid EAs based on the Binary Differential Evolution (BDE), and Binary Threshold Accepting (BTA), namely, (i) Parallel Binary Differential Evolution and Threshold Accepting (PB-DETA), where BDE and BTA work in tandem in every iteration, and (ii) its ablation variant, Parallel Binary Threshold Accepting and Differential Evolution (PB-TADE). Here, BTA is invoked to enhance the search capability and avoid premature convergence of BDE. For comparison purposes, we also parallelized two state-of-the-art algorithms: adaptive DE (ADE) and permutation based DE (DE-FS PM ), and named them PB-ADE and P-DE-FS PM respectively. Throughout, logistic regression (LR) is employed to compute the fitness function, namely, area under the receiver operator characteristic curve (AUC). The effectiveness of the proposed algorithms is tested over the five big datasets of varying dimensions. It is noteworthy that the PB-TADE turned out to be statistically significant than the rest. All the algorithms have shown the repeatability property. The proposed parallel model attained a speedup of 2.2–2.9. We also reported feature subset with high AUC and least cardinality.
ISSN:1386-7857
1573-7543
DOI:10.1007/s10586-022-03725-w