Big data: an optimized approach for cluster initialization

• Published in: Journal of big data 2023-12, Vol.10 (1), p.120-19, Article 120
• Main Authors: Gul, Marina, Rehman, M. Abdul
• Format: Article
• Language: English
• Subjects: Algorithms; Big Data; Cluster initialization; Clustering; Communications Engineering; Computational Science and Engineering; Computer Science; Data Mining and Knowledge Discovery; Database Management; Datasets; Information Storage and Retrieval; k-means; Large scale data; Mathematical Applications in Computer Science; Networks; Optimization; Variance analysis
• ISSN: 2196-1115; 2196-1115
• DOI: 10.1186/s40537-023-00798-1

The k-means, one of the most widely used clustering algorithm, is not only faster in computation but also produces comparatively better clusters. However, it has two major downsides, first it is sensitive to initialize k value and secondly, especially for larger datasets, the number of iterations could be very large, making it computationally hard. In order to address these issues, we proposed a scalable and cost-effective algorithm, called R-k-means, which provides an optimized solution for better clustering large scale high-dimensional datasets. The algorithm first selects O(R) initial points then reselect O(l) better initial points, using distance probability from dataset. These points are then again clustered into k initial points. An empirical study in a controlled environment was conducted using both simulated and real datasets. Experimental results showed that the proposed approach outperformed as compared to the previous approaches when the size of data increases with increasing number of dimensions.