Efficient distributed algorithms for distance join queries in spark-based spatial analytics systems

Apache Sedona (formerly GeoSpark) is a new in-memory cluster computing system for processing large-scale spatial data, which extends the core of Apache Spark to support spatial datatypes, partitioning techniques, spatial indexes, and spatial operations (e.g. spatial range, nearest neighbor, and spat...

Full description

Saved in:
Bibliographic Details
Published in:International journal of general systems 2023-04, Vol.52 (3), p.206-250
Main Authors: García-García, Francisco, Corral, Antonio, Iribarne, Luis, Vassilakopoulos, Michael
Format: Article
Language:English
Subjects:
Algorithms
apache sedona
Clusters
Computation
distributed spatial data processing
k closest pairs query
k nearest neighbor join query
locationspark
Optimization
Partitioning
Performance indices
Queries
simba
Spatial data
spatial query evaluation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Apache Sedona (formerly GeoSpark) is a new in-memory cluster computing system for processing large-scale spatial data, which extends the core of Apache Spark to support spatial datatypes, partitioning techniques, spatial indexes, and spatial operations (e.g. spatial range, nearest neighbor, and spatial join queries). Distance-based Join Queries (DJQs), like nearest neighbor join (kNNJQ) or closest pairs queries (kCPQ), are not supported by it. Therefore, in this paper, we investigate how to design and implement efficient DJQ distributed algorithms in Apache Sedona, using the most appropriate spatial partitioning and other optimization techniques. The results of an extensive set of experiments with real-world datasets are presented, demonstrating that the proposed kNNJQ and kCPQ distributed algorithms are efficient, scalable, and robust in Apache Sedona. Finally, Sedona is also compared to other similar cluster computing systems, showing the best performance for kCPQ and competitive results for kNNJQ.
ISSN:0308-1079
1563-5104
DOI:10.1080/03081079.2023.2173750