A mediation system for continuous spatial queries on a unified schema using Apache Spark

• Published in: Big earth data 2024, Vol.8 (1), p.115-141
• Main Authors: Ngo, Thi Thu Trang, Pinet, François, Sarramia, David, Kang, Myoung-Ah
• Format: Article
• Language: English
• Subjects: Computer Science; continuous queries; geospatial data; mediator; Streaming data; streaming data integration
• ISSN: 2096-4471; 2574-5417
• DOI: 10.1080/20964471.2023.2275854

Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things (IoT) systems and sensors in various domains. In this context, many applications require integrating data from several heterogeneous sources, either stream or static sources. Frameworks such as Apache Spark are able to integrate and process large datasets from different sources. However, these frameworks are hard to use when the data sources are heterogeneous and numerous. To address this issue, we propose a system based on mediation techniques for integrating stream and static data sources. The integration process of our system consists of three main steps: configuration, query expression and query execution. In the configuration step, an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources. In the query expression step, users express queries using customized SQL grammar on the mediated schema. Finally, our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster. The results are continuously returned to users. Our experiments show that our optimizations can improve query execution time by up to one order of magnitude, making complex streaming and spatial data analysis more accessible.