Performance-Aware Big Data Management for Remote Sensing Systems

Remote sensing data, whose dimensions increase exponentially and turn into big data with the new technologies, cause significant difficulties in transferring, storing, and processing because of consisting of gigantic coarse-grained files. This article proposes a novel two-phase big data management s...

Full description

Saved in:
Bibliographic Details
Published in:Arabian journal for science and engineering (2011) 2024-03, Vol.49 (3), p.3845-3865
Main Authors: Pekturk, Mustafa Kemal, Unal, Muhammet, Gokcen, Hadi
Format: Article
Language:English
Subjects:
Algorithms
Big Data
Data analysis
Data management
Engineering
Humanities and Social Sciences
multidisciplinary
Network topologies
New technology
Remote sensing
Remote sensing systems
Research Article-Computer Engineering and Computer Science
Resource allocation
Resource utilization
Science
Virtual environments
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Remote sensing data, whose dimensions increase exponentially and turn into big data with the new technologies, cause significant difficulties in transferring, storing, and processing because of consisting of gigantic coarse-grained files. This article proposes a novel two-phase big data management system on the geo-distributed private cloud that takes advantage of network topology and resource utilization in a distributed manner. The system optimizes resource allocation to facilitate efficient and extensive data analysis for remote sensing applications by minimizing file fragmentation, resulting in faster analysis. In order to simulate the proposed system, different network topologies are created using virtual machines. Moreover, the proposed method named performance-aware assignment is compared with well-known methods such as random assignment, Hungarian algorithm, and Hadoop Distributed File System, also famous in the big data era. The experimental results indicate that performance-aware assignment outperforms random assignment, Hungarian algorithm, and Hadoop Distributed File System, achieving 36%, 26%, and 71% more stored data, respectively, within the same time while also exhibiting lower IOPS values. In addition, it optimizes resource usage in data centers, which is particularly important for preventing resource exhaustion.
ISSN:2193-567X
1319-8025
2191-4281
DOI:10.1007/s13369-023-08172-2