DICER: Data Intensive Computing Environment and Runtime for Evaluating Unprecedented Scale of Geospatial-Temporal Human Mobility Data

• Main Authors: De, Debraj, Thakur, Gautam Malviya, McGaha, Jesse, Brown, Chance, Nie, Xiuling, Thomas, Todd, Gaboardi, James D., Sparks, Kevin, Burger, Annetta, McBride, Elizabeth C., Kim, Joon-Seok, Amichi, Licia, Gunaratne, Chathika, Christopher, Carter, Zubko, Dan
• Format: Conference Proceeding
• Language: English
• Subjects: Big Data; big data analytics; cloud computing; Cluster computing; Computer architecture; Geospatial temporal data; human mobility; mobility metrics; Runtime; Systems architecture; Technological innovation; test and evaluation (T&E); Urban areas
• ISSN: 2375-0324
• DOI: 10.1109/MDM61037.2024.00037

With the significant increase in sources and volume of human mobility data through commercial data vendors as well as microsimulation of cities, the scale of geospatial-temporal data to analyze and assess for mobility characterization has grown to the level of Big Data. There are mobility related commercial organizations deploying scalable computing, but often the system architecture, workflow, and intermediate processing components are not fully disclosed in relevant scope. Current research literature has a notable lack of studies demonstrating architectures and workflows for human mobility analytics that are implemented on a TeraByte scale of geospatial-temporal data. In this context, this paper presents a hyperscale-level system solution named DICER (Data Intensive Computing Environment and Runtime) for processing and analytics of geospatial-temporal data at big data scale. Although the cluster computing architecture of DICER with Apache Spark job running on Kubernetes cluster is not new, there are innovations in the workflow, hierarchical processing logic, and a wide range of intermediate preprocessing and mobility metrics calculation. We have performed case studies to validate the effectiveness of DICER system solution by performing detailed analytics and assessment of human mobility microsimulation output at three different scopes and scale, including a usecase with 16.97 TeraByte and 259.2 Billion rows of data. In addition, we have presented another case study of utilizing DICER to perform the same mobility processing and comparative analytics on large-scale commercially available geospatial-temporal data. All these case studies validate the efficiency and usefulness of DICER in computing population mobility characteristics from geospatial-temporal trajectory data at an unprecedented scale (not only just data volume, but also combination of: number of user entities, temporal frequency, spatial resolution, data duration).