The Australian Geoscience Data Cube — Foundations and lessons learned

• Published in: Remote sensing of environment 2017-12, Vol.202, p.276-292
• Main Authors: Lewis, Adam, Oliver, Simon, Lymburner, Leo, Evans, Ben, Wyborn, Lesley, Mueller, Norman, Raevksi, Gregory, Hooke, Jeremy, Woodcock, Rob, Sixsmith, Joshua, Wu, Wenjun, Tan, Peter, Li, Fuqin, Killough, Brian, Minchin, Stuart, Roberts, Dale, Ayers, Damien, Bala, Biswajit, Dwyer, John, Dekker, Arnold, Dhu, Trevor, Hicks, Andrew, Ip, Alex, Purss, Matt, Richards, Clare, Sagar, Stephen, Trenham, Claire, Wang, Peter, Wang, Lan-Wei
• Format: Article
• Language: English
• Subjects: Archives & records; Australia; Australian Geoscience Data Cube; Big data; Collection management; Computer applications; computer software; Cubes; Data cube; Data management; Data processing; Developing countries; Earth; Earth science; Earth stations; Geometric correction; geometry; High performance computing; High performance data; Information processing; infrastructure; Landsat; LDCs; Management systems; Pixel quality; provenance; radiometry; Reflectance; Remote sensing; Satellite observation; Satellite tracking; Satellites; Source code; Studies; Time series; time series analysis
• ISSN: 0034-4257; 1879-0704
• DOI: 10.1016/j.rse.2017.03.015

The Australian Geoscience Data Cube (AGDC) aims to realise the full potential of Earth observation data holdings by addressing the Big Data challenges of volume, velocity, and variety that otherwise limit the usefulness of Earth observation data. There have been several iterations and AGDC version 2 is a major advance on previous work. The foundations and core components of the AGDC are: (1) data preparation, including geometric and radiometric corrections to Earth observation data to produce standardised surface reflectance measurements that support time-series analysis, and collection management systems which track the provenance of each Data Cube product and formalise re-processing decisions; (2) the software environment used to manage and interact with the data; and (3) the supporting high performance computing environment provided by the Australian National Computational Infrastructure (NCI). A growing number of examples demonstrate that our data cube approach allows analysts to extract rich new information from Earth observation time series, including through new methods that draw on the full spatial and temporal coverage of the Earth observation archives. To enable easy-uptake of the AGDC, and to facilitate future cooperative development, our code is developed under an open-source, Apache License, Version 2.0. This open-source approach is enabling other organisations, including the Committee on Earth Observing Satellites (CEOS), to explore the use of similar data cubes in developing countries. •Data-cubes can be used to analyse continental time-series of remote sensing data.•Geometric and spectral corrections are essential for comparability of measurements.•AGDC-v2 uses new tools including NoSQL and netCDF to improve flexibility.•Data-cubes allow new approaches to old problems, such as best pixel mosaics.