Unlocking the Australian Landsat Archive – From dark data to High Performance Data infrastructures

• Published in: GeoResJ (Amsterdam) 2015-06, Vol.6, p.135-140
• Main Authors: Purss, Matthew B.J., Lewis, Adam, Oliver, Simon, Ip, Alex, Sixsmith, Joshua, Evans, Ben, Edberg, Roger, Frankish, Glenn, Hurst, Lachlan, Chan, Tai
• Format: Article
• Language: English
• Subjects: Big data; Data rescue; Earth Observation; High Performance Computing; High Performance Data; Landsat
• ISSN: 2214-2428; 2214-2428
• DOI: 10.1016/j.grj.2015.02.010

Earth Observation data acquired by the Landsat missions are of immense value to the global community and constitute the world’s longest continuous civilian Earth Observation program. However, because of the costs of data storage infrastructure these data have traditionally been stored in raw form on tape storage infrastructures which introduces a data retrieval and processing overhead that limits the efficiency of use of this data. As a consequence these data have become ‘dark data’ with only limited use in a piece-meal and labor intensive manner. The Unlocking the Landsat Archive project was set up in 2011 to address this issue and to help realize the true value and potential of these data. The key outcome of the project was the migration of the raw Landsat data that was housed in tape archives at Geoscience Australia to High Performance Data facilities hosted by the National Computational Infrastructure (a super computer facility located at the Australian National University). Once this migration was completed the data were calibrated to produce a living and accessible archive of sensor and scene independent data products derived from Landsat-5 and Landsat-7 data for the period 1998–2012. The calibrated data were organized into High Performance Data structures, underpinned by ISO/OGC standards and web services, which have opened up a vast range of opportunities to efficiently apply these data to applications across multiple scientific domains.