Data access reorganizations in compiling out-of-core data parallel programs on distributed memory machines

This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core progra...

Full description

Saved in:
Bibliographic Details
Main Authors: Kandemir, M., Bordawekar, R., Choudhary, A.
Format: Conference Proceeding
Language:English
Subjects:
Computational Intelligence Society
Concurrent computing
Cost function
Data flow computing
Large-scale systems
Message passing
Optimizing compilers
Parallel machines
Physics computing
Program processors
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate file layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow efficient file access, and (3) partitioning the available node memory among references based on I/O cost estimation. Our experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude.
ISSN:1063-7133
DOI:10.1109/IPPS.1997.580956