Hierarchical Read–Write Optimizations for Scientific Applications with Multi-variable Structured Datasets
详细信息 查看全文
- 作者：Preeti Malakar ; Venkatram Vishwanath
- 关键词：Read ; Write ; Multi ; variable dataset ; Scientific applications
- 刊名：International Journal of Parallel Programming
- 出版年：2017
- 出版时间：February 2017
- 年：2017
- 卷：45
- 期：1
- 页码：94-108
- 全文大小：
- 刊物类别：Computer Science
- 刊物主题：Theory of Computation; Processor Architectures; Software Engineering/Programming and Operating Systems;
- 出版者：Springer US
- ISSN：1573-7640
- 卷排序：45
文摘
Large-scale scientific applications spend a significant amount of time in reading and writing data. These simulations run on supercomputers which are architected with high-bandwidth, low-latency, and complex topology interconnects. Yet, few efforts exist that fully exploit the interconnect features for I/O. MPI-IO optimizations suffer from significant network contention at large core counts making I/O a critical bottleneck at extreme scales. We propose HieRO, which leverages the fast interconnect and performs hierarchical optimizations for I/O in scientific applications with structured datasets. HieRO performs reads/writes in multiple stages using carefully chosen leader processes who invoke the MPI-IO calls. Additionally, HieRO considers the application’s domain decomposition and access patterns and fully utilizes the on-chip interconnect at each multicore node. We evaluate the efficacy of our optimizations with two scientific applications, WRF and S3D, with I/O access patterns commonly used in a wide gamut of applications. We evaluate our approaches on two supercomputers, the Edison Cray XC30 and the Mira Blue Gene/Q, representing systems with diverse interconnects and parallel filesystems. We demonstrate that algorithmic changes can lead to significant improvements in parallel read/write. HieRO is able to achieve more than \(40\times \) read time improvements for WRF and achieve up to \(40\times \) read and \(13\times \) write time improvements for S3D on 524288 cores.