Accelerating big data analytics on HPC clusters using two-level storage

详细信息    查看全文

• 作者：Pengfei Xuan^a ; Walter B. Ligon^b ; Pradip K. Srimani^a ; Rong Ge^a ; Feng Luo^a ; ^{luofeng@clemson.edu}
• 关键词：Two-level storage ; In-memory file system ; Parallel file system ; Data-intensive computing
• 刊名：Parallel Computing
• 出版年：2017
• 出版时间：January 2017
• 年：2017
• 卷：61
• 期：Complete
• 页码：18-34
• 全文大小：3193 K
• 卷排序：61

文摘

Develop a new two-level storage system that integrates an upper-level in-memory file system with a lower-level parallel file system. Model and compare I/O throughput of two-level storage to HDFS and OrangeFS. Build a prototype of two-level storage system with Tachyon and OrangeFS, Conduct experiments on real systems show that the proposed two-level storage delivers higher aggregate I/O throughputs than HDFS and OrangeFS and achieves weak scalability on both read and write.