MaMR: High-performance MapReduce programming model for material cloud applications

详细信息    查看全文

• 作者：Weipeng Jing^a ; ^{weipeng.jing@outlook.com} ; Danyu Tong^a ; Yangang Wang^b ; ^{wangyg@sccas.cn} ; Jingyuan Wang^c ; Yaqiu Liu^a ; Peng Zhao^a
• 关键词：Materials ; Programming model ; MapReduce ; BSP ; Merge phase
• 刊名：Computer Physics Communications
• 出版年：2017
• 出版时间：February 2017
• 年：2017
• 卷：211
• 期：Complete
• 页码：79-87
• 全文大小：1628 K
• 卷排序：211

文摘

With the increasing data size in materials science, existing programming models no longer satisfy the application requirements. MapReduce is a programming model that enables the easy development of scalable parallel applications to process big data on cloud computing systems. However, this model does not directly support the processing of multiple related data, and the processing performance does not reflect the advantages of cloud computing. To enhance the capability of workflow applications in material data processing, we defined a programming model for material cloud applications that supports multiple different Map and Reduce functions running concurrently based on hybrid share-memory BSP called MaMR. An optimized data sharing strategy to supply the shared data to the different Map and Reduce stages was also designed. We added a new merge phase to MapReduce that can efficiently merge data from the map and reduce modules. Experiments showed that the model and framework present effective performance improvements compared to previous work.