GPU-In-Hadoop: MapReduce on Distributed Heterogeneous Platforms.
详细信息
文摘
There are four main challenges that have arisen as the scales of high performance distributed systems grow. Those challenges are the resilience to failure,the programmability,the heterogeneity,and the energy efficiency of those systems. Accomplishing all four without sacrificing performance requires a rethinking of legacy distributed programming models processors and homogeneous clusters. In this paper,the Hadoop system is integrated with CUDA to implement the utilization of heterogeneous processors in a distributed system. This process is achieved by exploiting the implicit data parallelism of mapper and reducer in the Hadoop MapReduce. Combining Hadoop with CUDA provides three excellent merits. First,both of Hadoop and CUDA are easy-to-learn and flexible application language. Second,Hadoop produces the reliability guarantees and distributed file system. Third,the low power consumption and performance acceleration of parallel processors are provided by CUDA. Four approaches will be presented using JCUDA,JNI,and Hadoop Pipes,as well as Hadoop streaming,to extend to Hadoop the support execution of user-written kernels on GPU.