面向千兆以太网的动态RDMA通信方法
|
摘要
针对RDMA传输在千兆以太网中连接建立等待时间过长造成传输效率降低的问题,提出一种动态RDMA通信方法,并进行建模和实验。该方法在保持对传统通信方法兼容的基础上,提出间接传输方法及对应的动态选择策略。通过扩展NIC的硬件逻辑并预留相应的缓存资源,动态RDMA通信方法可以在连接未建立成功的情况下进行数据传输,同时继续保持连接的建立。该过程实现了数据传输与连接建立的并行进行,且对上层协议透明。以千兆以太网NIC为原型构建端系统模型,并搭建测试平台进行实验。实验结果表明,动态RDMA通信方法以很小的额外硬件开销解决了连接建立等待时间过长的问题,提供了优于传统RDMA通信方法的传输性能。
Waiting for connection establishment can be inefficient during remote direct memory access(RDMA) transport for Gigabit Ethernet. Aiming at solving this problem, a hardware accelerated dynamic RDMA method is proposed in this paper. This method allows starting RDMA transport at the same time with connection establishment by sending packages first to indirection buffer in network interface card(NIC) and copying them later to host memory. The terminal models of dynamic RDMA method are built and the simulation platforms for perform experiments are developed. Experimental results show that the dynamic RDMA method not only can solve the long waiting problem of connection establishment with little extra hardware cost, but also can provide higher transport performance compared with traditional RDMA methods.
Waiting for connection establishment can be inefficient during remote direct memory access(RDMA) transport for Gigabit Ethernet. Aiming at solving this problem, a hardware accelerated dynamic RDMA method is proposed in this paper. This method allows starting RDMA transport at the same time with connection establishment by sending packages first to indirection buffer in network interface card(NIC) and copying them later to host memory. The terminal models of dynamic RDMA method are built and the simulation platforms for perform experiments are developed. Experimental results show that the dynamic RDMA method not only can solve the long waiting problem of connection establishment with little extra hardware cost, but also can provide higher transport performance compared with traditional RDMA methods.
引文
[1]BROWN D.Are new approaches needed for developing long-term strategies for STEM information?[J].Learned Publishing,2017,30(3):21-28.
[2]KAGAN M.Performance evaluation of the RDMA over ethernet(RoCE)standard in enterprise data centers infrastructure[C]//The 23rd International Teletraffic Congress.San Francisco,USA:ACM,2011:9-15.
[3]GRANT R E,RASHTI M J,AFSAHI A,et al.RDMAcapable iWARP over datagrams[C]//Parallel&Distributed Processing Symposium.Boston,USA:IEEE,2013:628-639.
[4]GUO C,WU H,SONI G,et al.RDMA over commodity Ethernet at scale[C]//Conference on ACM SIGCOMM.Florianópolis,Brazil:ACM,2016:202-215.
[5]WANG Shao-gang,XU Wei-xia,WU Dan,et al.Fast NICbased RDMA implementation for adaptive unreliable networks[C]//The 11th International Conference on Computer Systems and Applications(AICCSA).Ifrane,Morocco:IEEE,2014:302-309.
[6]MOUZAKITIS A,PINTO C,NIKOLAEV N,et al.Lightweight and generic RDMA engine para-virtualization for the KVM hypervisor[C]//International Conference on High PERFORMANCE Computing&Simulation.Genoa,Italy:IEEE,2017:737-744.
[7]董勇,周恩强,卢宇彤,等.基于天河2高速互连网络实现混合层次文件系统H~2FS高速通信[J].计算机学报,2017,40(9):1961-1979.DONG Yong,ZHOU En-qiang,LU Yu-tong,et al.The implementation of communicating operation in hybrid hierarchy file system H2FS with TH-Express 2[J].Chinese Journal of Computers,2017,40(9):1961-1979.
[8]MA S,KIM J,MOON S.Exploring low-latency interconnect for scaling out software routers[C]//IEEEInternational Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era.Barcelona,Spain:IEEE,2016:9-15.
[9]夏军,庞征斌,刘路,等.一种基于NIC的RDMA可靠传输协议的设计与实现[J].计算机工程与科学,2014,36(2):216-221.XIA Jun,PANG zhen-bin,LIU Lu,et al.Design and implementation of a NIC based RDMA reliable communication protocol[J].Computer Engineering and Science,2014,36(2):216-221.
[10]FREY P W,ALONSO G.Minimizing the hidden cost of RDMA[C]//The 29th IEEE International Conference on Distributed Computing Systems.Montreal,Canada:IEEE,2009:553-560.
[11]MAC A P,RUSSELL R D.A performance study to guide RDMA programming decisions[C]//The 14th International Conference on High Performance Computing and Communication&2012 IEEE 9th International Conference on Embedded Software and Systems(HPCC-ICESS).Liverpool,United Kingdom:IEEE,2012:778-785.
[12]SUBRAMONI H,LAI P,LUO M,et al.RDMA over Ethernet-a preliminary study[C]//IEEE International Conference on Cluster Computing and Workshops.New Orleans,USA:IEEE,2009:1-9.
[13]ZHANG W,HAO M,XU Z.Communication optimization for RDMA-based science data transmission tools[J].Journal of Super Computing,2016,72(9):3312-3327.
[14]MAC A P,RUSSELL R D.An efficient method for stream semantics over RDMA[C]//The 28th International Parallel and Distributed Processing Symposium.Phoenix,USA:IEEE,2014:841-851.
[15]苏文,章隆兵,高翔,等.基于Cache锁和直接缓存访问的网络处理优化方法[J].计算机研究与发展,2014,51(3):681-690.SU Wen,ZHANG Long-bing,GAO Xiang,et al.A cache locking and direct cache access based network processing optimization method[J].Journal of Computer Research and Development,2014,51(3):681-690.
[16]ANDREW S T,DAVID J W.Computer networks[M].New Jersey,USA:Pearson,2011.
[17]JIN H W,NARRAVULA S,BROWN G,et al.Performance evaluation of rdma over ip:a case study with the ammasso gigabit ethernetnic[C]//The 14th IEEE International Symposium on High Performance Distributed Computing(HPDC-14).Virginia,USA:IEEE,2005:598-605.
[18]LI Long-fei,HE Zhan-zhuang,WANG Jian-feng,et al.Implementation of gigabit ethernet controller with fault tolerance and prevention mechanism[C]//2017 Prognostics and System Health Management Conference(PHM-Harbin).Harbin,China:IEEE,2017:1-8.
[2]KAGAN M.Performance evaluation of the RDMA over ethernet(RoCE)standard in enterprise data centers infrastructure[C]//The 23rd International Teletraffic Congress.San Francisco,USA:ACM,2011:9-15.
[3]GRANT R E,RASHTI M J,AFSAHI A,et al.RDMAcapable iWARP over datagrams[C]//Parallel&Distributed Processing Symposium.Boston,USA:IEEE,2013:628-639.
[4]GUO C,WU H,SONI G,et al.RDMA over commodity Ethernet at scale[C]//Conference on ACM SIGCOMM.Florianópolis,Brazil:ACM,2016:202-215.
[5]WANG Shao-gang,XU Wei-xia,WU Dan,et al.Fast NICbased RDMA implementation for adaptive unreliable networks[C]//The 11th International Conference on Computer Systems and Applications(AICCSA).Ifrane,Morocco:IEEE,2014:302-309.
[6]MOUZAKITIS A,PINTO C,NIKOLAEV N,et al.Lightweight and generic RDMA engine para-virtualization for the KVM hypervisor[C]//International Conference on High PERFORMANCE Computing&Simulation.Genoa,Italy:IEEE,2017:737-744.
[7]董勇,周恩强,卢宇彤,等.基于天河2高速互连网络实现混合层次文件系统H~2FS高速通信[J].计算机学报,2017,40(9):1961-1979.DONG Yong,ZHOU En-qiang,LU Yu-tong,et al.The implementation of communicating operation in hybrid hierarchy file system H2FS with TH-Express 2[J].Chinese Journal of Computers,2017,40(9):1961-1979.
[8]MA S,KIM J,MOON S.Exploring low-latency interconnect for scaling out software routers[C]//IEEEInternational Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era.Barcelona,Spain:IEEE,2016:9-15.
[9]夏军,庞征斌,刘路,等.一种基于NIC的RDMA可靠传输协议的设计与实现[J].计算机工程与科学,2014,36(2):216-221.XIA Jun,PANG zhen-bin,LIU Lu,et al.Design and implementation of a NIC based RDMA reliable communication protocol[J].Computer Engineering and Science,2014,36(2):216-221.
[10]FREY P W,ALONSO G.Minimizing the hidden cost of RDMA[C]//The 29th IEEE International Conference on Distributed Computing Systems.Montreal,Canada:IEEE,2009:553-560.
[11]MAC A P,RUSSELL R D.A performance study to guide RDMA programming decisions[C]//The 14th International Conference on High Performance Computing and Communication&2012 IEEE 9th International Conference on Embedded Software and Systems(HPCC-ICESS).Liverpool,United Kingdom:IEEE,2012:778-785.
[12]SUBRAMONI H,LAI P,LUO M,et al.RDMA over Ethernet-a preliminary study[C]//IEEE International Conference on Cluster Computing and Workshops.New Orleans,USA:IEEE,2009:1-9.
[13]ZHANG W,HAO M,XU Z.Communication optimization for RDMA-based science data transmission tools[J].Journal of Super Computing,2016,72(9):3312-3327.
[14]MAC A P,RUSSELL R D.An efficient method for stream semantics over RDMA[C]//The 28th International Parallel and Distributed Processing Symposium.Phoenix,USA:IEEE,2014:841-851.
[15]苏文,章隆兵,高翔,等.基于Cache锁和直接缓存访问的网络处理优化方法[J].计算机研究与发展,2014,51(3):681-690.SU Wen,ZHANG Long-bing,GAO Xiang,et al.A cache locking and direct cache access based network processing optimization method[J].Journal of Computer Research and Development,2014,51(3):681-690.
[16]ANDREW S T,DAVID J W.Computer networks[M].New Jersey,USA:Pearson,2011.
[17]JIN H W,NARRAVULA S,BROWN G,et al.Performance evaluation of rdma over ip:a case study with the ammasso gigabit ethernetnic[C]//The 14th IEEE International Symposium on High Performance Distributed Computing(HPDC-14).Virginia,USA:IEEE,2005:598-605.
[18]LI Long-fei,HE Zhan-zhuang,WANG Jian-feng,et al.Implementation of gigabit ethernet controller with fault tolerance and prevention mechanism[C]//2017 Prognostics and System Health Management Conference(PHM-Harbin).Harbin,China:IEEE,2017:1-8.