多核架构下硬件辅助I/O虚拟化研究与实现
|
摘要
近些年来,虚拟化技术由于其突出的优点受到工业界和学术界的共同关注和重视,但是虚拟化技术会带来额外性能开销,特别是对于偏重于I/O访问的任务。如何切实提高I/O虚拟化的效率,尤其是对于基于Many-Core处理器平台虚拟化系统而言,是一个亟待解决的问题。硬件辅助I/O虚拟化技术,支持客户操作系统在几乎没有虚拟机监控器干涉下,安全可靠的直接进行I/O设备访问,因而能够避免大量的虚拟化开销。目前,硬件辅助I/O虚拟化技术被认为是解决I/O虚拟化的最基本措施。
本文结合实际的工程任务需要,在处理器的PCIE总线接口层设计实现了一种面向多核架构的硬件辅助虚拟化系统。针对基于多核架构虚拟化系统IO访问频繁,可靠性要求高的特点,该设计从DMA访问、I/O中断处理和PIO访问3个方面着手进行专门设计。对于DMA访问,通过增加IO空间映射,优化了IO TLB缓存组织结构以及采用预取技术来实现设备DMA访问过程安全可靠以及高效性;对于I/O中断处理,则采取芯片内直接集成中断映射表加速中断重映射过程,采用基于事件队列异步中断处理机制处理主要的I/O中断——MSI中断,加速了该中断的处理并且减少该中断处理对CPU工作的影响;对于PIO访问,则采用Tag分配与记分牌相结合的管理机制,确保了所有的PIO访问可靠性以及PIO数据的正确返回。
本文还针对多核处理器上运行的虚拟化系统需要频繁地进行共享设备访问,而现有的硬件辅助I/O虚拟化不支持共享设备直接访问的缺点,在PCIE总线接口层内实现了一种硬件辅助的自虚拟化结构,允许多客户操作系统,在几乎没有虚拟机监控器的干涉下,直接访问共享设备,有效的提高了虚拟化系统对共享设备访问的效率。与已有的硬件辅助的自虚拟化技术相比,该结构具有硬件实现代价低,可灵活配置性等优点。
Recently, Virtualization Technology appeals the common interests of both the industry and academy due to its obvious advantages. However, it also encounters the problem of serious performance overheads, especially for the I/O-intensive workload; therefore it is an urgent task to solve the performance overheads caused by I/O virtualization, particularly for the virtualization systems based on many-core processor. The hardware-assisted I/O virtualization Technology, which enables the Guest OSs directly and safely accesses the device without the VMM software's interference, is considered to be one of the best solutions to the overheads problem.
This paper presents a design of hardware-assisted I/O virtualization for many-core platform, which is based on the PCIE interface layer for the practical project demands. To reach the requirements of frequent I/O access and high reliability in the virtualization systems based on the many-core processor, this design make special efforts on the DMA remapping, interrupt remapping and PIO which is for the core to configurate the devices’control and state registers. For DMA remapping, it add the I/O space mapping function to ensure isolation between different domains. It speed up the address translation procedure between guest virtual address and host physical address through optimizing the structure of I/O Translate Lookaside Buffer and prefetching technique. For interrupt remapping, this design integrate interrupt remapping table on the chip to accelerate the remapping procedure, in addition, it adopt an asynchronous event queue mechanism to handle the MSI interrupt,a dominant I/O interrupt type, which could accelerate the speed of handling this kind interrupts and reduce the performance impact on the cores. For PIO, this design uses a tactics which combines the tag assignment with score board mechanism, to improve the reliability and remap the PIO completion data.
Since there are so many I/O accesses towards the shared device in the virtualization system based on many-core platform and they are not applicable to the design mentioned above, this paper addresses a solution named Hardware-assisted Self-Virtualization, which enables multiple guest OS directly access the device. Compared with existing Self-Virtualization Technology, my solution has such advantages as low hardware cost and flexible configuration.
本文结合实际的工程任务需要,在处理器的PCIE总线接口层设计实现了一种面向多核架构的硬件辅助虚拟化系统。针对基于多核架构虚拟化系统IO访问频繁,可靠性要求高的特点,该设计从DMA访问、I/O中断处理和PIO访问3个方面着手进行专门设计。对于DMA访问,通过增加IO空间映射,优化了IO TLB缓存组织结构以及采用预取技术来实现设备DMA访问过程安全可靠以及高效性;对于I/O中断处理,则采取芯片内直接集成中断映射表加速中断重映射过程,采用基于事件队列异步中断处理机制处理主要的I/O中断——MSI中断,加速了该中断的处理并且减少该中断处理对CPU工作的影响;对于PIO访问,则采用Tag分配与记分牌相结合的管理机制,确保了所有的PIO访问可靠性以及PIO数据的正确返回。
本文还针对多核处理器上运行的虚拟化系统需要频繁地进行共享设备访问,而现有的硬件辅助I/O虚拟化不支持共享设备直接访问的缺点,在PCIE总线接口层内实现了一种硬件辅助的自虚拟化结构,允许多客户操作系统,在几乎没有虚拟机监控器的干涉下,直接访问共享设备,有效的提高了虚拟化系统对共享设备访问的效率。与已有的硬件辅助的自虚拟化技术相比,该结构具有硬件实现代价低,可灵活配置性等优点。
Recently, Virtualization Technology appeals the common interests of both the industry and academy due to its obvious advantages. However, it also encounters the problem of serious performance overheads, especially for the I/O-intensive workload; therefore it is an urgent task to solve the performance overheads caused by I/O virtualization, particularly for the virtualization systems based on many-core processor. The hardware-assisted I/O virtualization Technology, which enables the Guest OSs directly and safely accesses the device without the VMM software's interference, is considered to be one of the best solutions to the overheads problem.
This paper presents a design of hardware-assisted I/O virtualization for many-core platform, which is based on the PCIE interface layer for the practical project demands. To reach the requirements of frequent I/O access and high reliability in the virtualization systems based on the many-core processor, this design make special efforts on the DMA remapping, interrupt remapping and PIO which is for the core to configurate the devices’control and state registers. For DMA remapping, it add the I/O space mapping function to ensure isolation between different domains. It speed up the address translation procedure between guest virtual address and host physical address through optimizing the structure of I/O Translate Lookaside Buffer and prefetching technique. For interrupt remapping, this design integrate interrupt remapping table on the chip to accelerate the remapping procedure, in addition, it adopt an asynchronous event queue mechanism to handle the MSI interrupt,a dominant I/O interrupt type, which could accelerate the speed of handling this kind interrupts and reduce the performance impact on the cores. For PIO, this design uses a tactics which combines the tag assignment with score board mechanism, to improve the reliability and remap the PIO completion data.
Since there are so many I/O accesses towards the shared device in the virtualization system based on many-core platform and they are not applicable to the design mentioned above, this paper addresses a solution named Hardware-assisted Self-Virtualization, which enables multiple guest OS directly access the device. Compared with existing Self-Virtualization Technology, my solution has such advantages as low hardware cost and flexible configuration.
引文
[1] Rich Ublig, Gil Neiger, et al. Intel Virtualization Technology. IEEE Computer Magazine, 2005, 38(5): 48-56.
[2] Microsoft Corp, AMD Inc. Virtual Server 2005 R2 SP1 and Next Generation AMD Opteron? Processors Increase the Business Value of Virtualization. www.microsoft.com/korea/virtualization/businessvalue.mspx.
[3] VMware Inc. Virtualization Overview. www.vmware.com/pdf/virtualization.pdf.
[4] AMD Inc. AMD64 Virtualization Codenamed“Pacifica”Technology: Secure Virtual Machine Architecture Reference Manual. www.amd.com/us-en/assets.
[5] Enterprise Management Associates. Virtualization: Exposing the Intangible Enterprise. Enterprise Management, 2006. www.infoworld.com/event/virtualization.
[6] Sun Microsystems. Virtualization Strategy: From Servers to Services. http://br.sun.com/sunnews/events/2006/virtual/pdfs.
[7] W.J.Armstrong, R.L.Arndt, et al. Advanced Virtualization Capabilities of Power5 Systems, IBM Journal. RES & DEV, 2005, 49: 523-532.
[8] W.J.Armstrong, R.L.Arndt, et al. IBM POWER6 Partition Mobility: Moving virtual servers seamlessly between physical systems. IBM Journal. RES & DEV, 2006, 51: 757-762.
[9] John L.HENNESSY, David A. PATTERSON著,白跃彬译.计算机系统结构:量化研究方法(第四版).电子工业出版社, 2007.
[10] Robert Rose. Survey of System Virtualization Technology. 2004.2. http://citeseer.ist.psu.edu/720518.html.
[11] Susanta Nanda, Tzi-cker Chiueh. A survey on Virtualization Techonologies. http://www.ecsl.cs.sunysb.edu/tr/TR179.pdf.
[12] VMware Inc. Understanding Full Virtualization, Paravirtualization, and Hardware Assist. www.vmware.com/files/pdf/VMware_paravirtualization.pdf.
[13] Barham P, Dragovic B, Fraser K et al. Xen and the art of virtualization. Proceedings of the 19th ACM SOSP, 2003: 164–177.
[14] A. Menon, J. R. Santos, Y. Turner. Diagnosing Performance Overheads in the Xen Virtual Machine Environment. In the First ACM/USENIX Conference on Virtual Execution Environments (VEE’05), 2005: 83-93.
[15] A. Menon, A. L. Cox, W. Zwaenepoel. Optimizing Network Virtualization in Xen. In Second ACM/USENIX Conference on Virtual Execution Environments (VEE’06), 2006: 105-114.
[16] Scott Rixner. Network Virtualization: Breaking the Performances of Barrier. ACM QUEUE, 2008(1): 38-52.
[17] www.xsigo.com/_downloads/Brochures/Xsigo_Solutions_Overview.pdf.
[18] Abramson D, Jackson J, et al. Intel Virtualization Technology for Directed I/O. www.intel.com/technology /vt/.
[19] AMD Inc. AMD I/O Virtualization Technology (IOMMU) Specification Revision 1.20. www.amd.com/us-en/assets.
[20] Keith Adams, Ole Agesen. A Comparison of Software and Hardware Techniques for X86 Virtualization. ASPLOS, 2006: 15-21.
[21] Sugerman J, Venkitachalam G, et al. Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual Machine. Proceedings of the USENIX, 2001: 1-14.
[22]董耀祖,周正伟.基于X86架构的系统虚拟机技术与应用.计算机工程, 2006,32(13): 1-13.
[23] Jeanna N. Matthews, Eli M. Dow, et al. Running Xen: A Hands-On Guide to the Art of Virtualization. Prentice Hall Inc., 2008.
[24] Janick Bergeron著,张春,陈凯新等译.编写测试平台: HDL模型的功能验证(第2版).电子工业出版社, 2006.
[25]孙世昶.虚拟化技术研究及PCI设备直接访问的实现.硕士学位论文,大连理工大学, 2006.
[26] Sun Microsystems Inc. UltraSPARC T2 Supplement to the Ultra SPARC Architecture 2007. www.sun.com.
[27] Vineet Chadha, Ramesh Illikkal, et al. I/O Processing in a Virtualizde Platform: A Simulation-Driven Approach. VEE, 2007: 68-75.
[28] Robert. P. Goldberg. Survey of virtual machine research. IEEE Computer Magazine, 1974, 6: 34–45.
[29] Keir Fraser, Steven Hand, et al. Safe Hardware Access with the Xen Virtual Machine Monitor. In 1st Workshop on OASIS, 2004: 35-43.
[30] Himanshu Raj and Karsten Schwan. High Performance and Scalable I/O Virtualization via Self-Virtualized Devices. In Proc. Of HPDC’07.
[31] PCI-SIG. PCI Express base specification 1.0a, 2002. www.pcisig.com/specifications.
[32] C. A. Waldspurger. Memory resource management in VMware ESX server. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation, 2002: 181–194.
[33] Jiangyong Zhang, Anand S., et al. An Interposed 2-Level I/O Scheduling Framework for Performance Virtualization. www.cse.psu.edu/~anand/csl/papers/metrics05b.pdf.
[34] Seetharami R. Seelam, Patricia J. Teller. Virtual I/O Scheduler: A Scheduler of Schedulers for Performance Virtualization. VEE, 2007: 109-117.
[35] L. Youseff, R. Wolski, B. Gorda, and C. Krintz. Evaluating the Performance Impactof Xen on MPI and Process Execution for HPC Systems. In Proc. Of International Workshop on Virtualization Technologies in Distributed Computing (VTDC), 2006.
[36] J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In Proc. of USENIX ATC, 2006.
[37] Intel Corp. Intel IXP2400 Network Processor: Hardware Reference Manual. www.intel.com/design/network/products.
[38] MindShare.Inc, Ravi Budruck, et al. PCI Express System Architecture. Addison Wesley, 2003.
[39] P. Apparao, R. Iyer, et al. Characterization and Analysis of a Server Consolidation Benchmark. VEE, 2008: 278-276.
[40] S. Govindan, A. R. Nath, A. Das. Xen and Co.: Comminuation-aware CPU Sheduling for consolidated Xen-based Hosting Platform. VEE, 2007: 55-76.
[41]黄歆媚,雷航. X86平台上高效高安全性的设备虚拟化模型.计算机应用, 2008, 28(8) :37-40.
[42] A. B. Nagarajan, F. Mueller, et al. Proactive Fault Tolerance for HPC with Xen Virtualization. ICS, 2007: 23-33.
[43] H. Chen, R. Chen, et al. Live Updating Operating Systems Using Virtualization. VEE, 2006: 35-45.
[44] HP Corp. HP Integrity VM Accelerated Virtual I/O (AVIO) Overview. http://www.docs.hp.com/en/vse.html.
[45] J. N. Mattews, W. Hu, M. Hapuarachchi, et al. Quantifying the Performance Isolation Properties of Virtualization System. ExpCS, 2007: 119-126.
[2] Microsoft Corp, AMD Inc. Virtual Server 2005 R2 SP1 and Next Generation AMD Opteron? Processors Increase the Business Value of Virtualization. www.microsoft.com/korea/virtualization/businessvalue.mspx.
[3] VMware Inc. Virtualization Overview. www.vmware.com/pdf/virtualization.pdf.
[4] AMD Inc. AMD64 Virtualization Codenamed“Pacifica”Technology: Secure Virtual Machine Architecture Reference Manual. www.amd.com/us-en/assets.
[5] Enterprise Management Associates. Virtualization: Exposing the Intangible Enterprise. Enterprise Management, 2006. www.infoworld.com/event/virtualization.
[6] Sun Microsystems. Virtualization Strategy: From Servers to Services. http://br.sun.com/sunnews/events/2006/virtual/pdfs.
[7] W.J.Armstrong, R.L.Arndt, et al. Advanced Virtualization Capabilities of Power5 Systems, IBM Journal. RES & DEV, 2005, 49: 523-532.
[8] W.J.Armstrong, R.L.Arndt, et al. IBM POWER6 Partition Mobility: Moving virtual servers seamlessly between physical systems. IBM Journal. RES & DEV, 2006, 51: 757-762.
[9] John L.HENNESSY, David A. PATTERSON著,白跃彬译.计算机系统结构:量化研究方法(第四版).电子工业出版社, 2007.
[10] Robert Rose. Survey of System Virtualization Technology. 2004.2. http://citeseer.ist.psu.edu/720518.html.
[11] Susanta Nanda, Tzi-cker Chiueh. A survey on Virtualization Techonologies. http://www.ecsl.cs.sunysb.edu/tr/TR179.pdf.
[12] VMware Inc. Understanding Full Virtualization, Paravirtualization, and Hardware Assist. www.vmware.com/files/pdf/VMware_paravirtualization.pdf.
[13] Barham P, Dragovic B, Fraser K et al. Xen and the art of virtualization. Proceedings of the 19th ACM SOSP, 2003: 164–177.
[14] A. Menon, J. R. Santos, Y. Turner. Diagnosing Performance Overheads in the Xen Virtual Machine Environment. In the First ACM/USENIX Conference on Virtual Execution Environments (VEE’05), 2005: 83-93.
[15] A. Menon, A. L. Cox, W. Zwaenepoel. Optimizing Network Virtualization in Xen. In Second ACM/USENIX Conference on Virtual Execution Environments (VEE’06), 2006: 105-114.
[16] Scott Rixner. Network Virtualization: Breaking the Performances of Barrier. ACM QUEUE, 2008(1): 38-52.
[17] www.xsigo.com/_downloads/Brochures/Xsigo_Solutions_Overview.pdf.
[18] Abramson D, Jackson J, et al. Intel Virtualization Technology for Directed I/O. www.intel.com/technology /vt/.
[19] AMD Inc. AMD I/O Virtualization Technology (IOMMU) Specification Revision 1.20. www.amd.com/us-en/assets.
[20] Keith Adams, Ole Agesen. A Comparison of Software and Hardware Techniques for X86 Virtualization. ASPLOS, 2006: 15-21.
[21] Sugerman J, Venkitachalam G, et al. Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual Machine. Proceedings of the USENIX, 2001: 1-14.
[22]董耀祖,周正伟.基于X86架构的系统虚拟机技术与应用.计算机工程, 2006,32(13): 1-13.
[23] Jeanna N. Matthews, Eli M. Dow, et al. Running Xen: A Hands-On Guide to the Art of Virtualization. Prentice Hall Inc., 2008.
[24] Janick Bergeron著,张春,陈凯新等译.编写测试平台: HDL模型的功能验证(第2版).电子工业出版社, 2006.
[25]孙世昶.虚拟化技术研究及PCI设备直接访问的实现.硕士学位论文,大连理工大学, 2006.
[26] Sun Microsystems Inc. UltraSPARC T2 Supplement to the Ultra SPARC Architecture 2007. www.sun.com.
[27] Vineet Chadha, Ramesh Illikkal, et al. I/O Processing in a Virtualizde Platform: A Simulation-Driven Approach. VEE, 2007: 68-75.
[28] Robert. P. Goldberg. Survey of virtual machine research. IEEE Computer Magazine, 1974, 6: 34–45.
[29] Keir Fraser, Steven Hand, et al. Safe Hardware Access with the Xen Virtual Machine Monitor. In 1st Workshop on OASIS, 2004: 35-43.
[30] Himanshu Raj and Karsten Schwan. High Performance and Scalable I/O Virtualization via Self-Virtualized Devices. In Proc. Of HPDC’07.
[31] PCI-SIG. PCI Express base specification 1.0a, 2002. www.pcisig.com/specifications.
[32] C. A. Waldspurger. Memory resource management in VMware ESX server. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation, 2002: 181–194.
[33] Jiangyong Zhang, Anand S., et al. An Interposed 2-Level I/O Scheduling Framework for Performance Virtualization. www.cse.psu.edu/~anand/csl/papers/metrics05b.pdf.
[34] Seetharami R. Seelam, Patricia J. Teller. Virtual I/O Scheduler: A Scheduler of Schedulers for Performance Virtualization. VEE, 2007: 109-117.
[35] L. Youseff, R. Wolski, B. Gorda, and C. Krintz. Evaluating the Performance Impactof Xen on MPI and Process Execution for HPC Systems. In Proc. Of International Workshop on Virtualization Technologies in Distributed Computing (VTDC), 2006.
[36] J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In Proc. of USENIX ATC, 2006.
[37] Intel Corp. Intel IXP2400 Network Processor: Hardware Reference Manual. www.intel.com/design/network/products.
[38] MindShare.Inc, Ravi Budruck, et al. PCI Express System Architecture. Addison Wesley, 2003.
[39] P. Apparao, R. Iyer, et al. Characterization and Analysis of a Server Consolidation Benchmark. VEE, 2008: 278-276.
[40] S. Govindan, A. R. Nath, A. Das. Xen and Co.: Comminuation-aware CPU Sheduling for consolidated Xen-based Hosting Platform. VEE, 2007: 55-76.
[41]黄歆媚,雷航. X86平台上高效高安全性的设备虚拟化模型.计算机应用, 2008, 28(8) :37-40.
[42] A. B. Nagarajan, F. Mueller, et al. Proactive Fault Tolerance for HPC with Xen Virtualization. ICS, 2007: 23-33.
[43] H. Chen, R. Chen, et al. Live Updating Operating Systems Using Virtualization. VEE, 2006: 35-45.
[44] HP Corp. HP Integrity VM Accelerated Virtual I/O (AVIO) Overview. http://www.docs.hp.com/en/vse.html.
[45] J. N. Mattews, W. Hu, M. Hapuarachchi, et al. Quantifying the Performance Isolation Properties of Virtualization System. ExpCS, 2007: 119-126.