对象存储原型系统设计及相关实现
|
摘要
数字以强大的信息表达能力以及单一的处理、传输和存储方式,融合了整个信息技术。半个世纪以来,作为数字信息载体的存储技术得到飞速发展。不断增长的存储需求和管理成本催生了基于对象的存储技术,而“对象”也有望成为下一代存储技术的标准接口。
在分析当前流行的网络存储体系结构及存储协议的基础上,对基于对象存储技术以及T10的SCSI OSD(Object-Based Storage Device Commands)标准作了深入研究。“对象”是传统块接口和文件接口的折中,基于对象存储系统在I/O性能、跨平台、可扩展性以及安全性等方面都表现不错。
实现了一个符合T10 SCSI OSD标准的对象存储原型系统,包括对象存储设备和客户端。客户端的SCSI对象设备驱动是一个Linux SCSI上层驱动,基于Linux块设备子系统实现,用来管理所有检测到的OSD设备。iSCSI启动设备是Linux SCSI协议栈的底层驱动,为客户端提供通过IP网络访问iSCSI目标设备的iSCSI通路。iSCSI目标设备实现iSCSI传输协议的Target部分。对象存储服务模块负责管理物理存储介质和处理OSD命令,以上模块均在Linux内核空间实现。
测试并分析了基于对象存储原型系统的性能,得出的结论是:通过在对象存储原型系统引入聚合读/写机制可以大大提升系统的I/O带宽。
By virtue of its powerful capability in information presentation and uniform way to process, transmit and store, digit has unified information technology. Half a century has witnessed the development of storage technology. Ever increasing storage demands and management costs has given birth to Object-Based Storage technology and“object”is on the verge of becoming the next standard storage interface.
On the basis of the analysis of the current prevailing network storage architecture and the storage protocols, Object-Based Storage technology and T10 SCSI OSD standard are studied insightfully. Objects can be regarded as convergence of files and blocks. Object-Based Storage System proved advantageous in performance, cross-platform, scalability and security.
An object-based storage prototype system based on T10 SCSI OSD standard was designed, which involves object-based storage device and client. Client’s SCSI object driver is realized as a SCSI upper level device driver, whose main function is to manage all detected OSD devices; iSCSI initiator is a SCSI low level driver providing iSCSI transport to access remote iSCSI targets over IP networks; iSCSI target implements the target side of iSCSI transport protocol; Object-based storage server manages the physical storage media and processes OSD commands. All are realized as Linux kernel mudules.
In the end, according to the performance test and analysis, we conclude that by introducing mechanism of aggregated read/wirte I/O throughput of object-based storage prototype system is drastically improved.
在分析当前流行的网络存储体系结构及存储协议的基础上,对基于对象存储技术以及T10的SCSI OSD(Object-Based Storage Device Commands)标准作了深入研究。“对象”是传统块接口和文件接口的折中,基于对象存储系统在I/O性能、跨平台、可扩展性以及安全性等方面都表现不错。
实现了一个符合T10 SCSI OSD标准的对象存储原型系统,包括对象存储设备和客户端。客户端的SCSI对象设备驱动是一个Linux SCSI上层驱动,基于Linux块设备子系统实现,用来管理所有检测到的OSD设备。iSCSI启动设备是Linux SCSI协议栈的底层驱动,为客户端提供通过IP网络访问iSCSI目标设备的iSCSI通路。iSCSI目标设备实现iSCSI传输协议的Target部分。对象存储服务模块负责管理物理存储介质和处理OSD命令,以上模块均在Linux内核空间实现。
测试并分析了基于对象存储原型系统的性能,得出的结论是:通过在对象存储原型系统引入聚合读/写机制可以大大提升系统的I/O带宽。
By virtue of its powerful capability in information presentation and uniform way to process, transmit and store, digit has unified information technology. Half a century has witnessed the development of storage technology. Ever increasing storage demands and management costs has given birth to Object-Based Storage technology and“object”is on the verge of becoming the next standard storage interface.
On the basis of the analysis of the current prevailing network storage architecture and the storage protocols, Object-Based Storage technology and T10 SCSI OSD standard are studied insightfully. Objects can be regarded as convergence of files and blocks. Object-Based Storage System proved advantageous in performance, cross-platform, scalability and security.
An object-based storage prototype system based on T10 SCSI OSD standard was designed, which involves object-based storage device and client. Client’s SCSI object driver is realized as a SCSI upper level device driver, whose main function is to manage all detected OSD devices; iSCSI initiator is a SCSI low level driver providing iSCSI transport to access remote iSCSI targets over IP networks; iSCSI target implements the target side of iSCSI transport protocol; Object-based storage server manages the physical storage media and processes OSD commands. All are realized as Linux kernel mudules.
In the end, according to the performance test and analysis, we conclude that by introducing mechanism of aggregated read/wirte I/O throughput of object-based storage prototype system is drastically improved.
引文
[1] Jim Gray. “What Next? A Few Remaining Problems in Information Technology”. http://research.microsoft.com/~gray/talks/Gray_Turing_FCRC.pdf. 1998
[2] Intel cooperation. Object-based Storage-The Next Wave of Storage Technology and Devices. Intel White Paper. http://www.intel.com/labs/storage/osd.7-12. 2003
[3] 张江陵,冯丹. 海量信息存储. 第一版. 北京:科学出版社,2003. 111~124
[4] 郑纬民, 舒继武. 下一代分布式智能网络存储系统的发展趋势. 世界电信, 2004, 17(8): 16~19
[5] Mesnier M, Ganger G. R, Riedel E Object-based storage. Communications Magazine, IEEE, 2003, 41(8): 84~90
[6] 谭毓安, 余锋, 曹元大. 面向对象的网络存储技术. 高性能计算技术, 2003, 4(7): 10~16
[7] 冯丹,史伟,覃灵军. 基于对象文件系统的研究与实现. 华中科技大学学报. 已录用.
[8] T10 Technical Committee of INCITS. SCSI Architecture Model-3(SAM-3). Project T10/1561-D, Revision 14. Septemper 2004.
[9] T10 Technical Committee of INCITS. SCSI Object-Based Storage Device Commands-2(OSD-2). Project T10/1721-D, Revision 0. October 2004.
[10] Yingping Lu, David Du, and Tom Ruwart. QoS provisioning framework for an osd-based storage system. In: Proceedings of 13th NASA Goddard, 22nd IEEE Conference on Mass Storage Systems and Technologies. American: IEEE Computer Society, 2005. 135~139
[11] KleinOsowski K, Ruwart T, Lijia D J. Communicating Quality of Service Requirements to an Object-Based Storage Device. Mass Storage Systems and Technologies, 2005. In: Proceedings of 22nd IEEE/13th NASA Goddard Conference on. America: IEEE Computer Society, 2005. 224~231
[12] Michael Factor,David Nagle,Dalit Naor,et al. The OSD security protocol. In: Proceedings of 3rd International IEEE Security in Storage Workshop. American: IEEEComputer Society, 2005. 343~349
[13] Bogdan C Popescu, Maarten van Steen, Andrew S Tanenbaum. A Security Architecture for Object-Based Distributed Systems. In: Proceedings of 18th Annual Computer Security Applications Conference (ACSAC '02). America: IEEE Computer Society, 2002.161~164
[14] Kher, Yongdae Kim. Decentralized Authentication Mechanisms for Object-based Storage Devices. Security in Storage Workshop, 2003. SISW '03. In: Proceedings of the Second IEEE International. America: IEEE Computer Society, 2003. 1~3
[15] Dan Feng, Lingfang Zeng, Fang Wang, et al. Adaptive policy-trigger mechanism for OBSS. In: Proceedings of 19th International Conference on Machine Learning and Cybernetics. China: IEEE Computer Society, 2005. 591~595
[16] Dan Feng, Lingjun Qin, Lingfang Zeng, et al. A Scalable Object-Based Intelligent Storage Device. In: Proceedings of 2004 International Conference on Machine Learning and Cybernetics. 2004. 26~29
[17] Lingfang Zeng, Dan Feng, Lingjun Qin. SOSS: Smart Object-Based Storage System. In: Proceedings of the Third International Conference on Machine Learning and Cybernetics. America: IEEE Computer Society, 2004. 3263~3266
[18] Mesnier M, Ganger G. R, Riedel E. Object-based storage: pushing more functionality into storage. Potentials, IEEE, 2005, 24(2): 31~34
[19] Gibson G A, Nagle D, Amiri K, et al. A Cost-Effective, High-Bandwidth Storage Architecture. In: Proceedings of the 8th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 1998.
[20] Gibson G A, Nagle D, Amiri K, et al. Filesystems for Network-attached Secure Disks. Technical Report CMU-CS, March 1999: 1~18
[21] Gibson G A, Nagle D, William Courtright, et al. NASD Scalable Storage Systems. USENIX 1999, Linux Workshop. Monterey, CA, America: USENIX, 1999. 1~6
[22] Peter J. Braam, Cluster File Systems Inc. The Lustre Storage Architecture. Cluster File Systems, Inc Whiter Paper. http://www.clusterfs.com. 2004. 14~28
[23] Cluster File Systems Inc. Lustre: A Scalable, High-performance File System. Cluster File Systems, Inc Whiter Paper. http://www.clusterfs.com. 2002. 1~12
[24] TOP500 LIST FOR NOVEMBER 2005. http://www.top500.org/lists/2005/11/basic.2005.
[25] Panasas Inc. Panasas ActiveScale Storage Cluster. http://www.panasas.com. 2004. 4~18
[26] Hong Tang, Aziz Gulbeden, Jingyu Zhou, et al. The Panasas ActiveScale Storage Cluster-Delivering Scalable High Bandwidth Storage. In: Proceedings of the 2004 ACM/IEEE conference of Supercomputing. America: IEEE Computer Society, 2004. 53~58
[27] J. Menon, D. A. Pease, L. Duyanovich, et al. IBM Storage Tank—A heterogeneous scalable SAN file system. IBM SYSTEM JOURNAL, 42(2), 2003: 250~261
[28] J. Satran, A. Teperman. Object Store Based SAN File Systems. IBM SYSTEMS JOURNAL, 2003, 47(1): 25~29
[29] EMC Centera, content addressed storage, product description. http://www.emc.com/pdf/products/centera/centeraguide.pdf. 2002.
[30] 张葱仔,冯丹,覃灵军. 对象存储控制器的硬件设计与仿真. 计算机工程与科学. 已录用.
[31] TimeSys Corporation. TimeSys Linux GPL: Performance Advanced for Embedded Systems. Timesys Corpporation White Paper. http://www.timesys.com. 2004. 1~6
[32] 李善平,陈文智. Linux 内核指导. 第一版. 浙江:浙江大学出版社,2002. 399~470
[33] Jonathan Corbet, Alessandro Rubini, Greg Kroah-hartman. Linux Device Drivers. 3rd edition. American: O’Reilly, 2005. 458~482
[34] IETF. Draft-ietf-ips-iSCSI-20. 2003
[35] 王德强,谢立. 基于规则的文件系统自主存取控制研究. 计算机应用与软件,2004, 21(5): 70~76
[36] Wei-Guang Teng, Cheng-Yue Chang, Ming-Syan Chen, et al. Integrating Web Caching and Web Prefetching in Client-Side Proxies. IEEE Transactions on Parallel and Distributed System, 2005, 16(5): 444~455
[37] Philip A.Bernstein, Shankar Pal. Context-Based Prefetch for Implementing Objects on Relations. In: Proceedings of the 25th VLDB conference Edinburgh. America, Scotland: IEEE Computer Society, 1999. 102~107
[38] Y. Solihin, J. Lee, J. Torrellas. Using a User-Level Memory Thread for CorrelationPrefetching. In: Proceedings of 29th Ann. International Symposium of Computer Architecture (ISCA 02). America: IEEE Press, 2002. 171~182
[39] IOzone file system benchmark. http://www.iozone.org. 2006
[40] 田颖,许鲁. 分布式文件系统中的负载平衡. 计算机工程,2003,19(19):41~44
[41] S. Brandt, L. Xue, E. Miller, et al. Efficient metadata managemnt in large distributed file systems. In: Proceedings of Twentieth IEEE/Eleventh NASA Goddard Conference on Mass Storage Systems and Technology. America: IEEE Computer Society, 2005. 224~231
[42] F. Wang, S. Brandt, E. Miller, et al. OBFS: a file system for object-based storage devices. In: Proceedings of 12th NASA Goddard, 21st IEEE Conference on Mass Storage Systems and Technologies. American: IEEE Computer Society, 2004. 134~156
[43] T. S. Rajesh Kumar, R. Govindara jan, C. P. Ravi Kumar. Optimal Code and Data Layout in Embedded System. In: Proceedings of the 16th International Conference on VLSI Design (VLSI’03). America: IEEE Computer Society, 2003. 1063~1069
[2] Intel cooperation. Object-based Storage-The Next Wave of Storage Technology and Devices. Intel White Paper. http://www.intel.com/labs/storage/osd.7-12. 2003
[3] 张江陵,冯丹. 海量信息存储. 第一版. 北京:科学出版社,2003. 111~124
[4] 郑纬民, 舒继武. 下一代分布式智能网络存储系统的发展趋势. 世界电信, 2004, 17(8): 16~19
[5] Mesnier M, Ganger G. R, Riedel E Object-based storage. Communications Magazine, IEEE, 2003, 41(8): 84~90
[6] 谭毓安, 余锋, 曹元大. 面向对象的网络存储技术. 高性能计算技术, 2003, 4(7): 10~16
[7] 冯丹,史伟,覃灵军. 基于对象文件系统的研究与实现. 华中科技大学学报. 已录用.
[8] T10 Technical Committee of INCITS. SCSI Architecture Model-3(SAM-3). Project T10/1561-D, Revision 14. Septemper 2004.
[9] T10 Technical Committee of INCITS. SCSI Object-Based Storage Device Commands-2(OSD-2). Project T10/1721-D, Revision 0. October 2004.
[10] Yingping Lu, David Du, and Tom Ruwart. QoS provisioning framework for an osd-based storage system. In: Proceedings of 13th NASA Goddard, 22nd IEEE Conference on Mass Storage Systems and Technologies. American: IEEE Computer Society, 2005. 135~139
[11] KleinOsowski K, Ruwart T, Lijia D J. Communicating Quality of Service Requirements to an Object-Based Storage Device. Mass Storage Systems and Technologies, 2005. In: Proceedings of 22nd IEEE/13th NASA Goddard Conference on. America: IEEE Computer Society, 2005. 224~231
[12] Michael Factor,David Nagle,Dalit Naor,et al. The OSD security protocol. In: Proceedings of 3rd International IEEE Security in Storage Workshop. American: IEEEComputer Society, 2005. 343~349
[13] Bogdan C Popescu, Maarten van Steen, Andrew S Tanenbaum. A Security Architecture for Object-Based Distributed Systems. In: Proceedings of 18th Annual Computer Security Applications Conference (ACSAC '02). America: IEEE Computer Society, 2002.161~164
[14] Kher, Yongdae Kim. Decentralized Authentication Mechanisms for Object-based Storage Devices. Security in Storage Workshop, 2003. SISW '03. In: Proceedings of the Second IEEE International. America: IEEE Computer Society, 2003. 1~3
[15] Dan Feng, Lingfang Zeng, Fang Wang, et al. Adaptive policy-trigger mechanism for OBSS. In: Proceedings of 19th International Conference on Machine Learning and Cybernetics. China: IEEE Computer Society, 2005. 591~595
[16] Dan Feng, Lingjun Qin, Lingfang Zeng, et al. A Scalable Object-Based Intelligent Storage Device. In: Proceedings of 2004 International Conference on Machine Learning and Cybernetics. 2004. 26~29
[17] Lingfang Zeng, Dan Feng, Lingjun Qin. SOSS: Smart Object-Based Storage System. In: Proceedings of the Third International Conference on Machine Learning and Cybernetics. America: IEEE Computer Society, 2004. 3263~3266
[18] Mesnier M, Ganger G. R, Riedel E. Object-based storage: pushing more functionality into storage. Potentials, IEEE, 2005, 24(2): 31~34
[19] Gibson G A, Nagle D, Amiri K, et al. A Cost-Effective, High-Bandwidth Storage Architecture. In: Proceedings of the 8th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 1998.
[20] Gibson G A, Nagle D, Amiri K, et al. Filesystems for Network-attached Secure Disks. Technical Report CMU-CS, March 1999: 1~18
[21] Gibson G A, Nagle D, William Courtright, et al. NASD Scalable Storage Systems. USENIX 1999, Linux Workshop. Monterey, CA, America: USENIX, 1999. 1~6
[22] Peter J. Braam, Cluster File Systems Inc. The Lustre Storage Architecture. Cluster File Systems, Inc Whiter Paper. http://www.clusterfs.com. 2004. 14~28
[23] Cluster File Systems Inc. Lustre: A Scalable, High-performance File System. Cluster File Systems, Inc Whiter Paper. http://www.clusterfs.com. 2002. 1~12
[24] TOP500 LIST FOR NOVEMBER 2005. http://www.top500.org/lists/2005/11/basic.2005.
[25] Panasas Inc. Panasas ActiveScale Storage Cluster. http://www.panasas.com. 2004. 4~18
[26] Hong Tang, Aziz Gulbeden, Jingyu Zhou, et al. The Panasas ActiveScale Storage Cluster-Delivering Scalable High Bandwidth Storage. In: Proceedings of the 2004 ACM/IEEE conference of Supercomputing. America: IEEE Computer Society, 2004. 53~58
[27] J. Menon, D. A. Pease, L. Duyanovich, et al. IBM Storage Tank—A heterogeneous scalable SAN file system. IBM SYSTEM JOURNAL, 42(2), 2003: 250~261
[28] J. Satran, A. Teperman. Object Store Based SAN File Systems. IBM SYSTEMS JOURNAL, 2003, 47(1): 25~29
[29] EMC Centera, content addressed storage, product description. http://www.emc.com/pdf/products/centera/centeraguide.pdf. 2002.
[30] 张葱仔,冯丹,覃灵军. 对象存储控制器的硬件设计与仿真. 计算机工程与科学. 已录用.
[31] TimeSys Corporation. TimeSys Linux GPL: Performance Advanced for Embedded Systems. Timesys Corpporation White Paper. http://www.timesys.com. 2004. 1~6
[32] 李善平,陈文智. Linux 内核指导. 第一版. 浙江:浙江大学出版社,2002. 399~470
[33] Jonathan Corbet, Alessandro Rubini, Greg Kroah-hartman. Linux Device Drivers. 3rd edition. American: O’Reilly, 2005. 458~482
[34] IETF. Draft-ietf-ips-iSCSI-20. 2003
[35] 王德强,谢立. 基于规则的文件系统自主存取控制研究. 计算机应用与软件,2004, 21(5): 70~76
[36] Wei-Guang Teng, Cheng-Yue Chang, Ming-Syan Chen, et al. Integrating Web Caching and Web Prefetching in Client-Side Proxies. IEEE Transactions on Parallel and Distributed System, 2005, 16(5): 444~455
[37] Philip A.Bernstein, Shankar Pal. Context-Based Prefetch for Implementing Objects on Relations. In: Proceedings of the 25th VLDB conference Edinburgh. America, Scotland: IEEE Computer Society, 1999. 102~107
[38] Y. Solihin, J. Lee, J. Torrellas. Using a User-Level Memory Thread for CorrelationPrefetching. In: Proceedings of 29th Ann. International Symposium of Computer Architecture (ISCA 02). America: IEEE Press, 2002. 171~182
[39] IOzone file system benchmark. http://www.iozone.org. 2006
[40] 田颖,许鲁. 分布式文件系统中的负载平衡. 计算机工程,2003,19(19):41~44
[41] S. Brandt, L. Xue, E. Miller, et al. Efficient metadata managemnt in large distributed file systems. In: Proceedings of Twentieth IEEE/Eleventh NASA Goddard Conference on Mass Storage Systems and Technology. America: IEEE Computer Society, 2005. 224~231
[42] F. Wang, S. Brandt, E. Miller, et al. OBFS: a file system for object-based storage devices. In: Proceedings of 12th NASA Goddard, 21st IEEE Conference on Mass Storage Systems and Technologies. American: IEEE Computer Society, 2004. 134~156
[43] T. S. Rajesh Kumar, R. Govindara jan, C. P. Ravi Kumar. Optimal Code and Data Layout in Embedded System. In: Proceedings of the 16th International Conference on VLSI Design (VLSI’03). America: IEEE Computer Society, 2003. 1063~1069