对象存储系统中数据管理策略研究
|
摘要
目前,在超级计算领域,对高I/O吞吐率、高并行性和高扩展性的存储系统要求日益迫切,而传统的存储系统已经难以满足应用需求。实际的需求推动着存储系统不断向前发展,一种新兴的存储模式,对象存储系统应运而生。
对象存储系统中文件系统是分布式文件系统。与传统分布式文件系统不同的是,对象存储将数据管理功能下移到智能化的对象存储设备上。对象存储设备负责管理磁盘空间,是对象存储系统中数据的载体,也是存储系统管理的核心。
单个对象存储设备性能的提升直接影响对象存储系统的性能。可以从硬件和软件两个方面改善其性能。硬件方面主要针对网络存储的特征,设计I/O吞吐率高、低功耗和可扩展性好的嵌入式对象存储设备控制器,它采用Xscale微架构处理器和交换式架构实现。交换式架构支持任意接口到其他接口的并行事务,可以解决传统I/O总线瓶颈,实现并行I/O。软件方面主要是在对象设备端开发基于扩展哈希的对象文件系统XOBFS(eXtensible hashing Object-based Storage File System),实现对象存储系统存储资源管理。
在XOBFS中,磁盘空间被划分成多个区域单元,区域中定长块分配策略与位图管理空闲块方式结合,使得系统在长期运行中不会产生外部碎片,性能不会退化;对象属性用扩展哈希管理,在哈希桶中采用相同对象属性相邻存放策略。属性管理的实现,为更好地组织与管理数据,以及改善数据共享做好准备。最后通过合成Trace对XOBFS进行仿真测试,测试结果表明XOBFS的I/O吞吐率比传统文件系统高,且不会随时间增长而老化。XOBFS为对象存储设备端提供了一种有效的数据管理方法。
With the increasing demand of high data throughput, highly parallel data transfer and highly scalable storage system, especially in the field of super computing, the traditional storage system cannot meet the requirements any longer. With the driven needs of applications, storage system must have changes. A new storage model, Object-based Storage System, OBS for short, comes up. The file system used in the OBS, is a distributed file system. The storage management model down to the intelligent devices, which is different from the traditional one. The Object-based Storage Device, OSD for short, which is the core part of the OBS, carries the data in the OBS and also takes charge of the management of space in the physical disc.
The performance of the single OSD can be improved via two methods. The first one is to design a new Object-based Storage Device Controller, OSC for short. OSC is based on the switch fabric architecture. And two CPUs with Xscale architecture are included in OSC. The switch fabric breaks through the bottle-neck of I/O system, and implements the parallel I/O access in the hardware level. The second one is the management of disc space. XOBFS, which stands for eXtensible hashing Object-based Storage File System, uses the fixed-block allocation policy and manages the free space with the bitmap method. And XOBFS organizes the metadata in extensible hashing structure. All of this cannot make fragmentation after long term running. The performance does not degenerate as the file system ages.
By collecting blocks of same size into regions, XOBFS can avoid the system fragmentation as the system ages. However, on the worst condition, there would be a waste of about 10 percent of total disc space. For organizing and managing data effectively, the object attributes are included in XOBFS. Compared with the traditional file systems, the throughput of XOBFS is the best one. XOBFS provides an effective method for storage management in OSD.
对象存储系统中文件系统是分布式文件系统。与传统分布式文件系统不同的是,对象存储将数据管理功能下移到智能化的对象存储设备上。对象存储设备负责管理磁盘空间,是对象存储系统中数据的载体,也是存储系统管理的核心。
单个对象存储设备性能的提升直接影响对象存储系统的性能。可以从硬件和软件两个方面改善其性能。硬件方面主要针对网络存储的特征,设计I/O吞吐率高、低功耗和可扩展性好的嵌入式对象存储设备控制器,它采用Xscale微架构处理器和交换式架构实现。交换式架构支持任意接口到其他接口的并行事务,可以解决传统I/O总线瓶颈,实现并行I/O。软件方面主要是在对象设备端开发基于扩展哈希的对象文件系统XOBFS(eXtensible hashing Object-based Storage File System),实现对象存储系统存储资源管理。
在XOBFS中,磁盘空间被划分成多个区域单元,区域中定长块分配策略与位图管理空闲块方式结合,使得系统在长期运行中不会产生外部碎片,性能不会退化;对象属性用扩展哈希管理,在哈希桶中采用相同对象属性相邻存放策略。属性管理的实现,为更好地组织与管理数据,以及改善数据共享做好准备。最后通过合成Trace对XOBFS进行仿真测试,测试结果表明XOBFS的I/O吞吐率比传统文件系统高,且不会随时间增长而老化。XOBFS为对象存储设备端提供了一种有效的数据管理方法。
With the increasing demand of high data throughput, highly parallel data transfer and highly scalable storage system, especially in the field of super computing, the traditional storage system cannot meet the requirements any longer. With the driven needs of applications, storage system must have changes. A new storage model, Object-based Storage System, OBS for short, comes up. The file system used in the OBS, is a distributed file system. The storage management model down to the intelligent devices, which is different from the traditional one. The Object-based Storage Device, OSD for short, which is the core part of the OBS, carries the data in the OBS and also takes charge of the management of space in the physical disc.
The performance of the single OSD can be improved via two methods. The first one is to design a new Object-based Storage Device Controller, OSC for short. OSC is based on the switch fabric architecture. And two CPUs with Xscale architecture are included in OSC. The switch fabric breaks through the bottle-neck of I/O system, and implements the parallel I/O access in the hardware level. The second one is the management of disc space. XOBFS, which stands for eXtensible hashing Object-based Storage File System, uses the fixed-block allocation policy and manages the free space with the bitmap method. And XOBFS organizes the metadata in extensible hashing structure. All of this cannot make fragmentation after long term running. The performance does not degenerate as the file system ages.
By collecting blocks of same size into regions, XOBFS can avoid the system fragmentation as the system ages. However, on the worst condition, there would be a waste of about 10 percent of total disc space. For organizing and managing data effectively, the object attributes are included in XOBFS. Compared with the traditional file systems, the throughput of XOBFS is the best one. XOBFS provides an effective method for storage management in OSD.
引文
[1]王晓倩,崔宾. GriPhyN网格物理网络. http://grid.hust.edu.cn/article_59.html
[2]歌盟科技.国际巨型机系统发展状况. http://www.gemen.com.cn/supercomputer /itl_hpc.htm
[3] Mesnier M, Ganger G. R, Riedel E. Object-based storage. Communications Magazine, IEEE, 2003, 41(8): 84～90
[4] Garth A Gibson, David F Nagle, William Courtright, et al. NASD Scalable Storage Systems. In: USENIX 1999, Linux Workshop. Monterey, CA, America: USENIX, 1999. 1～6
[5] John B Lohmeyer, George O Penokie, Paul D Aloisi, et al. Information technology-SCSI Object-Based Storage Device Commands (OSD). INCITS T10 Working Draft. http://www.t10.org/drafts.htm. 2005. 16～121
[6] Yingping Lu, David H C Du, Tom Ruwart. Qos Provisioning Framework for an OSD-based Storage System. In: The 22nd IEEE/13th NASA Goddard Conference on Mass Storage System Technologies (MSST’05). American: IEEE Computer Society, 2005. 135～139
[7] Panasas, Inc. Activescale Storage Cluster. Panasas White Paper. http://www.panasas.com. 2004. 2～8
[8] Du D H C. Intelligent Storage for Information Retrieval. In: Next Generation Web Services Practices. NWeSP 2005. International Conference on. 2005, 8: 214～220
[9] Peter J Braam, Cluster File Systems Inc. The Lustre Storage Architecture. Cluster File Systems, Inc Whiter Paper. http://www.clusterfs.com. 2003. 14～28
[10] G. A Gibson, D P Nagle, K Amiri, et al. Filesystems for Network-attached Secure Disks. Technical Report CMU-CS, 1997, March: 1～18
[11] J Menon, D A Pease, R Rees L. et al. IBM Storage Tank-A heterogeneous scalable SAN file system. IBM SYSTEMS JOURNAL, 2003, 42(2):250～261
[12] J Satran, A Teperman. Object Store Based SAN File Systems. IBM SYSTEMS JOURNAL, 2003, 47(1):25～29
[13] O. Rodeh, A. Teperman. zFS - a scalable distributed file system using object disks. In: Proceedings of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies(MSST 2003). 2003. 207~218
[14] Wang Feng, Brandt Scott A., Miller Ethan L., et al. OBFS: A File System for Object-based Storage Devices. In: Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST2004). 2004. 101~118
[15]覃灵军.基于对象的主动存储关键技术研究: [博士学位论文].华中科技大学图书馆. 2006
[16] Cluster File Systems Inc. Lustre: A Scalable, High-performance File System. Cluster File Systems, Inc Whiter Paper. Http://www.clusterfs.com. 2002. 1～12
[17] J Menon, D A Pease, R Rees L. et al. IBM Storage Tank-A heterogeneous scalable SAN file system. IBM SYSTEMS JOURNAL, 2003, 42(2):250～261
[18] J Satran, A Teperman. Object Store Based SAN File Systems. IBM SYSTEMS JOURNAL, 2003, 47(1):25～29
[19] Hong Tang, Aziz Gulbeden, Jingyu Zhou, et al. The Panasas Activescale Storage Cluster-Delivering Scalable High Bandwidth Storage. In: The 2004 ACM/IEEE conference of Supercomputing. America: IEEE Computer Society, 2004. 53～58
[20] M. Factor, K. Meth, D. Naor, et al. Object Storage: The Future Building Block for Storage Systems. In: Proceedings of the 2nd International IEEE Symposium on Mass Storage Systems and Technologies. 2005. 119~123
[21] Feng Dan, Ling-Jun Qin, Ling-Fang Zeng, et al. A Scalable Object-Based Intelligent Storage Device. In: Proceedings of 2004 International Conference on Machine Learning and Cybernetics.Shanghai, 2004. 26～29
[22] A. Azagury, V. Dreizin, M. Factor, et al. Towards an object store. In: Proceedings of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies(MSST2003). 2003. 165~176
[23]张葱仔.对象存储控制器的硬件设计与实现: [硕士学位论文].华中科技大学图书馆. 2006
[24] Intel . Intel 80314 Companion Chip. http://www.intel.com/design/iio/80314.htm
[25] Intel. Intel 80200 Processor. http://www.intel.com/design/iio/80200.htm
[26] TimeSys Corporation. TimeSys Linux GPL: Performance Advanced for Embedded Systems. Timesys Corpporation White Paper. http://www.timesys.com. 2004. 1~6
[27] Nick Garnett, Jonathan Larmour, Andrew Lunn, et al. eCos Reference Manual. ecos.sourceware.org/docs-latest/ref/ecos-ref.html
[28] Red Hat. RedBoot User's Guide. http://ecos.sourceware.org/docs-latest/redboot/ redboot-guide.html
[29] Intel. IQ80315 Processor Evaluation Platform. http://www.intel.com/design/iio/ 80315.htm
[30]庞丽萍.操作系统原理.第三版.武汉:华中科技大学出版社, 2000. 10~12
[31] Feng Wang. File System Workload Analysis For Large Scientific Computing Applications. In: Proceedings of 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2004), College Park, MD, April 2004. 67～83
[32] T. Y. Ts'o and S. Tweedie. Planned extensions to the Linux EXT2/EXT3 filesystem. In Proceedings of the Freenix Track: 2002 USENIX Annual Technical Conference, pages 235–244, Monterey, CA, June 2002. USENIX
[33] Fagin, R., J. Nievergelt, N. Pippenger, and R. Strong,“Extendible hashing: A fast access method. for dynamic files,”ACM Transaction on Database Systems, 1979;4(3), pp. 315-344.
[34] Steve Heller. Extensible hashing. Dr. Dobb's Journal, 1989, 14(11):66~70
[35] Stolting Brodal, D. Demaine, Ian Munro. Fast allocation and deallocation with an improved buddy system. Acta Informatica, 2005, 41(4):273~291
[36] Jiri Schindler, John Linwood Griffin, Christopher R. Lumb. Track-Aligned Extents : Matching Access Patterns to Disk Drive Characteristics. In: Proceedings of the 2002 Conference on File and Storage Technologies. 2002. 259~274
[37] Mesnier M, Ganger G R, Riedel E. Object-based storage: pushing more functionality into storage. Potentials, IEEE, 2005, 24(2): 31～34
[38]谢黎明,冯丹,覃灵军.对象存储系统中属性管理方法研究与实现.计算机研究与发展, 2007, 44(增刊I):115~121
[39] E. J. O’Neil, P. E. O’Neil, and G. Weikum. The LRU-K page replacement algorithm for database buffering. In Proc. ACM SIGMOD’93: Int’l. Conf. on Mgmnt. of Data, Washington, DC, May. 1993. 297~306
[40] Otoo E, Olken F, Shoshani A. Disk cache replacement algorithm for storage resource managers in data grids. In: Proc. of the IEEE/ACM SC 2002 Conf. onSupercomputing. Los Alamitos: IEEE Computer Society, 2002. 1-15.
[41]龚玮.对象存储文件系统的设计与实现: [硕士学位论文].华中科技大学图书馆. 2006
[42] Hans Reiser. RKernel korner: trees in the Reiser4 filesystem, Part I. Linux Journal, 2002, 2002(104):8~8
[43] Hans Reiser. Reiser4, Part II : designing trees that cache well. Linux Journal, 2003, 2003(109):4~4
[44] Steve Best, David Gordon, Ibrahim Haddad. Kernel korner: IBM's journaled filesystem. Linux Journal, 2003, 2003(105):9~18
[2]歌盟科技.国际巨型机系统发展状况. http://www.gemen.com.cn/supercomputer /itl_hpc.htm
[3] Mesnier M, Ganger G. R, Riedel E. Object-based storage. Communications Magazine, IEEE, 2003, 41(8): 84～90
[4] Garth A Gibson, David F Nagle, William Courtright, et al. NASD Scalable Storage Systems. In: USENIX 1999, Linux Workshop. Monterey, CA, America: USENIX, 1999. 1～6
[5] John B Lohmeyer, George O Penokie, Paul D Aloisi, et al. Information technology-SCSI Object-Based Storage Device Commands (OSD). INCITS T10 Working Draft. http://www.t10.org/drafts.htm. 2005. 16～121
[6] Yingping Lu, David H C Du, Tom Ruwart. Qos Provisioning Framework for an OSD-based Storage System. In: The 22nd IEEE/13th NASA Goddard Conference on Mass Storage System Technologies (MSST’05). American: IEEE Computer Society, 2005. 135～139
[7] Panasas, Inc. Activescale Storage Cluster. Panasas White Paper. http://www.panasas.com. 2004. 2～8
[8] Du D H C. Intelligent Storage for Information Retrieval. In: Next Generation Web Services Practices. NWeSP 2005. International Conference on. 2005, 8: 214～220
[9] Peter J Braam, Cluster File Systems Inc. The Lustre Storage Architecture. Cluster File Systems, Inc Whiter Paper. http://www.clusterfs.com. 2003. 14～28
[10] G. A Gibson, D P Nagle, K Amiri, et al. Filesystems for Network-attached Secure Disks. Technical Report CMU-CS, 1997, March: 1～18
[11] J Menon, D A Pease, R Rees L. et al. IBM Storage Tank-A heterogeneous scalable SAN file system. IBM SYSTEMS JOURNAL, 2003, 42(2):250～261
[12] J Satran, A Teperman. Object Store Based SAN File Systems. IBM SYSTEMS JOURNAL, 2003, 47(1):25～29
[13] O. Rodeh, A. Teperman. zFS - a scalable distributed file system using object disks. In: Proceedings of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies(MSST 2003). 2003. 207~218
[14] Wang Feng, Brandt Scott A., Miller Ethan L., et al. OBFS: A File System for Object-based Storage Devices. In: Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST2004). 2004. 101~118
[15]覃灵军.基于对象的主动存储关键技术研究: [博士学位论文].华中科技大学图书馆. 2006
[16] Cluster File Systems Inc. Lustre: A Scalable, High-performance File System. Cluster File Systems, Inc Whiter Paper. Http://www.clusterfs.com. 2002. 1～12
[17] J Menon, D A Pease, R Rees L. et al. IBM Storage Tank-A heterogeneous scalable SAN file system. IBM SYSTEMS JOURNAL, 2003, 42(2):250～261
[18] J Satran, A Teperman. Object Store Based SAN File Systems. IBM SYSTEMS JOURNAL, 2003, 47(1):25～29
[19] Hong Tang, Aziz Gulbeden, Jingyu Zhou, et al. The Panasas Activescale Storage Cluster-Delivering Scalable High Bandwidth Storage. In: The 2004 ACM/IEEE conference of Supercomputing. America: IEEE Computer Society, 2004. 53～58
[20] M. Factor, K. Meth, D. Naor, et al. Object Storage: The Future Building Block for Storage Systems. In: Proceedings of the 2nd International IEEE Symposium on Mass Storage Systems and Technologies. 2005. 119~123
[21] Feng Dan, Ling-Jun Qin, Ling-Fang Zeng, et al. A Scalable Object-Based Intelligent Storage Device. In: Proceedings of 2004 International Conference on Machine Learning and Cybernetics.Shanghai, 2004. 26～29
[22] A. Azagury, V. Dreizin, M. Factor, et al. Towards an object store. In: Proceedings of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies(MSST2003). 2003. 165~176
[23]张葱仔.对象存储控制器的硬件设计与实现: [硕士学位论文].华中科技大学图书馆. 2006
[24] Intel . Intel 80314 Companion Chip. http://www.intel.com/design/iio/80314.htm
[25] Intel. Intel 80200 Processor. http://www.intel.com/design/iio/80200.htm
[26] TimeSys Corporation. TimeSys Linux GPL: Performance Advanced for Embedded Systems. Timesys Corpporation White Paper. http://www.timesys.com. 2004. 1~6
[27] Nick Garnett, Jonathan Larmour, Andrew Lunn, et al. eCos Reference Manual. ecos.sourceware.org/docs-latest/ref/ecos-ref.html
[28] Red Hat. RedBoot User's Guide. http://ecos.sourceware.org/docs-latest/redboot/ redboot-guide.html
[29] Intel. IQ80315 Processor Evaluation Platform. http://www.intel.com/design/iio/ 80315.htm
[30]庞丽萍.操作系统原理.第三版.武汉:华中科技大学出版社, 2000. 10~12
[31] Feng Wang. File System Workload Analysis For Large Scientific Computing Applications. In: Proceedings of 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2004), College Park, MD, April 2004. 67～83
[32] T. Y. Ts'o and S. Tweedie. Planned extensions to the Linux EXT2/EXT3 filesystem. In Proceedings of the Freenix Track: 2002 USENIX Annual Technical Conference, pages 235–244, Monterey, CA, June 2002. USENIX
[33] Fagin, R., J. Nievergelt, N. Pippenger, and R. Strong,“Extendible hashing: A fast access method. for dynamic files,”ACM Transaction on Database Systems, 1979;4(3), pp. 315-344.
[34] Steve Heller. Extensible hashing. Dr. Dobb's Journal, 1989, 14(11):66~70
[35] Stolting Brodal, D. Demaine, Ian Munro. Fast allocation and deallocation with an improved buddy system. Acta Informatica, 2005, 41(4):273~291
[36] Jiri Schindler, John Linwood Griffin, Christopher R. Lumb. Track-Aligned Extents : Matching Access Patterns to Disk Drive Characteristics. In: Proceedings of the 2002 Conference on File and Storage Technologies. 2002. 259~274
[37] Mesnier M, Ganger G R, Riedel E. Object-based storage: pushing more functionality into storage. Potentials, IEEE, 2005, 24(2): 31～34
[38]谢黎明,冯丹,覃灵军.对象存储系统中属性管理方法研究与实现.计算机研究与发展, 2007, 44(增刊I):115~121
[39] E. J. O’Neil, P. E. O’Neil, and G. Weikum. The LRU-K page replacement algorithm for database buffering. In Proc. ACM SIGMOD’93: Int’l. Conf. on Mgmnt. of Data, Washington, DC, May. 1993. 297~306
[40] Otoo E, Olken F, Shoshani A. Disk cache replacement algorithm for storage resource managers in data grids. In: Proc. of the IEEE/ACM SC 2002 Conf. onSupercomputing. Los Alamitos: IEEE Computer Society, 2002. 1-15.
[41]龚玮.对象存储文件系统的设计与实现: [硕士学位论文].华中科技大学图书馆. 2006
[42] Hans Reiser. RKernel korner: trees in the Reiser4 filesystem, Part I. Linux Journal, 2002, 2002(104):8~8
[43] Hans Reiser. Reiser4, Part II : designing trees that cache well. Linux Journal, 2003, 2003(109):4~4
[44] Steve Best, David Gordon, Ibrahim Haddad. Kernel korner: IBM's journaled filesystem. Linux Journal, 2003, 2003(105):9~18