面向移动设备的云存储统一接口研究与实现
摘要
随着经济的发展和科技的进步,智能手机、平板电脑等移动设备已经成为人们日常工作和生活中必不可少的组成部分。同时,由于接入互联网的带宽明显改善,以及各种云存储服务的兴起,移动设备可以方便地通过云存储服务接口,将数据存储到各种云存储系统中。
但另一方面,对于上层的应用程序来说,这些云存储服务接口往往各不相同,没有提供符合文件系统标准的接口,用户终端无法透明一致地访问远程存储服务,也不能在多个云存储系统间进行数据的共享和同步等操作。因此随着云存储服务的迅速发展,访问接口的差异化给用户开发和使用云存储服务带来越来越大的困难,人们对异构云存储服务接口规范化的需求也日益突出。
本文设计并实现了一个支持异构云存储的统一接口服务CSUI,通过对不同云存储接口的转换处理,为移动客户端提供虚拟接口,能够较好地满足各类移动设备通过网络对数据进行无缝存储、管理和使用的需求,也为云存储服务标准化提供了一种解决方案。论文的主要工作包括:
1、针对移动设备本身的存储和计算资源不足以及功耗受限的问题,设计并实现CSUI的系统架构,通过虚拟接口层与插件机制,为访问多种异构云存储服务提供统一接口,实现对多种存储服务的透明支持。
2、根据移动设备的特点,为减少CSUI系统的网络流量、降低访问延迟并提高系统的性能,设计了一组可动态加载的模块化优化方法,包括基于使用成本的LCU缓存替换算法等缓存管理策略、文件压缩与聚集的流量控制技术等。
3、为增强CSUI在主流移动设备上的实用性,针对Linux和Android平台分别设计并实现了相应的客户端程序。由于Android系统默认情况下并不包含FUSE模块,需要重新对Android的系统内核进行配置、编译等操作。
4、经过对原型系统的测试结果表明,CSUI有效地实现了对POSIX标准接口的支持,用户可以对云端数据执行各种常用的文件操作命令,与已有的解决方案相比,本方法能够基本保持系统性能不变,但明显简化了客户端的实现并降低对资源的使用要求。
With the development of economic and technology, mobile devices, such as intelligent mobile phone and tablet PC, have become essential components in people's daily life and work. At the same time, the bandwidth to access Internet is significantly improved. With the rapid growth of cloud storage services, mobile devices can easily store their data through the interface of cloud storage service in a variety of cloud storage systems.
On the other hand, these cloud storage services are different for the upper applications, and they do not provide consistent interface for the file system. User’s mobile devices can’t access remote storage services transparently and consistently, can’t be in more than one system for data sharing and synchrony either. Along with the rapid growth of cloud storage services, differences in the interface of the services affect the usage more than ever before. The demand of interface standardization on heterogeneous cloud storage service is also becoming more and more prominent.
This paper designs and implements CSUI, a unified interface service for cloud storage. CSUI can provide mobile devides with virtual interface by conversion processing for different cloud storage interface. The service meets the needs of mobile devices for data management and usage in seamless cloud storage, and also gives a new solution for the cloud storage service standardization. The main work is as follows:
Firstly, in order to solve the shortage of local storage and computing resources, and power constrained problem in mobile devices, we design and implement the CSUI system. By using the virtual interface layer and the plugin mechanism, CSUI provide a unified interface and transparently supports for heterogeneous cloud storage services.
Secondly, according to the characteristics of mobile equipment, we design a group of optimization methods that can be dynamically loaded to reduce network flow, decrease access delay and improve the performance of the system. These methods include LCU cache replacement algorithm, file compression technology and so on.
Thirdly, in order to enhance availability in mobile equipment, we design and implement the CSUI’s client program for Linux and Android operating system. Because the Android system doesn’t contain FUSE module by default, so we need to configure and rebuild the kernel of Android system.
Fourthly, we test our system and find that it can support the POSIX standard interface. Compared with the existing solution, CSUI can do as well as the others in the aspect of performance, further more, it simplifies the clients and reduces the requirements of resources.
但另一方面,对于上层的应用程序来说,这些云存储服务接口往往各不相同,没有提供符合文件系统标准的接口,用户终端无法透明一致地访问远程存储服务,也不能在多个云存储系统间进行数据的共享和同步等操作。因此随着云存储服务的迅速发展,访问接口的差异化给用户开发和使用云存储服务带来越来越大的困难,人们对异构云存储服务接口规范化的需求也日益突出。
本文设计并实现了一个支持异构云存储的统一接口服务CSUI,通过对不同云存储接口的转换处理,为移动客户端提供虚拟接口,能够较好地满足各类移动设备通过网络对数据进行无缝存储、管理和使用的需求,也为云存储服务标准化提供了一种解决方案。论文的主要工作包括:
1、针对移动设备本身的存储和计算资源不足以及功耗受限的问题,设计并实现CSUI的系统架构,通过虚拟接口层与插件机制,为访问多种异构云存储服务提供统一接口,实现对多种存储服务的透明支持。
2、根据移动设备的特点,为减少CSUI系统的网络流量、降低访问延迟并提高系统的性能,设计了一组可动态加载的模块化优化方法,包括基于使用成本的LCU缓存替换算法等缓存管理策略、文件压缩与聚集的流量控制技术等。
3、为增强CSUI在主流移动设备上的实用性,针对Linux和Android平台分别设计并实现了相应的客户端程序。由于Android系统默认情况下并不包含FUSE模块,需要重新对Android的系统内核进行配置、编译等操作。
4、经过对原型系统的测试结果表明,CSUI有效地实现了对POSIX标准接口的支持,用户可以对云端数据执行各种常用的文件操作命令,与已有的解决方案相比,本方法能够基本保持系统性能不变,但明显简化了客户端的实现并降低对资源的使用要求。
With the development of economic and technology, mobile devices, such as intelligent mobile phone and tablet PC, have become essential components in people's daily life and work. At the same time, the bandwidth to access Internet is significantly improved. With the rapid growth of cloud storage services, mobile devices can easily store their data through the interface of cloud storage service in a variety of cloud storage systems.
On the other hand, these cloud storage services are different for the upper applications, and they do not provide consistent interface for the file system. User’s mobile devices can’t access remote storage services transparently and consistently, can’t be in more than one system for data sharing and synchrony either. Along with the rapid growth of cloud storage services, differences in the interface of the services affect the usage more than ever before. The demand of interface standardization on heterogeneous cloud storage service is also becoming more and more prominent.
This paper designs and implements CSUI, a unified interface service for cloud storage. CSUI can provide mobile devides with virtual interface by conversion processing for different cloud storage interface. The service meets the needs of mobile devices for data management and usage in seamless cloud storage, and also gives a new solution for the cloud storage service standardization. The main work is as follows:
Firstly, in order to solve the shortage of local storage and computing resources, and power constrained problem in mobile devices, we design and implement the CSUI system. By using the virtual interface layer and the plugin mechanism, CSUI provide a unified interface and transparently supports for heterogeneous cloud storage services.
Secondly, according to the characteristics of mobile equipment, we design a group of optimization methods that can be dynamically loaded to reduce network flow, decrease access delay and improve the performance of the system. These methods include LCU cache replacement algorithm, file compression technology and so on.
Thirdly, in order to enhance availability in mobile equipment, we design and implement the CSUI’s client program for Linux and Android operating system. Because the Android system doesn’t contain FUSE module by default, so we need to configure and rebuild the kernel of Android system.
Fourthly, we test our system and find that it can support the POSIX standard interface. Compared with the existing solution, CSUI can do as well as the others in the aspect of performance, further more, it simplifies the clients and reduces the requirements of resources.
引文
[1] Milojicic D.Microsoft's Jim Gray on computing's breakthroughs,lessons,and future [J].Distributed Systems Online,IEEE.2004,5(1).4.1~4.8.
[2] Ansley Post, Juan Navarro, Petr Kuznetsov, Peter Druschel.Autonomous Storage Management for Personal Devices with PodBase [C].In USENIX Annual Technical Conference (USENIX ATC), 2011.
[3] Larry Ponemon.The Cost of a Lost Laptop [R]. Intel Corporation, Apr.2009.
[4] Abadi D. Data management in the cloud: Limitations and opportunities [C].Conference on Data Engineering. Shanghai: IEEEE, 2009:3-12.
[5]舒继武,薛巍,付长冬.网络存储系统与技术的现状与发展趋势[J].中国计算机学会通讯,2004.
[6] Foster I, Zhao Y, Raicu I, Lu S. Cloud Computing and Grid Computing 360-Degree Compared[C]. Grid Computing Environments Workshop(GCE), 2008.
[7] Armbrust M, Fox A, Griffith R, et al.Above the clouds: A Berkeley view of cloud computing [R]. Berkeley: EECS Department, 2009.
[8] Baidu.云手机[EB/OL]. http://baike.baidu.com/view/4692226.htm,2011.
[9]博思数据研究中心. 2011-2015年中国智能手机行业市场分析与投资前景研究报告[R].Technical report, 2011.
[10] H.Choi and H.Varian. Predicting the present with google trends [R]. Technical report, Google Inc, 2009.
[11] Palankar,M.R. and Iamnitchi,A. and Ripeanu,M. and Garfinkel,S. Amazon S3 for science grdi: a viable solution [C]. Proceedings of the 2008 international workshop on Data-aware distributed computing, 2008:55~64.
[12]金海,吴松,廖小飞,袁平鹏等.存储技术的研究发展现状与趋势[R].2009中国计算机科学技术发展报告,中国计算机学会文集,2010.
[13] Gallmeister, B.O. POSIX.4: Programming for the real world [M]. O’Reilly & Associate Inc. 1995.
[14] Google. Google Docs [EB/OL]. http://www.google.com/apps/intl/en/business/docs.html Oct, 2010.
[15] R.Jones. GmailFS:Gmail virtual file system [EB/OL]. http://richard.jones.name/google-hacks/gmail-filesystem/gmail-filesystem.html.
[16] Szeredi M. FUSE: Filesystem in userspace [EB/OL]. http://fuse.sourceforge.net. 2010.
[17]毛华坚.支持异构网络存储服务的YaFS文件系统研究与实现[D].长沙:国防科学技术大学研究生院,2009.
[18] Nokia. Symbian [EB/OL]. http://en.wikipedia.org/wiki/Symbian/.2010.
[19] Microsoft.Windows Mobile [EB/OL]. http://zh.wikipedia.org/wiki/Windows_Mobile/.2010.
[20] Apple.iOS [EB/OL].http://developer.apple.com/.2010.
[21] HP.Web OS [EB/OL].http://en.wikipedia.org/wiki/WebOS/.2011.
[22] Google.Android [EB/OL]. http://code.google.com/intl/zh-CN/android/.2011.
[23] Buyya, R. Chee Shin Yeo Venugopal, S. Market-Oriented Cloud Computing: Vision, Hype, and Reality for Delivering IT Services as Computing Utilities[C]. High Performance Computing and Communications(HPCC), 2008.
[24] Jiyi Wu, Lingdi Ping, et al.Cloud Storage as the Infrastructure of Cloud Computing [C]. Intelligent Computing and Cognitive Informatics (ICICCI), 2010:380~383.
[25] Microsoft.Windows Live skyDrive [EB/OL]. http://www.windowslive.cn/skydrive/.2010.
[26] Amazon.Cloud Drive [EB/OL]. https://www.amazon.com/clouddrive/learnmore.2010.
[27] Dropbox. Dropbox [EB/OL]. http://en.wikipedia.org/wiki/Dropbox_(service).2010.
[28] Ubuntu.Ubuntu One [EB/OL]. https://one.ubuntu.com/developer/.2010.
[29] Apple.iCloud Dropbox. http://www.apple.com/icloud/.2010.
[30]邓倩妮,陈全.云计算及其关键技术[J].高性能计算发展与应用,2009,26(1):2-6.
[31]程莹,张云勇,房秉毅,徐雷.云计算标准化现状分析[J].电信科学,2010,S1:6~10.
[32] M. Vrable,S. Savage and G. M. Voelker. Cumulus: Filesystem Backup to the Cloud [C]. In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST '09), Feb.2009:225~238.
[33] Huajian Mao, Nong Xiao, Weisong Shi, et al. Wukong: Toward a Cloud-Orient File Service for Mobile Devices[C].Proc of the 7th International Conference on Services Computing. Miami: IEEE, 2010:498-505.
[34]王庆波.虚拟化与云计算[M].北京:电子工业出版社,2009.
[35] Daniel P. Bovet, Marco Cesati.Understanding the Linux Kernel[M]. O'Reilly Media, 2005. ISBN 0-596-00565-2.
[36] ORACLE.JNI [EB/OL]. http://java.sun.com/docs/books/jni/.2010.
[37] Roy Thomas Fielding.Architectural styles and the design of network-based software architectures: [D].Irvine: University of California, 2000.
[38] Davis.A, Du Zhang.A comparative study of DCOM and SOAP[C].Multimedia Software Engineering. 2002 Proceedings Fourth International Symposium on .2002: 48~55.
[39] Bram Cohen.Incentives Build Robustness in BitTorrent[C]. Workshop on Economics of Peer-to-Peer systems, 2003.
[40] Garfinkel S. An evaluation of amazon’s grid computing services: EC2, S3 and SQS[R]. Massachusetts: HarvardUniversity, 2007.
[41] Google. Google Storage for Develop Developers [EB/OL]. http://code.google.com/apis/storage/docs/getting-started.html. 2010.
[42] Microsoft .Live Mesh [EB/OL]. http://explore.live.com/windows-live-mesh.2010.
[43] Microsoft .REST API [EB/OL]. http://msdn.microsoft.com/en-us/library/hh243648.aspx.2011.
[44]张冬.大话存储Ⅱ-存储系统架构与底层原理极限剖析[M].北京:清华大学出版社.2011.
[45] Moshe Bar. Linux文件系统[M].北京:清华大学出版社.2003.
[46] FIPS PUB 197. Advanced Encryption Standard [EB/OL]. http://csrc.nist.gov/publications, NIST, 2001.
[47] J?rg Roth.Patterns of Mobile Interaction [J].Personal and Ubiquitous Computing. London, UK. September, 2002, 6(4):282-289.
[48]赵英杰.网络存储服务器缓存替换策略研究[D].长沙:国防科学技术大学研究生院,2010.
[49]袁华华.移动数据库中复制与缓存相关技术的研究[D].武汉:湖北工业大学,2008.
[50]吴乐南.数据压缩[M].北京:电子工业出版社.2005.
[51] Yang,P. Deng,H. and Ma,Y. Seamless integration of 3G and 802.11 wireless network [C]. in Proceeding of the 5th ACM international workshop on Mobility management and wireless access.2007.
[52] Sanderson D. Google App Engine编程[M].南京:东南大学出版社.2011.
[53] Tan,W. Lam,F. Lau,W. An Empirical Study on 3G Network Capacity and Performance [C].26th IEEE International Conference on Computer Communications, 2007:1514-1522.
[54] Apache Software Foundation. Tomcat User Guide [EB/OL]. http://tomcat.apache.org/tomcat-6.0-doc/index.html. 2010.
[55] Oracle. Java Servlet API [EB/OL]. http://java.sun.com/products/servlet/index.html. 2010.
[56]孙广磊.征服Python:语言基础与典型应用[M].北京:人民邮电出版社.2007.
[57] Randy Rizun, Dan Moore, Adrian Petrescu. Amazon-S3-based file system [EB/OL]. http://code.google.com/p/s3fs/.2011.
[58] Yinjin Fu, Hong Jiang, Nong Xiao, Lei Tian, Fang Liu. AA-Dedupe: An Application-Aware Source Deduplication Approach for Cloud Backup Services in the Personal Computing Environment [C]. Proceedings of IEEE Cluster 2011 (Cluster 2011), Austin, Texas. September, 2011.
[2] Ansley Post, Juan Navarro, Petr Kuznetsov, Peter Druschel.Autonomous Storage Management for Personal Devices with PodBase [C].In USENIX Annual Technical Conference (USENIX ATC), 2011.
[3] Larry Ponemon.The Cost of a Lost Laptop [R]. Intel Corporation, Apr.2009.
[4] Abadi D. Data management in the cloud: Limitations and opportunities [C].Conference on Data Engineering. Shanghai: IEEEE, 2009:3-12.
[5]舒继武,薛巍,付长冬.网络存储系统与技术的现状与发展趋势[J].中国计算机学会通讯,2004.
[6] Foster I, Zhao Y, Raicu I, Lu S. Cloud Computing and Grid Computing 360-Degree Compared[C]. Grid Computing Environments Workshop(GCE), 2008.
[7] Armbrust M, Fox A, Griffith R, et al.Above the clouds: A Berkeley view of cloud computing [R]. Berkeley: EECS Department, 2009.
[8] Baidu.云手机[EB/OL]. http://baike.baidu.com/view/4692226.htm,2011.
[9]博思数据研究中心. 2011-2015年中国智能手机行业市场分析与投资前景研究报告[R].Technical report, 2011.
[10] H.Choi and H.Varian. Predicting the present with google trends [R]. Technical report, Google Inc, 2009.
[11] Palankar,M.R. and Iamnitchi,A. and Ripeanu,M. and Garfinkel,S. Amazon S3 for science grdi: a viable solution [C]. Proceedings of the 2008 international workshop on Data-aware distributed computing, 2008:55~64.
[12]金海,吴松,廖小飞,袁平鹏等.存储技术的研究发展现状与趋势[R].2009中国计算机科学技术发展报告,中国计算机学会文集,2010.
[13] Gallmeister, B.O. POSIX.4: Programming for the real world [M]. O’Reilly & Associate Inc. 1995.
[14] Google. Google Docs [EB/OL]. http://www.google.com/apps/intl/en/business/docs.html Oct, 2010.
[15] R.Jones. GmailFS:Gmail virtual file system [EB/OL]. http://richard.jones.name/google-hacks/gmail-filesystem/gmail-filesystem.html.
[16] Szeredi M. FUSE: Filesystem in userspace [EB/OL]. http://fuse.sourceforge.net. 2010.
[17]毛华坚.支持异构网络存储服务的YaFS文件系统研究与实现[D].长沙:国防科学技术大学研究生院,2009.
[18] Nokia. Symbian [EB/OL]. http://en.wikipedia.org/wiki/Symbian/.2010.
[19] Microsoft.Windows Mobile [EB/OL]. http://zh.wikipedia.org/wiki/Windows_Mobile/.2010.
[20] Apple.iOS [EB/OL].http://developer.apple.com/.2010.
[21] HP.Web OS [EB/OL].http://en.wikipedia.org/wiki/WebOS/.2011.
[22] Google.Android [EB/OL]. http://code.google.com/intl/zh-CN/android/.2011.
[23] Buyya, R. Chee Shin Yeo Venugopal, S. Market-Oriented Cloud Computing: Vision, Hype, and Reality for Delivering IT Services as Computing Utilities[C]. High Performance Computing and Communications(HPCC), 2008.
[24] Jiyi Wu, Lingdi Ping, et al.Cloud Storage as the Infrastructure of Cloud Computing [C]. Intelligent Computing and Cognitive Informatics (ICICCI), 2010:380~383.
[25] Microsoft.Windows Live skyDrive [EB/OL]. http://www.windowslive.cn/skydrive/.2010.
[26] Amazon.Cloud Drive [EB/OL]. https://www.amazon.com/clouddrive/learnmore.2010.
[27] Dropbox. Dropbox [EB/OL]. http://en.wikipedia.org/wiki/Dropbox_(service).2010.
[28] Ubuntu.Ubuntu One [EB/OL]. https://one.ubuntu.com/developer/.2010.
[29] Apple.iCloud Dropbox. http://www.apple.com/icloud/.2010.
[30]邓倩妮,陈全.云计算及其关键技术[J].高性能计算发展与应用,2009,26(1):2-6.
[31]程莹,张云勇,房秉毅,徐雷.云计算标准化现状分析[J].电信科学,2010,S1:6~10.
[32] M. Vrable,S. Savage and G. M. Voelker. Cumulus: Filesystem Backup to the Cloud [C]. In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST '09), Feb.2009:225~238.
[33] Huajian Mao, Nong Xiao, Weisong Shi, et al. Wukong: Toward a Cloud-Orient File Service for Mobile Devices[C].Proc of the 7th International Conference on Services Computing. Miami: IEEE, 2010:498-505.
[34]王庆波.虚拟化与云计算[M].北京:电子工业出版社,2009.
[35] Daniel P. Bovet, Marco Cesati.Understanding the Linux Kernel[M]. O'Reilly Media, 2005. ISBN 0-596-00565-2.
[36] ORACLE.JNI [EB/OL]. http://java.sun.com/docs/books/jni/.2010.
[37] Roy Thomas Fielding.Architectural styles and the design of network-based software architectures: [D].Irvine: University of California, 2000.
[38] Davis.A, Du Zhang.A comparative study of DCOM and SOAP[C].Multimedia Software Engineering. 2002 Proceedings Fourth International Symposium on .2002: 48~55.
[39] Bram Cohen.Incentives Build Robustness in BitTorrent[C]. Workshop on Economics of Peer-to-Peer systems, 2003.
[40] Garfinkel S. An evaluation of amazon’s grid computing services: EC2, S3 and SQS[R]. Massachusetts: HarvardUniversity, 2007.
[41] Google. Google Storage for Develop Developers [EB/OL]. http://code.google.com/apis/storage/docs/getting-started.html. 2010.
[42] Microsoft .Live Mesh [EB/OL]. http://explore.live.com/windows-live-mesh.2010.
[43] Microsoft .REST API [EB/OL]. http://msdn.microsoft.com/en-us/library/hh243648.aspx.2011.
[44]张冬.大话存储Ⅱ-存储系统架构与底层原理极限剖析[M].北京:清华大学出版社.2011.
[45] Moshe Bar. Linux文件系统[M].北京:清华大学出版社.2003.
[46] FIPS PUB 197. Advanced Encryption Standard [EB/OL]. http://csrc.nist.gov/publications, NIST, 2001.
[47] J?rg Roth.Patterns of Mobile Interaction [J].Personal and Ubiquitous Computing. London, UK. September, 2002, 6(4):282-289.
[48]赵英杰.网络存储服务器缓存替换策略研究[D].长沙:国防科学技术大学研究生院,2010.
[49]袁华华.移动数据库中复制与缓存相关技术的研究[D].武汉:湖北工业大学,2008.
[50]吴乐南.数据压缩[M].北京:电子工业出版社.2005.
[51] Yang,P. Deng,H. and Ma,Y. Seamless integration of 3G and 802.11 wireless network [C]. in Proceeding of the 5th ACM international workshop on Mobility management and wireless access.2007.
[52] Sanderson D. Google App Engine编程[M].南京:东南大学出版社.2011.
[53] Tan,W. Lam,F. Lau,W. An Empirical Study on 3G Network Capacity and Performance [C].26th IEEE International Conference on Computer Communications, 2007:1514-1522.
[54] Apache Software Foundation. Tomcat User Guide [EB/OL]. http://tomcat.apache.org/tomcat-6.0-doc/index.html. 2010.
[55] Oracle. Java Servlet API [EB/OL]. http://java.sun.com/products/servlet/index.html. 2010.
[56]孙广磊.征服Python:语言基础与典型应用[M].北京:人民邮电出版社.2007.
[57] Randy Rizun, Dan Moore, Adrian Petrescu. Amazon-S3-based file system [EB/OL]. http://code.google.com/p/s3fs/.2011.
[58] Yinjin Fu, Hong Jiang, Nong Xiao, Lei Tian, Fang Liu. AA-Dedupe: An Application-Aware Source Deduplication Approach for Cloud Backup Services in the Personal Computing Environment [C]. Proceedings of IEEE Cluster 2011 (Cluster 2011), Austin, Texas. September, 2011.