数字有机体数据库系统批量同步设计与实现
摘要
随着计算机的普及,信息技术的发展,越来越多的信息被数字化,在网络上传播,数字化信息也呈爆炸式增长。在近几年和未来的日子里,如何存储和管理这些数据都将成为人们关注的焦点。八、九十年代,数据被存放在昂贵的中小型机器中,由管理员维护数据,不但需要较大的投资和管理精力,而且一旦机器出现故障,将会中断系统对外服务,造成数据丢失,给企业带来损失。近年来随着人们对分布式系统深入研究、硬件技术的高速发展、网络技术的逐渐成熟、以及开源操作系统Linux的迅速崛起,使开发者们对操作系统进行不断的扩展开发,出现了一批以Linux为基础的分布式服务器系统,来存储和管理这些数据。服务器系统由多台高性能的PC通过高速局域网络连接,在分布式操作系统的协调管理下形成有机整体,保证数据的高度可靠,又使系统的服务能力较单机有显著提高。
本教研室在多年分布式并行系统研发的基础上,结合P2P技术,开发出了数字有机体系统,以资源管理为根本出发点,满足不同的网络应用需要。数字有机体系统的主要任务是信息管理和任务调度。在分布式信息管理中,如何实现各个服务器的数据资源一致是研究的一个重点:在已实现的系统中,保证强数据一致性应用占有绝大部分,它们有很广阔的应用空间,像银行,政府,大型企业等领域;但是对于娱乐媒体等普通应用却有些浪费。为了适应不同用户的需要和保证系统能够适用不同的领域,提高资源信息的访问速度是设计实现当中必须考虑的重点问题。
所以数字有机体数据库系统,采用了两种不同的设计理念,强数据一致性和弱数据一致性。本文是在弱数据一致性理念上设计和实现了数字有机体数据库批量同步系统,主要目的是在用户允许的数据同步一致范围内能够尽快的为用户提供服务。该系统采用了对数据库资源的批量同步,提高了数据资源同步的速度,极大地扩充了数字有机体数据库系统的应用范围。
测试表明数字有机体数据库批量同步系统所提供的批量同步接口达到了设计的功能要求,在运行中能很好的满足一般用户的使用要求。
With the popularization of computer and the development of the information technology, more and more information were digitized, and transmited in the internet, the data increased in a drastically speed. As a result, in the recent years and future, how to store and manage these data will become a focus concerned by human. In the 80’s and 90’s, these data were stored in the expensive middle or mini computers. And the system manager must back up the data periodically. So the enterprise should make a huge investment, and once the machine gets some error, the service will be stopped , make lots od data loss and the enterprise will suffer from some loss. In recent years, people impenetrate the distribute system, along with the high development of the hardware, the high-speed network come to be mature, and the open-source operating system of Linux has been developed quickly, more and more developer make extern develop and many distributed system based on Linux has been developed to store and manage data. These systems are generally composed by high-performance PCs connected by the high-speed LAN. Managed by the distributed operating system, it can guarantee the reliability of the data and the high performance compare to one PC.
A new generation of distributed system - DOS (Digital Organism System) has been designed to arrange the resources and storage spaces of computers in a wide area network in the our research office. DOS, which combines with distributed character of P2P, focuses on resource management to meet the need of various network applications. The primary task of DOS is resource management and task dispatch. For the distributed information management, the most important part is data consistency in the every servers,the DOS system have the ability keep strong data consistency for most of application, such as bank, government,big comperty and so on. But it is lavish for application of media,net information etc. so, in order to meet more user can use it, the efficiency of resource accessing must be improved and taken it into account all the time in the system designed and implement.
In the DOS system, introduced two method-strong and week data consistency. This document has designed and implemented batch synchronization system of DOSDB based on the theory of week data consistency, and the arm is that offer servers for users as soon as possible.The system use batch synchronize database resource,it improve the speed and make DOSDBMS have more using field.
The result of test has shown that this system can meet the demands of user effectively.
本教研室在多年分布式并行系统研发的基础上,结合P2P技术,开发出了数字有机体系统,以资源管理为根本出发点,满足不同的网络应用需要。数字有机体系统的主要任务是信息管理和任务调度。在分布式信息管理中,如何实现各个服务器的数据资源一致是研究的一个重点:在已实现的系统中,保证强数据一致性应用占有绝大部分,它们有很广阔的应用空间,像银行,政府,大型企业等领域;但是对于娱乐媒体等普通应用却有些浪费。为了适应不同用户的需要和保证系统能够适用不同的领域,提高资源信息的访问速度是设计实现当中必须考虑的重点问题。
所以数字有机体数据库系统,采用了两种不同的设计理念,强数据一致性和弱数据一致性。本文是在弱数据一致性理念上设计和实现了数字有机体数据库批量同步系统,主要目的是在用户允许的数据同步一致范围内能够尽快的为用户提供服务。该系统采用了对数据库资源的批量同步,提高了数据资源同步的速度,极大地扩充了数字有机体数据库系统的应用范围。
测试表明数字有机体数据库批量同步系统所提供的批量同步接口达到了设计的功能要求,在运行中能很好的满足一般用户的使用要求。
With the popularization of computer and the development of the information technology, more and more information were digitized, and transmited in the internet, the data increased in a drastically speed. As a result, in the recent years and future, how to store and manage these data will become a focus concerned by human. In the 80’s and 90’s, these data were stored in the expensive middle or mini computers. And the system manager must back up the data periodically. So the enterprise should make a huge investment, and once the machine gets some error, the service will be stopped , make lots od data loss and the enterprise will suffer from some loss. In recent years, people impenetrate the distribute system, along with the high development of the hardware, the high-speed network come to be mature, and the open-source operating system of Linux has been developed quickly, more and more developer make extern develop and many distributed system based on Linux has been developed to store and manage data. These systems are generally composed by high-performance PCs connected by the high-speed LAN. Managed by the distributed operating system, it can guarantee the reliability of the data and the high performance compare to one PC.
A new generation of distributed system - DOS (Digital Organism System) has been designed to arrange the resources and storage spaces of computers in a wide area network in the our research office. DOS, which combines with distributed character of P2P, focuses on resource management to meet the need of various network applications. The primary task of DOS is resource management and task dispatch. For the distributed information management, the most important part is data consistency in the every servers,the DOS system have the ability keep strong data consistency for most of application, such as bank, government,big comperty and so on. But it is lavish for application of media,net information etc. so, in order to meet more user can use it, the efficiency of resource accessing must be improved and taken it into account all the time in the system designed and implement.
In the DOS system, introduced two method-strong and week data consistency. This document has designed and implemented batch synchronization system of DOSDB based on the theory of week data consistency, and the arm is that offer servers for users as soon as possible.The system use batch synchronize database resource,it improve the speed and make DOSDBMS have more using field.
The result of test has shown that this system can meet the demands of user effectively.
引文
[1]刘心松,王典荫.《数字有机体抗毁结构的研究》.西部广播电视, 2005年第二期
[2]王典荫,刘心松.《下一代计算机系统-数字有机体》.西部广播电视, 2005年第一期
[3]邱元杰.数字有机体软件需求规约.电子科技大学8010研究室
[4]刘大昕,周军.一种维护分布式数据库中数据一致性的方法.应用科技, 2003年9月
[5]刘文辉.分布式系统一致性方法.喀什师范学院学报2003年5月
[6] Ceri S.Pelagatti G. Distributed Database-Principles and systems. McGraw-Hill computer science series.1984
[7]陈珉,喻丹丹,涂国庆.分布式数据库系统中数据一致性维护方法研究.国防科技大学学报2002 Vol.24 No.3
[8] Mohd Adib Omar, Fazilah Haron, Rosni Abdullah. Adaptive data consistency management in collaborative editin. IEEE 2004.
[9] Andrew S.Tanenbaum, Maarten van Steen著杨剑峰,常晓波,李敏译.分布式系统:原理与泛型.北京:清华大学出版社, 2004
[10]朱海滨,蔡开裕,樊爱华,宋辉.分布式系统原理与设计.长沙:国防科技大学出版社, 1997
[11] Rajkumar Buyya编郑伟民,石威,汪东升等译.高性能集群计算:结构与系统(第一卷).北京:电子工业出版社2001
[12]邵佩英.分布式数据库系统及其应用.科学出版社(2000)
[13] M.Tamer, Patrick Valduriez. Principles of Distributed Database Systems. Prentice-Hall清华大学出版社(影印版)2002
[14]萨师煊,王珊.数据库系统概论[M].北京:高等教育出版社, 1995.
[15]冯玉才.数据库基础[M].北京:高等教育出版社, 1990.5
[16] Hector Garcia-Molina,Jeffrey D.Ullman,Jennifer Widom.数据库系统实现.斯坦福大学2000.
[17] Stephanos Androutsellis-Theotokis.“White Paper: A Survey of Peer-to-Peer File Sharing Technologies”ELTRUN. Athens University of Economics and Business, Greece,2002
[18] Ragib Hasan,Zahid Anwar, William Yurcik, et al. A Survey of Peer-to-Peer Storage Techniques for Distributed File Systems. Proceeding of the International Conference on Information Technology:Coding and Computing(ITCC’05),2005
[19] Skype. http://www.skype.com
[20] Dejan S.Milojicic,Vana Kalogeraki Rajan Lukose,Kiran Nagaraja,Jim Pruyne,Bruno Richard,Sami Rollins,Zhichen Xu,HP Laboratories Palo Alto,”Peer-to-Peer Computing”,University of California
[21] Gnutella. http://www.gnutella.com
[22] M.Ripeanu. Peer-to-peer Architecture Case Study: Gnutella. In Proceedings of International Conference on P2P Computing, 2001.
[23] Distributed Hash Table,Technical Definition. http://ceria.dauphine.fr/SDDS-bibliograhie.html
[24] Ion Stoica, Robert Morris, David Karger, M.Frans Kaashoek, Hari Balakrishnan. Chord A Scalable Peer-to-peer Lookup Service for Internet Applications. In Proceeding of ACM SIGCOMM 2001, San Diego,California, USA.
[25]邱元杰.数字有机体软件概要设计.电子科技大学8010研究室
[26] Rasolofo Y, et al. Approach to Collection Selection and Results Merg2ing for Distributed Information Retrieval. Proceedings of the Tenth In2 ternational Conference on Information and Knowledge Management, At2 lanta, 2001.
[27] M.Mat Deris, J.H Abawajy and H.M. Suzuri .An efficient replicated data access approach for large-scale. IEEE international Sympoalum on cluster computing 2004.
[28]陈建英,刘心松,左朝树等.基于数据动态冗余的分布式并行系统重构.计算机应用研究, 2004, Vol.21, No.11: 229-231, 256
[29]胡华平,金士尧等.分布式实时系统的高可靠性研究与实现.计算机研究与发展, 1998.9,第9期
[30] John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski,Patrick Eaton, Dennis Geels, Ramakrishna Gummadi, Sean Rhea,Hakim Weatherspoon, Westley Weimer, Chris Wells, and Ben Zhao. OceanStore: An Architecture for GlobalScale Persistent Storage. 2000 ACM 089791886/97/05
[31] Jim Gray, Andreas Reuter著,孟小峰等译.事务处理:概念与技术=Transaction processing: concepts and techniques.北京:机械工业出版社2004
[32] Gerhard Weikum,Gottfried Vossen著,陈立军等译.事务信息系统=Transactional information systems:并发控制与恢复的理论、算法与实践.北京:机械工业出版社2006
[33]廖国琼,刘云生等.日志驱动主从弱一致性实时复制策略.计算机科学2005 Vol.32 No.7
[34]胡忠顺.分布式数据库操作中的并行控制理论与算法.上海海运学院学报1997 vol 18 No.1
[35]陈俊伽.关系数据库并发控制机制研究.科学技术与工程2007 vol 7 No.18
[36]王迤冉.数据库并发控制的研究.周口师范学院学报2004 vol 21 No.2
[37]石兵,李兴明.故障网络的分布式恢复策略.数据通信2001年第2期
[38] Cem Kaner, Jack Falk, Hung Quoc Nguyen著,王峰等译.计算机软件测试.北京:机械工业出版社, 2004.5
[2]王典荫,刘心松.《下一代计算机系统-数字有机体》.西部广播电视, 2005年第一期
[3]邱元杰.数字有机体软件需求规约.电子科技大学8010研究室
[4]刘大昕,周军.一种维护分布式数据库中数据一致性的方法.应用科技, 2003年9月
[5]刘文辉.分布式系统一致性方法.喀什师范学院学报2003年5月
[6] Ceri S.Pelagatti G. Distributed Database-Principles and systems. McGraw-Hill computer science series.1984
[7]陈珉,喻丹丹,涂国庆.分布式数据库系统中数据一致性维护方法研究.国防科技大学学报2002 Vol.24 No.3
[8] Mohd Adib Omar, Fazilah Haron, Rosni Abdullah. Adaptive data consistency management in collaborative editin. IEEE 2004.
[9] Andrew S.Tanenbaum, Maarten van Steen著杨剑峰,常晓波,李敏译.分布式系统:原理与泛型.北京:清华大学出版社, 2004
[10]朱海滨,蔡开裕,樊爱华,宋辉.分布式系统原理与设计.长沙:国防科技大学出版社, 1997
[11] Rajkumar Buyya编郑伟民,石威,汪东升等译.高性能集群计算:结构与系统(第一卷).北京:电子工业出版社2001
[12]邵佩英.分布式数据库系统及其应用.科学出版社(2000)
[13] M.Tamer, Patrick Valduriez. Principles of Distributed Database Systems. Prentice-Hall清华大学出版社(影印版)2002
[14]萨师煊,王珊.数据库系统概论[M].北京:高等教育出版社, 1995.
[15]冯玉才.数据库基础[M].北京:高等教育出版社, 1990.5
[16] Hector Garcia-Molina,Jeffrey D.Ullman,Jennifer Widom.数据库系统实现.斯坦福大学2000.
[17] Stephanos Androutsellis-Theotokis.“White Paper: A Survey of Peer-to-Peer File Sharing Technologies”ELTRUN. Athens University of Economics and Business, Greece,2002
[18] Ragib Hasan,Zahid Anwar, William Yurcik, et al. A Survey of Peer-to-Peer Storage Techniques for Distributed File Systems. Proceeding of the International Conference on Information Technology:Coding and Computing(ITCC’05),2005
[19] Skype. http://www.skype.com
[20] Dejan S.Milojicic,Vana Kalogeraki Rajan Lukose,Kiran Nagaraja,Jim Pruyne,Bruno Richard,Sami Rollins,Zhichen Xu,HP Laboratories Palo Alto,”Peer-to-Peer Computing”,University of California
[21] Gnutella. http://www.gnutella.com
[22] M.Ripeanu. Peer-to-peer Architecture Case Study: Gnutella. In Proceedings of International Conference on P2P Computing, 2001.
[23] Distributed Hash Table,Technical Definition. http://ceria.dauphine.fr/SDDS-bibliograhie.html
[24] Ion Stoica, Robert Morris, David Karger, M.Frans Kaashoek, Hari Balakrishnan. Chord A Scalable Peer-to-peer Lookup Service for Internet Applications. In Proceeding of ACM SIGCOMM 2001, San Diego,California, USA.
[25]邱元杰.数字有机体软件概要设计.电子科技大学8010研究室
[26] Rasolofo Y, et al. Approach to Collection Selection and Results Merg2ing for Distributed Information Retrieval. Proceedings of the Tenth In2 ternational Conference on Information and Knowledge Management, At2 lanta, 2001.
[27] M.Mat Deris, J.H Abawajy and H.M. Suzuri .An efficient replicated data access approach for large-scale. IEEE international Sympoalum on cluster computing 2004.
[28]陈建英,刘心松,左朝树等.基于数据动态冗余的分布式并行系统重构.计算机应用研究, 2004, Vol.21, No.11: 229-231, 256
[29]胡华平,金士尧等.分布式实时系统的高可靠性研究与实现.计算机研究与发展, 1998.9,第9期
[30] John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski,Patrick Eaton, Dennis Geels, Ramakrishna Gummadi, Sean Rhea,Hakim Weatherspoon, Westley Weimer, Chris Wells, and Ben Zhao. OceanStore: An Architecture for GlobalScale Persistent Storage. 2000 ACM 089791886/97/05
[31] Jim Gray, Andreas Reuter著,孟小峰等译.事务处理:概念与技术=Transaction processing: concepts and techniques.北京:机械工业出版社2004
[32] Gerhard Weikum,Gottfried Vossen著,陈立军等译.事务信息系统=Transactional information systems:并发控制与恢复的理论、算法与实践.北京:机械工业出版社2006
[33]廖国琼,刘云生等.日志驱动主从弱一致性实时复制策略.计算机科学2005 Vol.32 No.7
[34]胡忠顺.分布式数据库操作中的并行控制理论与算法.上海海运学院学报1997 vol 18 No.1
[35]陈俊伽.关系数据库并发控制机制研究.科学技术与工程2007 vol 7 No.18
[36]王迤冉.数据库并发控制的研究.周口师范学院学报2004 vol 21 No.2
[37]石兵,李兴明.故障网络的分布式恢复策略.数据通信2001年第2期
[38] Cem Kaner, Jack Falk, Hung Quoc Nguyen著,王峰等译.计算机软件测试.北京:机械工业出版社, 2004.5