内存数据库事务并发控制研究和设计
|
摘要
随着电力行业信息化程度不断普及,电力系统对现场数据的实时处理的需求日渐提高,催生了支持实时性的高性能数据库的诞生。内存容量的增加和成本的降低,使基于内存的高性能实时数据库成为主流技术趋势。内存数据库的数据主要存储在内存中,数据的读写都在内存中完成,没有因磁盘I/O带来很大的开销。因此,在海量数据的处理过程中,事务并发控制的好坏直接决定了数据处理性能的高低。本文的目标就是研究内存数据库事务并发控制技术,设计一套适合于内存数据库事务管理的方案,以满足电力行业海量数据处理的需求。
由于电力行业现场采集数据到达时刻极不均衡,具有较强的突发性,这给事务的并发控制带来了空前的挑战。本文通过分析现有事务并发控制技术,包括两阶段加锁协议、树形多粒度锁、意向锁、动态多粒度锁和死锁检测等,提出了本文要解决的问题:多核环境下高突发性事务流的并发控制问题。
通过对多核环境下高突发性事务流的并发控制问题的深入分析,本文基于现有的自主开发的内存数据库——CacheDB的构架,设计了一套适合电力行业应用的事务管理功能模块,包括事务管理器、事务调度器和锁管理器。该方案综合了意向锁和动态多粒度锁的优点,事务冲突多时,采用并发能力强的行级锁;事务冲突少时,采用加锁开销小的表级锁,很大程度地提高了系统的总体性能。此外,本文对传统的动态多粒度锁机制进行了改进,重点考虑了锁的相容及开销情况,优化了事务冲突的判决机制;针对动态多粒度锁,优化了现有的死锁检测机制,尽可能的减少不必要的死锁检测动作。
为了验证CacheDB事务管理方案的可行性,本文对闩和动态多粒度锁进行了仿真测试。测试结果显示,采用意向锁和动态多粒度锁相结合的事务并发控制设计方案,在数据到达不均衡条件下,系统总体事务处理性能得到明显提高。
With the development of technology, the disposal of real-time data requires the support of high capable Database. The data of Main Memory Database (MMDB) is memorized in Memory and all orders are processed in the Memory, thus Disc I/O is not the bottleneck of data visiting any more. In the process of great capacity for data, the control of transaction concurrency is crucial to the performance degree of data processing. In order to satisfy the demand of great capacity for simultaneity electric power, it is necessary to design a kind of fit-for scheme of CacheDB Transaction. The objective of this article is to research the related technologies of Memory Database Transaction, especially to deal with transaction concurrency control.
The extreme imbalance of great capacity for data in the industry of electric power brings the transaction of concurrency control an unprecedented challenge. With a responsibly professional analysis on current transaction concurrent control, my paper is trying to figure out emergent tasks, I deeply analyzed the key technologies concerned of current locking, including the protocol of two-phrase locking, the multiple locking of tree-format, the intension Lock, the dynamic multi-granularity locking and the detection of deadlock.
After deeply analyzing the crucial technologies and combining with the current framework of CacheDB, I design a series of procedures, which are fit to manage the transaction of CacheDB. My ideas is to adopt both intension lock and dynamic muti- granularity locking: when conflicts of transaction are frequently, row level lock will be in effect; on the contrary, table level lock will be in use.
This paper provides a way to improve the mechanism of dynamic muti-granularity, especially optimizing the weight of lock compatibility and cost to transaction conflicts. Comparing to dynamic multi-granularity, it aims to improve the mechanism of deadlock detection with a goal of reducing unnecessary deadlock detection preocedures.
To test the feasibility of the design of transaction management, I have done an imitating-test on latch and dynamic muti-granularity locking. Test Results Indicate that once the data scale reaches imbalance, the scheme of the transaction concurrency control that is a combination of Intension Lock and dynamic muti-granularity locking, turns out comparable processing feasibility.
由于电力行业现场采集数据到达时刻极不均衡,具有较强的突发性,这给事务的并发控制带来了空前的挑战。本文通过分析现有事务并发控制技术,包括两阶段加锁协议、树形多粒度锁、意向锁、动态多粒度锁和死锁检测等,提出了本文要解决的问题:多核环境下高突发性事务流的并发控制问题。
通过对多核环境下高突发性事务流的并发控制问题的深入分析,本文基于现有的自主开发的内存数据库——CacheDB的构架,设计了一套适合电力行业应用的事务管理功能模块,包括事务管理器、事务调度器和锁管理器。该方案综合了意向锁和动态多粒度锁的优点,事务冲突多时,采用并发能力强的行级锁;事务冲突少时,采用加锁开销小的表级锁,很大程度地提高了系统的总体性能。此外,本文对传统的动态多粒度锁机制进行了改进,重点考虑了锁的相容及开销情况,优化了事务冲突的判决机制;针对动态多粒度锁,优化了现有的死锁检测机制,尽可能的减少不必要的死锁检测动作。
为了验证CacheDB事务管理方案的可行性,本文对闩和动态多粒度锁进行了仿真测试。测试结果显示,采用意向锁和动态多粒度锁相结合的事务并发控制设计方案,在数据到达不均衡条件下,系统总体事务处理性能得到明显提高。
With the development of technology, the disposal of real-time data requires the support of high capable Database. The data of Main Memory Database (MMDB) is memorized in Memory and all orders are processed in the Memory, thus Disc I/O is not the bottleneck of data visiting any more. In the process of great capacity for data, the control of transaction concurrency is crucial to the performance degree of data processing. In order to satisfy the demand of great capacity for simultaneity electric power, it is necessary to design a kind of fit-for scheme of CacheDB Transaction. The objective of this article is to research the related technologies of Memory Database Transaction, especially to deal with transaction concurrency control.
The extreme imbalance of great capacity for data in the industry of electric power brings the transaction of concurrency control an unprecedented challenge. With a responsibly professional analysis on current transaction concurrent control, my paper is trying to figure out emergent tasks, I deeply analyzed the key technologies concerned of current locking, including the protocol of two-phrase locking, the multiple locking of tree-format, the intension Lock, the dynamic multi-granularity locking and the detection of deadlock.
After deeply analyzing the crucial technologies and combining with the current framework of CacheDB, I design a series of procedures, which are fit to manage the transaction of CacheDB. My ideas is to adopt both intension lock and dynamic muti- granularity locking: when conflicts of transaction are frequently, row level lock will be in effect; on the contrary, table level lock will be in use.
This paper provides a way to improve the mechanism of dynamic muti-granularity, especially optimizing the weight of lock compatibility and cost to transaction conflicts. Comparing to dynamic multi-granularity, it aims to improve the mechanism of deadlock detection with a goal of reducing unnecessary deadlock detection preocedures.
To test the feasibility of the design of transaction management, I have done an imitating-test on latch and dynamic muti-granularity locking. Test Results Indicate that once the data scale reaches imbalance, the scheme of the transaction concurrency control that is a combination of Intension Lock and dynamic muti-granularity locking, turns out comparable processing feasibility.
引文
[1] Hector Garcia-Molina, Kenneth Salem. Main memory database systems: an overview. IEEE. Knowledge and Data Engineering, IEEE Transactions on Volume 4, Issue 6, Dec. 1992, Page(s):509– 516
[2]施胜盛.内存数据库事务管理研究与实现:[硕士学位论文].杭州:浙江大学. 2006
[3]殷卫红,刘亚成.电网监控系统实时数据库的设计与实现.华东电力. 2006,第12期
[4] Abraham Silberschatz, Henry E. Korth, S. Sudarshan.数据库系统概念.杨冬青,唐世渭等译.北京:机械工业出版社. 2005. 398– 472
[5]李国徽,刘云生.实时数据库并发控制.小型微型计算机系统. 2001,第12期
[6]夏家莉.实时数据库系统中的并发控制协议及分析.计算机工程和应用. 2002,第7期
[7] John Goodacre.多重处理的设计选择:多处理器或多线程技术.电子设计和应用. 2006,第8期
[8] MySQL 6.0 Reference Manual. 2009
[9] Piyush Burte, Boanerges Aleman-Meza, D. Brent Weatherly. Transaction Management for a Main-Memory Database. 2001
[10] Atomic Commit In SQLite. http://www.sqlite.org. 2009
[11] Jim Gray. Notes on Data Base Operating Systems. Operating Systems, An Advanced Course, 1977, Page(s):393– 481
[12] Heiko Schuldt, Gustavo Alonso, Catriel Beeri. Atomicity and isolation for transactional processes. ACM Transactions on Database Systems (TODS), 27(1), 2002, Page(s):63– 116
[13] C. Mohan, Don Haderle, Bruce Lindsay et al. Aries: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Transactions on Database Systems, 17(1), 1992, Page(s): 94– 162
[14] K. P. Eswaran, J. N. Gray, R. A. Lorie. The notions of consistency and predicate locks in a database system. Communications of the ACM, Volume 19, 1976, Page(s): 624 - 633
[15] Alexander Thomasian. Concurrency Control: Methods, Performance, and Analysis. 1998
[16] Hector Garcia Milina, Jeffrey D. Ullman, Jennifer Widom.数据库系统实现.北京:机械工业出版社.杨冬青,唐世渭,徐其钧等译. 2006. 333 - 418
[17] Anindya Datta, Sang H. Son. A study of concurrency control in real-time, active database systems. IEEE Transactions on Knowledge and Data Engineering,14(3), 2002, Page(s): 465 - 484
[18] Jim Gray, R. A. Lorie, G. R. Putzolu et al. Granularity of Locks and Degrees of Consistency in a Shared DataBase, Modeling in Data Base Management System, 1976
[19] Jim Gray, R. A. Lorie, G. R. Putzolu. Granularity of locks in a shared database. 1975
[20] Vibby Gottemukkala, Tobin J. Lehman. Locking and Latching in a Memory-Resident Database System. Proceedings of the 18th International Conference on Very Large Data Bases ,1992, Page(s): 533 - 544
[21] Michael J. Carey. Granularity hierarchies in concurrency control. Proceedings of the 2nd ACM SIGACT-SIGMOD symposium on Principles of database systems, 1983, Page(s): 156 - 165
[22] J. R. Jordan, J. Banerjee, R. B. Batman. Precision locks. Proceedings of the 1981 ACM SIGMOD international conference on Management of data, 1981, Page(s): 143 - 147
[23] Lorry D. Molesky, Krithi Ramamritham. Efficient Locking for Shared Memory Database Systems. Technical Report : University of Massachusetts, 1994
[24] Tobin J. Lehman, Vibby Gottemukkala. The Design and Performance Evaluation of a Lock Manager for a Memory-Resident Database System, Performance of concurrency control mechanisms in centralized database systems, 1996, Page(s): 406 - 428
[25] Qilong Han, Haiwei Pan, Guisheng Yin. A Concurrency Control Algorithm Access to Temporal Data in Real-Time Database Systems. Computer and Computational Sciences, 2008. IMSCCS '08. International Multisymposiums, Oct. 2008, Page(s):168 - 171
[26] Khaled S. Maabreh, Alaa Al-Hamami. Increasing database concurrency control based on attribute level locking. Electronic Design, 2008. ICED 2008. International Conference, Dec. 2008, Page(s):1– 4
[27]朱铭,李航,王力.内存数据库的锁机制.微机发展. 2003,第13卷,第12期
[28] Oracle TimesTen: In-Memory Database Introduction, www.oracle.com/database/in-memory-database-cache.html. 2007
[29] PostgreSQL 8.3.6 Documentation. 2008
[30]董钢. SQL SERVER数据库封锁及死锁研究.计算机与现代化. 2005,第8期
[31]李晶. Sybase数据库系统并发控制.计算机工程. 1999,第11期
[32] Yong Jiang, Jie Li, Shoichi Nishimura. A general stochastic model for dynamic locking in database systems,IEEE Transactions on Computers, 53(3), 2004, Page(s):308 - 319
[33] Anha Mittal, Sivatama P. Dandamudi. Dynamic versus static locking in real-time parallel database systems,2004
[34] Martin Kot. Modeling real-time database concurrency control protocol two-phase-locking in Uppaal,2008
[35]彭正文,徐新爱.基于SMP的Linux内核自旋锁分析.江西教育学院学报. 2005,第3期
[36]韩凯,王京春.实时数据库调度策略综述.计算机工程与应用. 2004,第32期
[37] Y. M. P. Fernandes, A. Perkusich, Pedro F. Ribeiro Neto. Implementation of transactions scheduling for real-time database management. IEEE. 2004
[38] Ravi Sethi. A model of concurrent database transactions. Proceedings of the 22nd Annual Symposium on Foundations of Computer Science, Page(s): 175 - 184
[39]陈传波,王桦.实时数据库的事务调度研究.计算机应用. 2005,第9期
[40]葛俊杰.实时内存数据库的事务调度与数据恢复研究.青岛大学硕士论文. 2007
[41] H. V. Jagadish, Daniel Lieuwen, Rajeev Rastogi et al. Dali: A High Performance Main Memory Storage Manager. Proceedings of the 20th International Conference on Very Large Data Bases, 1994, Page(s):48 - 59
[42] Daniel P. Bovet, Marco Cesati.深入理解LINUX内核(第二版).陈莉君,冯锐,牛欣源等译.中国电力出版社. 2004, 168 - 196.
[2]施胜盛.内存数据库事务管理研究与实现:[硕士学位论文].杭州:浙江大学. 2006
[3]殷卫红,刘亚成.电网监控系统实时数据库的设计与实现.华东电力. 2006,第12期
[4] Abraham Silberschatz, Henry E. Korth, S. Sudarshan.数据库系统概念.杨冬青,唐世渭等译.北京:机械工业出版社. 2005. 398– 472
[5]李国徽,刘云生.实时数据库并发控制.小型微型计算机系统. 2001,第12期
[6]夏家莉.实时数据库系统中的并发控制协议及分析.计算机工程和应用. 2002,第7期
[7] John Goodacre.多重处理的设计选择:多处理器或多线程技术.电子设计和应用. 2006,第8期
[8] MySQL 6.0 Reference Manual. 2009
[9] Piyush Burte, Boanerges Aleman-Meza, D. Brent Weatherly. Transaction Management for a Main-Memory Database. 2001
[10] Atomic Commit In SQLite. http://www.sqlite.org. 2009
[11] Jim Gray. Notes on Data Base Operating Systems. Operating Systems, An Advanced Course, 1977, Page(s):393– 481
[12] Heiko Schuldt, Gustavo Alonso, Catriel Beeri. Atomicity and isolation for transactional processes. ACM Transactions on Database Systems (TODS), 27(1), 2002, Page(s):63– 116
[13] C. Mohan, Don Haderle, Bruce Lindsay et al. Aries: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Transactions on Database Systems, 17(1), 1992, Page(s): 94– 162
[14] K. P. Eswaran, J. N. Gray, R. A. Lorie. The notions of consistency and predicate locks in a database system. Communications of the ACM, Volume 19, 1976, Page(s): 624 - 633
[15] Alexander Thomasian. Concurrency Control: Methods, Performance, and Analysis. 1998
[16] Hector Garcia Milina, Jeffrey D. Ullman, Jennifer Widom.数据库系统实现.北京:机械工业出版社.杨冬青,唐世渭,徐其钧等译. 2006. 333 - 418
[17] Anindya Datta, Sang H. Son. A study of concurrency control in real-time, active database systems. IEEE Transactions on Knowledge and Data Engineering,14(3), 2002, Page(s): 465 - 484
[18] Jim Gray, R. A. Lorie, G. R. Putzolu et al. Granularity of Locks and Degrees of Consistency in a Shared DataBase, Modeling in Data Base Management System, 1976
[19] Jim Gray, R. A. Lorie, G. R. Putzolu. Granularity of locks in a shared database. 1975
[20] Vibby Gottemukkala, Tobin J. Lehman. Locking and Latching in a Memory-Resident Database System. Proceedings of the 18th International Conference on Very Large Data Bases ,1992, Page(s): 533 - 544
[21] Michael J. Carey. Granularity hierarchies in concurrency control. Proceedings of the 2nd ACM SIGACT-SIGMOD symposium on Principles of database systems, 1983, Page(s): 156 - 165
[22] J. R. Jordan, J. Banerjee, R. B. Batman. Precision locks. Proceedings of the 1981 ACM SIGMOD international conference on Management of data, 1981, Page(s): 143 - 147
[23] Lorry D. Molesky, Krithi Ramamritham. Efficient Locking for Shared Memory Database Systems. Technical Report : University of Massachusetts, 1994
[24] Tobin J. Lehman, Vibby Gottemukkala. The Design and Performance Evaluation of a Lock Manager for a Memory-Resident Database System, Performance of concurrency control mechanisms in centralized database systems, 1996, Page(s): 406 - 428
[25] Qilong Han, Haiwei Pan, Guisheng Yin. A Concurrency Control Algorithm Access to Temporal Data in Real-Time Database Systems. Computer and Computational Sciences, 2008. IMSCCS '08. International Multisymposiums, Oct. 2008, Page(s):168 - 171
[26] Khaled S. Maabreh, Alaa Al-Hamami. Increasing database concurrency control based on attribute level locking. Electronic Design, 2008. ICED 2008. International Conference, Dec. 2008, Page(s):1– 4
[27]朱铭,李航,王力.内存数据库的锁机制.微机发展. 2003,第13卷,第12期
[28] Oracle TimesTen: In-Memory Database Introduction, www.oracle.com/database/in-memory-database-cache.html. 2007
[29] PostgreSQL 8.3.6 Documentation. 2008
[30]董钢. SQL SERVER数据库封锁及死锁研究.计算机与现代化. 2005,第8期
[31]李晶. Sybase数据库系统并发控制.计算机工程. 1999,第11期
[32] Yong Jiang, Jie Li, Shoichi Nishimura. A general stochastic model for dynamic locking in database systems,IEEE Transactions on Computers, 53(3), 2004, Page(s):308 - 319
[33] Anha Mittal, Sivatama P. Dandamudi. Dynamic versus static locking in real-time parallel database systems,2004
[34] Martin Kot. Modeling real-time database concurrency control protocol two-phase-locking in Uppaal,2008
[35]彭正文,徐新爱.基于SMP的Linux内核自旋锁分析.江西教育学院学报. 2005,第3期
[36]韩凯,王京春.实时数据库调度策略综述.计算机工程与应用. 2004,第32期
[37] Y. M. P. Fernandes, A. Perkusich, Pedro F. Ribeiro Neto. Implementation of transactions scheduling for real-time database management. IEEE. 2004
[38] Ravi Sethi. A model of concurrent database transactions. Proceedings of the 22nd Annual Symposium on Foundations of Computer Science, Page(s): 175 - 184
[39]陈传波,王桦.实时数据库的事务调度研究.计算机应用. 2005,第9期
[40]葛俊杰.实时内存数据库的事务调度与数据恢复研究.青岛大学硕士论文. 2007
[41] H. V. Jagadish, Daniel Lieuwen, Rajeev Rastogi et al. Dali: A High Performance Main Memory Storage Manager. Proceedings of the 20th International Conference on Very Large Data Bases, 1994, Page(s):48 - 59
[42] Daniel P. Bovet, Marco Cesati.深入理解LINUX内核(第二版).陈莉君,冯锐,牛欣源等译.中国电力出版社. 2004, 168 - 196.