无线传感器网络能量高效的传输策略研究
|
摘要
无线传感器网络技术作为物联网的关键技术之一,正引领着一次信息技术革命。作为无线传感器网络技术发展的“瓶颈”,节点能量有限一直制约着传感器网络在各领域的应用。根据现有理论和实际应用得到数据显示,传感器节点的有限能量大部分消耗在数据传输上。因此,如何改善传感器网络中的数据传输策略,避免不必要的能量消耗,从而高效利用传感器节点有限的能量,已经成为一个重要的研究课题。
论文一方面针对当前大多数无线传感器网络传输协议不能高效利用传感器节点能量的问题,提出了一种基于信道和缓冲区状态的自适应传输方法,包括基于信道和缓冲区的传输机制(Channel and Buffer Based Transmission,简记为CBT)和基于信道和缓冲区的分片传输机制(Channel and Buffer Based Fragment Transmission,简记CBFT),CBFT在CBT的基础上结合虚拟分片传输技术。并将节点的数据传输问题建模成马尔可夫决策过程(Markov decision process,简记MDP)模型,用Q学习算法进行求解。仿真结果表明,传感器节点能量得到高效利用,无线传感器网络的寿命也在一定程度上得到延长。
另一方面,在很多情况下,无线传感器网络中的信道状态信息并不完全可知,只能通过观察得到部分信息。论文在观察上一次传输反馈信息的基础上,建立了基于离散时间POMDP模型的数据传输机制。并利用Linear-Q学习算法进行求解。仿真结果表明,在信息部分可观情况下,基于POMDP模型的传输机制能较好的利用有限信息进行数据传输控制,并能在一定程度上提高系统吞吐量和减少缓冲区溢出。
The wireless sensor network technology, as one of the key technologies of internet of things, is leading a revolution of information technology. However, the energy of sensor network node is limited, which is a“bottleneck”of the wireless sensor network technology and has restricted it’s applications in various fields. According to current theories and practical applications, most energy of the sensor network node is consumed in transmission. Therefore, how to improve the transmission strategy, which can avoid unnecessary energy consumption and use the energy efficiently, became an important research topic.
On the one hand, most current transmission protocols can not achieve energy-efficient of Wireless Sensor Networks (WSNs), an energy-efficient adaptive transmission based on channel and buffer state is proposed, including Channel and Buffer Based Transmission (CBT) and Channel and Buffer Based Fragment Transmission (CBFT). The adaptive transmission based on the current channel state to decide whether to transfer, avoid the energy waste caused by failed transmission. CBFT based on CBT, and combine with virtual fragment transmission technology. Data transfer problem of sensor node is modeled as Markov decision process, Q learning algorithm is proposed to solve the problem. The simulation results show that the usage of sensor node’s energy of sensor node is efficient. The lifetime of wireless sensor network also can be prolonged.
On the other hand, wireless sensor network channel side information is not fully known often, and only partially information can only be observed. In this paper, based on the observation of information which feedbacked from last transmission, the data transmission problem can established as a discrete time partially observable Markov decision processes model. Linear-Q learning algorithm is proposed to solve the problem. The simulation results also show that the transmission scheduling based on partially observable Markov decision processes model can improve the throughput and reduce the buffer overflow.
论文一方面针对当前大多数无线传感器网络传输协议不能高效利用传感器节点能量的问题,提出了一种基于信道和缓冲区状态的自适应传输方法,包括基于信道和缓冲区的传输机制(Channel and Buffer Based Transmission,简记为CBT)和基于信道和缓冲区的分片传输机制(Channel and Buffer Based Fragment Transmission,简记CBFT),CBFT在CBT的基础上结合虚拟分片传输技术。并将节点的数据传输问题建模成马尔可夫决策过程(Markov decision process,简记MDP)模型,用Q学习算法进行求解。仿真结果表明,传感器节点能量得到高效利用,无线传感器网络的寿命也在一定程度上得到延长。
另一方面,在很多情况下,无线传感器网络中的信道状态信息并不完全可知,只能通过观察得到部分信息。论文在观察上一次传输反馈信息的基础上,建立了基于离散时间POMDP模型的数据传输机制。并利用Linear-Q学习算法进行求解。仿真结果表明,在信息部分可观情况下,基于POMDP模型的传输机制能较好的利用有限信息进行数据传输控制,并能在一定程度上提高系统吞吐量和减少缓冲区溢出。
The wireless sensor network technology, as one of the key technologies of internet of things, is leading a revolution of information technology. However, the energy of sensor network node is limited, which is a“bottleneck”of the wireless sensor network technology and has restricted it’s applications in various fields. According to current theories and practical applications, most energy of the sensor network node is consumed in transmission. Therefore, how to improve the transmission strategy, which can avoid unnecessary energy consumption and use the energy efficiently, became an important research topic.
On the one hand, most current transmission protocols can not achieve energy-efficient of Wireless Sensor Networks (WSNs), an energy-efficient adaptive transmission based on channel and buffer state is proposed, including Channel and Buffer Based Transmission (CBT) and Channel and Buffer Based Fragment Transmission (CBFT). The adaptive transmission based on the current channel state to decide whether to transfer, avoid the energy waste caused by failed transmission. CBFT based on CBT, and combine with virtual fragment transmission technology. Data transfer problem of sensor node is modeled as Markov decision process, Q learning algorithm is proposed to solve the problem. The simulation results show that the usage of sensor node’s energy of sensor node is efficient. The lifetime of wireless sensor network also can be prolonged.
On the other hand, wireless sensor network channel side information is not fully known often, and only partially information can only be observed. In this paper, based on the observation of information which feedbacked from last transmission, the data transmission problem can established as a discrete time partially observable Markov decision processes model. Linear-Q learning algorithm is proposed to solve the problem. The simulation results also show that the transmission scheduling based on partially observable Markov decision processes model can improve the throughput and reduce the buffer overflow.
引文
[1] Estrin D, Govindan R, Heidemann J, Kumar S. Next century challenges: Scalable coordination in sensor networks. Proceedings of Mobieom, Seattle, USA, 1999, 263-270.
[2] Akyildiz I F, Su W, Sankarasubramaniam Y, et al A survey on sensor networks. Communieations, 2002, 40(8), 102-114
[3] Tubaishat M, Madria S. Sensor networks: an overview. Potentials, 2003, 22(2): 20-22.
[4] Cullar D, Estrin D, Strvastava M. Overview of sensor network. Computer, 2004, 37(8): 41-49.
[5] Yu H Y, Li O, Zhang X Y, et al . Wireless sensor networks theory. Technique and Implementation. Beijing: National Defense Industry Press, 2008, 355.
[6] Yick J, Mukherjee B, Ghosal D. Wireless sensor network survey. Computer Networks, 2008, 52(12): 2292-2330.
[7] Gao T, Greenspan D, Welsh M, Juang R, Alm R. Vital signs monitoring and patient tracking over a wireless network. Engineering in Medicine and Biology Society, 2005, 102-105.
[8]崔莉,鞠海玲,苗勇等.无线传感器网络研究进展.计算机研究与发展.2005, 42(1):163-174.
[9]吉林,丁华平,沈庆宏.基于无线传感器网络的桥梁结构健康监测.南京大学学报(自然科学版).2011, 47(1): 19-24.
[10] Bojkovic Z, Bakmaz B. A survey on wireless sensornetworks deployment. World Scientific and Engineering Academy and Society, 2008, 7(12): 1172-1181.
[11] Dennis. John L, Brendan B, O’Flynn. The Development of Environmentally Tested Antennas for Wireless Sensor Networks. EmNets'07, 2007, 6: 73-77.
[12] Arici T, Altunbask Y. Adaptive sensing for environment monitoring using wireless sensor networks. Wireless Communications and Networking Conference, Atlanta, GA, 2004.
[13] Mainwaring A, Polastre J, Szewezyk R, et al. Wireless sensor networks for habitual monitoring. The lst ACM International workshop on Wireless Sensor Networks and Applications, Atlanta, USA, 2002, 143-151.
[14]任丰原,黄海宁,林闯.无线传感器网络.软件学报, 2003, 14(7): 1282-1291.
[15]孙利民,李建中,陈渝,朱红松.无线传感器网络[M].北京:清华大学出版社,2005.
[16] Jaikaeo C, Srisathapornphat C, Shen C. Sensor information networking architectureand applications .Personal Communications, 2001, 8(4) :52-59 .
[17]王殊,阎毓杰,胡富平.无线传感器网络的理论及应用[M].北京:北京航空航天大学出版社, 2007
[18]许小丰.无线传感器网络路由信任评估与数据控制传输关键技术研究.北京邮电大学博士学位论文,北京, 2010.
[19] Kohvakka M, Suhonen J, Kuorilehto M et al. Energy-efficient neighbor discovery protocol for mobile wireless sensor networks. Ad Hoc Networks, 2009, 2: 24-41.
[20] E.Egea-Lo′pez J, Vales-Alonso A S, Mart?′nez-Sala et al. A wireless sensor networks MAC protocol for real-time applications. Pers Ubiquit Comput, 2008, 12: 111–122.
[21] Sohrabi K, Pottie G J. Performance of a novel self-organization protocol for wireless Ad hoc sensor networks. The IEEE 50th Vehicular Technology Conference. Amsterdam, 1999. 1222-1226.
[22] Sinhua A, Chandrakasan A. Dynamic power management in wireless sensor networks .Design & Test of Computers, 2001, 18(2): 62-74.
[23] Ye F, Yang H, Liu Z. Catching molesin sensor networks. Int’l Conf on Distributed Computing Systems. Toronto, Canada, 2007.
[24] Yu Y, Viktor K, Prasanna. Energy-Balanced task allocation for collaborative processing in wireless sensor networks. Mobile Networks and Applications, 2005, 10(1-2).
[25] Wei Y, Yu Z, Guan Y. Location verification algorithms for wireless sensor networks. Int’l Conf on Distributed Computing Systems. Toronto, Canada. 2007.
[26] Niculeseu D, Nath B. Ad Hoc positioning system (APS) using AOA. INFOCOM2003, SanFraneiseo, USA, 2003, 1734-1743.
[27] Cheng X Z, Thaeler A, Xue G L, etal. TPS: a time-based positioning seheme for outdoor wireless sensor networks. INF0C0M 2004, Hongkong, China, 2004, 2685-2696.
[28] Jian Y, Chen S G, Zhang Z, eat al. Protceting reeeiver-loeation privaey in wireless sensor network. IINFOCOM2007, Alaska, USA, 2007, 1955-1963.
[29] Sadler C, Martonosi M. Data compression algorithms for energy-constrained devicesin delay tolerant networks. ACM Confereneeon Embedded Networked Sensor Systems Colorado, USA, 2006, 265-278.
[30] Dietrich I, Dressler F. On the lifetime of wireless sensor networks. Trans. Sen.Netw. 2009, 5(1): 5-39.
[31]方维维,钱德沛,刘轶.无线传感器网络传输控制协议.软件学报, 2008, 19(6): 1439-1451.
[32] Iqbal A, Shahzad K, Khayam S A. SRVF: an energy-efficient link layerprotocol for reliable transmission over wireless sensor networks. International Conference on Communications, 2008, 146-150.
[33] Ci S, Sharif H, Nuli K. Study of an adaptive frame size predictor to enhance energy conservation in wireless sensor networks. Selected Areas in Communications,2005,23(2) : 282-292.
[34] Wang J, Zhai H, Fang Y. Opportunistic packet scheduling and media access control for wireless LANs and multi-hop ad hoc networks. Wireless Communications and Networking Conference, 2004, 3:1234-1239.
[35] Holland G, Vaidya N H, Bahl P. A rate-adaptive MAC protocol for multi-hop wireless networks. Mobile Computing and Networking,2001, 236-251 .
[36] Wang H S, Moayeri N. Finite-state Markov channel-a useful model for radio communication channels. Vehicular Technology, 1995, 44(1): 163-171.
[37] Berry R A, Gallager R G. Communication over fading channels with delay constraints. Inform. Theory, 2002, 48(5):1135–1149.
[38] Hoang A T, Motani M. Buffer and channel adaptive transmission over fading channels with imperfect channel state information. Wireless Communications and Networking Conference, 2004, 3:1891-1896.
[39] Phan C V, Kim J G.: An energy-efficient transmission strategy for wireless sensor networks. Wireless Communications and Networking Conference, 2007, 3408-3413.
[40] Srivastava R, Koksal C E. Energy optimal transmission scheduling in wireless sensor networks. Wireless Communications, 2010, 9(5):1650-1660.
[41] Johnston L A, Krishnamurthy V. Opportunistic file transfer over a fading channel: a POMDP search theory formulation with optimalthreshold policies. Wireless Communications, 2006, 5(2): 394–405.
[42] Karmokar A K, Djonin D. V, Bhargava V K. POMDP-based coding rate adaptation for type-I hybrid ARQ systems over fadingchannels with memory. Wireless Communications, 2006, 5(12):3512–3523.
[43] Djonin D. V, Karmokar A K, Bhargava V K, Joint rate and power adaptation for type-I hybrid ARQ systems over correlated fadingchannels under different buffer-cost constraints. Veh.Technol, 2008, 57(1):421–435.
[44] Ho C K, Oostveen J. Rate adaptation in time varying channel susing acknowledgement feedback. VTC’06, 2006, 1683–1687.
[45] Minsky M L. Theory of neural analog reinforcement system and its application to the brain model problem. New Jersey, USA Princeton Univ, 1954.
[46]高阳,陈世福,陆鑫.强化学习研究综述.自动化学报, 2004, 1(30): 86-100.
[47] Sutton R S, Barto A G. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
[48] Watkins C J C H, Dayan P. Q-learning. Machine Learning, 1992, 8:279-292.
[49] Rummery G, Niranjan M.“On-line Q-learning using connectionist systems”, Tech. Rep. Technical Report CUED/F-INFENG/TR 166, Cambridge, University Enginnering Department, 1994.
[50] Sutton R S, Barto A G. Williams R J. Reinforcement learning is direct adaptive optimal control. In Proceedings of IEEE American Control Conference, 1991, 2143-2146.
[51] Cao X R. Single sample path-based optimization of Markov chains. Journal of Optimization Theory and Applications, 1999, 100(3):527-548.
[52]胡奇英,刘建庸.马尔可夫决策过程引论.西安电子科技大学出版社, 2000.
[53] Cao X R, Chen H F. Perturbation Realization, Potentials and Sensitivity Analysis of Markov Processes. IEEE Trans. on Automatic Control, 1997, 42(10): 1382-1393.
[54] Cao X R. The Relations among Potentials, Perturbation Analysis, and Markov Decision Processes. Discrete Event Dynamic Systems: Theory and Applications, 1998, 8(1): 71-78.
[55]周雷.折扣和平均准则下SMDP基于性能势的统一强化学习算法.合肥工业大学硕士学位论文,合肥, 2006.
[56] Drake A. Observation of a Markov process through a noisy channel. Sc.D.Thesis, Mass., 1962.
[57] Cassandra A R. A survey of POMDP applications. AAAI fall symposium on planning with partially observable Markov decision processes, 1998, 17-24.
[58]桂林,武小悦.部分可观测马尔可夫决策过程算法综述.系统工程与电子技术, 2008, 30(6):1058-1064.
[59] Sondik E J. The optimal control of partially observable Markov processes. Department of Electrical Engineering, Stanford, CA, 1971.
[60] Cheng H. Algorithms for partially observed Markov decision processes. Faculty of Commerce and Business Administration, University of British Columbia, 1988.
[61] Littman M L, Cassandra A R, Kaelbling L P. Efficient dynamic programming updates in partially observable Markov decision processes. Technical ReportCS9519, Brown University, 1995.
[62] Cassandra A R, Littman M L, Zhang N L. Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. The Thirteenth Annual Conference on Uncertainty in Artificial Intelligence, Providence, Rhode Island, 1997.
[63] Zhou R, Hansen E. An improved grid based approximation algorithm for POMDPs. The 17th International Joint Conference on Artificial Intelligence, Washington, 2001, 707-716.
[64] Littman M L. Memoryless policies: theoretical limit ations and practical results. The Third International Conference on Simulation of Adaptive Behavior , Cambridge, MA , 1994
[65] Simmons R, Koenig S. Probabilistic robot navigation in partially observable environments. The International Joint Conference on Artificial Intelligence, 1995, 1080-1087.
[66] Littman M L, Cassandra A R, Kaelbling L P. Learning policies for partially observable environments: Scaling up. The Twelfth International Conference on Machine Learning, San Francisco, CA , Morgan Kaufmann , 1995, 362-370.
[67] Tang H, Yuan J B, Lu Y, Chen W J. Performance potential-based neuro-dynamic programming for SMDPs. Acta Automatic Sinica, 2005,31 (4):642-645.
[68] Anh T H, and Mehul M. Cross-layer adaptive transmission: optimal strategies in fading channels. Communications, 2008, 56(5): 799-807.
[2] Akyildiz I F, Su W, Sankarasubramaniam Y, et al A survey on sensor networks. Communieations, 2002, 40(8), 102-114
[3] Tubaishat M, Madria S. Sensor networks: an overview. Potentials, 2003, 22(2): 20-22.
[4] Cullar D, Estrin D, Strvastava M. Overview of sensor network. Computer, 2004, 37(8): 41-49.
[5] Yu H Y, Li O, Zhang X Y, et al . Wireless sensor networks theory. Technique and Implementation. Beijing: National Defense Industry Press, 2008, 355.
[6] Yick J, Mukherjee B, Ghosal D. Wireless sensor network survey. Computer Networks, 2008, 52(12): 2292-2330.
[7] Gao T, Greenspan D, Welsh M, Juang R, Alm R. Vital signs monitoring and patient tracking over a wireless network. Engineering in Medicine and Biology Society, 2005, 102-105.
[8]崔莉,鞠海玲,苗勇等.无线传感器网络研究进展.计算机研究与发展.2005, 42(1):163-174.
[9]吉林,丁华平,沈庆宏.基于无线传感器网络的桥梁结构健康监测.南京大学学报(自然科学版).2011, 47(1): 19-24.
[10] Bojkovic Z, Bakmaz B. A survey on wireless sensornetworks deployment. World Scientific and Engineering Academy and Society, 2008, 7(12): 1172-1181.
[11] Dennis. John L, Brendan B, O’Flynn. The Development of Environmentally Tested Antennas for Wireless Sensor Networks. EmNets'07, 2007, 6: 73-77.
[12] Arici T, Altunbask Y. Adaptive sensing for environment monitoring using wireless sensor networks. Wireless Communications and Networking Conference, Atlanta, GA, 2004.
[13] Mainwaring A, Polastre J, Szewezyk R, et al. Wireless sensor networks for habitual monitoring. The lst ACM International workshop on Wireless Sensor Networks and Applications, Atlanta, USA, 2002, 143-151.
[14]任丰原,黄海宁,林闯.无线传感器网络.软件学报, 2003, 14(7): 1282-1291.
[15]孙利民,李建中,陈渝,朱红松.无线传感器网络[M].北京:清华大学出版社,2005.
[16] Jaikaeo C, Srisathapornphat C, Shen C. Sensor information networking architectureand applications .Personal Communications, 2001, 8(4) :52-59 .
[17]王殊,阎毓杰,胡富平.无线传感器网络的理论及应用[M].北京:北京航空航天大学出版社, 2007
[18]许小丰.无线传感器网络路由信任评估与数据控制传输关键技术研究.北京邮电大学博士学位论文,北京, 2010.
[19] Kohvakka M, Suhonen J, Kuorilehto M et al. Energy-efficient neighbor discovery protocol for mobile wireless sensor networks. Ad Hoc Networks, 2009, 2: 24-41.
[20] E.Egea-Lo′pez J, Vales-Alonso A S, Mart?′nez-Sala et al. A wireless sensor networks MAC protocol for real-time applications. Pers Ubiquit Comput, 2008, 12: 111–122.
[21] Sohrabi K, Pottie G J. Performance of a novel self-organization protocol for wireless Ad hoc sensor networks. The IEEE 50th Vehicular Technology Conference. Amsterdam, 1999. 1222-1226.
[22] Sinhua A, Chandrakasan A. Dynamic power management in wireless sensor networks .Design & Test of Computers, 2001, 18(2): 62-74.
[23] Ye F, Yang H, Liu Z. Catching molesin sensor networks. Int’l Conf on Distributed Computing Systems. Toronto, Canada, 2007.
[24] Yu Y, Viktor K, Prasanna. Energy-Balanced task allocation for collaborative processing in wireless sensor networks. Mobile Networks and Applications, 2005, 10(1-2).
[25] Wei Y, Yu Z, Guan Y. Location verification algorithms for wireless sensor networks. Int’l Conf on Distributed Computing Systems. Toronto, Canada. 2007.
[26] Niculeseu D, Nath B. Ad Hoc positioning system (APS) using AOA. INFOCOM2003, SanFraneiseo, USA, 2003, 1734-1743.
[27] Cheng X Z, Thaeler A, Xue G L, etal. TPS: a time-based positioning seheme for outdoor wireless sensor networks. INF0C0M 2004, Hongkong, China, 2004, 2685-2696.
[28] Jian Y, Chen S G, Zhang Z, eat al. Protceting reeeiver-loeation privaey in wireless sensor network. IINFOCOM2007, Alaska, USA, 2007, 1955-1963.
[29] Sadler C, Martonosi M. Data compression algorithms for energy-constrained devicesin delay tolerant networks. ACM Confereneeon Embedded Networked Sensor Systems Colorado, USA, 2006, 265-278.
[30] Dietrich I, Dressler F. On the lifetime of wireless sensor networks. Trans. Sen.Netw. 2009, 5(1): 5-39.
[31]方维维,钱德沛,刘轶.无线传感器网络传输控制协议.软件学报, 2008, 19(6): 1439-1451.
[32] Iqbal A, Shahzad K, Khayam S A. SRVF: an energy-efficient link layerprotocol for reliable transmission over wireless sensor networks. International Conference on Communications, 2008, 146-150.
[33] Ci S, Sharif H, Nuli K. Study of an adaptive frame size predictor to enhance energy conservation in wireless sensor networks. Selected Areas in Communications,2005,23(2) : 282-292.
[34] Wang J, Zhai H, Fang Y. Opportunistic packet scheduling and media access control for wireless LANs and multi-hop ad hoc networks. Wireless Communications and Networking Conference, 2004, 3:1234-1239.
[35] Holland G, Vaidya N H, Bahl P. A rate-adaptive MAC protocol for multi-hop wireless networks. Mobile Computing and Networking,2001, 236-251 .
[36] Wang H S, Moayeri N. Finite-state Markov channel-a useful model for radio communication channels. Vehicular Technology, 1995, 44(1): 163-171.
[37] Berry R A, Gallager R G. Communication over fading channels with delay constraints. Inform. Theory, 2002, 48(5):1135–1149.
[38] Hoang A T, Motani M. Buffer and channel adaptive transmission over fading channels with imperfect channel state information. Wireless Communications and Networking Conference, 2004, 3:1891-1896.
[39] Phan C V, Kim J G.: An energy-efficient transmission strategy for wireless sensor networks. Wireless Communications and Networking Conference, 2007, 3408-3413.
[40] Srivastava R, Koksal C E. Energy optimal transmission scheduling in wireless sensor networks. Wireless Communications, 2010, 9(5):1650-1660.
[41] Johnston L A, Krishnamurthy V. Opportunistic file transfer over a fading channel: a POMDP search theory formulation with optimalthreshold policies. Wireless Communications, 2006, 5(2): 394–405.
[42] Karmokar A K, Djonin D. V, Bhargava V K. POMDP-based coding rate adaptation for type-I hybrid ARQ systems over fadingchannels with memory. Wireless Communications, 2006, 5(12):3512–3523.
[43] Djonin D. V, Karmokar A K, Bhargava V K, Joint rate and power adaptation for type-I hybrid ARQ systems over correlated fadingchannels under different buffer-cost constraints. Veh.Technol, 2008, 57(1):421–435.
[44] Ho C K, Oostveen J. Rate adaptation in time varying channel susing acknowledgement feedback. VTC’06, 2006, 1683–1687.
[45] Minsky M L. Theory of neural analog reinforcement system and its application to the brain model problem. New Jersey, USA Princeton Univ, 1954.
[46]高阳,陈世福,陆鑫.强化学习研究综述.自动化学报, 2004, 1(30): 86-100.
[47] Sutton R S, Barto A G. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
[48] Watkins C J C H, Dayan P. Q-learning. Machine Learning, 1992, 8:279-292.
[49] Rummery G, Niranjan M.“On-line Q-learning using connectionist systems”, Tech. Rep. Technical Report CUED/F-INFENG/TR 166, Cambridge, University Enginnering Department, 1994.
[50] Sutton R S, Barto A G. Williams R J. Reinforcement learning is direct adaptive optimal control. In Proceedings of IEEE American Control Conference, 1991, 2143-2146.
[51] Cao X R. Single sample path-based optimization of Markov chains. Journal of Optimization Theory and Applications, 1999, 100(3):527-548.
[52]胡奇英,刘建庸.马尔可夫决策过程引论.西安电子科技大学出版社, 2000.
[53] Cao X R, Chen H F. Perturbation Realization, Potentials and Sensitivity Analysis of Markov Processes. IEEE Trans. on Automatic Control, 1997, 42(10): 1382-1393.
[54] Cao X R. The Relations among Potentials, Perturbation Analysis, and Markov Decision Processes. Discrete Event Dynamic Systems: Theory and Applications, 1998, 8(1): 71-78.
[55]周雷.折扣和平均准则下SMDP基于性能势的统一强化学习算法.合肥工业大学硕士学位论文,合肥, 2006.
[56] Drake A. Observation of a Markov process through a noisy channel. Sc.D.Thesis, Mass., 1962.
[57] Cassandra A R. A survey of POMDP applications. AAAI fall symposium on planning with partially observable Markov decision processes, 1998, 17-24.
[58]桂林,武小悦.部分可观测马尔可夫决策过程算法综述.系统工程与电子技术, 2008, 30(6):1058-1064.
[59] Sondik E J. The optimal control of partially observable Markov processes. Department of Electrical Engineering, Stanford, CA, 1971.
[60] Cheng H. Algorithms for partially observed Markov decision processes. Faculty of Commerce and Business Administration, University of British Columbia, 1988.
[61] Littman M L, Cassandra A R, Kaelbling L P. Efficient dynamic programming updates in partially observable Markov decision processes. Technical ReportCS9519, Brown University, 1995.
[62] Cassandra A R, Littman M L, Zhang N L. Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. The Thirteenth Annual Conference on Uncertainty in Artificial Intelligence, Providence, Rhode Island, 1997.
[63] Zhou R, Hansen E. An improved grid based approximation algorithm for POMDPs. The 17th International Joint Conference on Artificial Intelligence, Washington, 2001, 707-716.
[64] Littman M L. Memoryless policies: theoretical limit ations and practical results. The Third International Conference on Simulation of Adaptive Behavior , Cambridge, MA , 1994
[65] Simmons R, Koenig S. Probabilistic robot navigation in partially observable environments. The International Joint Conference on Artificial Intelligence, 1995, 1080-1087.
[66] Littman M L, Cassandra A R, Kaelbling L P. Learning policies for partially observable environments: Scaling up. The Twelfth International Conference on Machine Learning, San Francisco, CA , Morgan Kaufmann , 1995, 362-370.
[67] Tang H, Yuan J B, Lu Y, Chen W J. Performance potential-based neuro-dynamic programming for SMDPs. Acta Automatic Sinica, 2005,31 (4):642-645.
[68] Anh T H, and Mehul M. Cross-layer adaptive transmission: optimal strategies in fading channels. Communications, 2008, 56(5): 799-807.