基于自主容器云平台的大数据日志采集系统
|
摘要
随着云计算、虚拟化、容器云等技术的应用以及国产自主可控的要求,越来越多的服务会部署在国产的服务器以及自主可控的容器云平台上。自主可控云平台上的服务在运行的过程中,需要获取日志数据,来进行一系列的监控、统计、分析以及预测工作。然而由于国产服务器的特殊性以及容器云平台的特殊性,传统的日志采集方式(包含实时采集和集中式采集)并不能够很好地适用于自主可控的容器云平台,为了提高服务的质量和性能并且保证日志数据的一定全面性,本文提出一种自适应的数据采集算法,能够根据服务器的负载自我调整日志数据采集发送量。当服务器负载较低的时候,提高数据采集和发送的数量,甚至是做到实时采集;当服务器负载较高时,降低对日志数据的采集和发送量,从而降低对服务器负载的压力,提高服务自身的质量和性能。最后通过理论和实验分析验证该数据采集算法在保证数据一定的全面性的同时,能有效缓解国产服务器的压力。
With the application of technologies such as cloud computing,virtualization,container cloud,and domestically-controlled and controllable requirements,more and more services will be deployed on domestic servers and self-controlled container cloud. In the process of running the service,we need to obtain log data for a series of monitoring,statistics,analysis and forecasting work. However,due to the particularity of the domestic server and the particularity of self-controlled container cloud,the traditional log collection methods( including real-time acquisition and centralized acquisition) are not well suited for self-controlled container cloud. In order to improve the quality and performance of the service and ensure a certain comprehensiveness of the log data,an adaptive data acquisition algorithm is proposed,which can self-adjust the log data collection and transmission according to the load of the server. When the server load is low,the number of data collection and transmission is increased,even in realtime collection; when the server load is high,the collection and transmission of log data is reduced,thereby reducing the pressure on the server load,and improving the quality and performance of the service itself. Finally,theoretical and experimental analysis proves that the data acquisition algorithm can effectively alleviate the pressure of domestic servers while ensuring a certain comprehensiveness of data.
With the application of technologies such as cloud computing,virtualization,container cloud,and domestically-controlled and controllable requirements,more and more services will be deployed on domestic servers and self-controlled container cloud. In the process of running the service,we need to obtain log data for a series of monitoring,statistics,analysis and forecasting work. However,due to the particularity of the domestic server and the particularity of self-controlled container cloud,the traditional log collection methods( including real-time acquisition and centralized acquisition) are not well suited for self-controlled container cloud. In order to improve the quality and performance of the service and ensure a certain comprehensiveness of the log data,an adaptive data acquisition algorithm is proposed,which can self-adjust the log data collection and transmission according to the load of the server. When the server load is low,the number of data collection and transmission is increased,even in realtime collection; when the server load is high,the collection and transmission of log data is reduced,thereby reducing the pressure on the server load,and improving the quality and performance of the service itself. Finally,theoretical and experimental analysis proves that the data acquisition algorithm can effectively alleviate the pressure of domestic servers while ensuring a certain comprehensiveness of data.
引文
[1]梁洪波,吴玉.网络边界安全与日志文件分析[J].电脑知识与技术,2007,2(9):629-631.
[2]臧建林.利用集中日志服务器管理网络[J].网管员世界,2009(1):59-60.
[3]曾建伟,黄奕维,林伟坚.防火墙日志分析的关键技术[J].教育信息技术,2018(S1):97-98.
[4] KUMAR V,THAKUR R S. Web log analysis tools:At a glance[C]//Proceedings of the 2018 International Conference on Recent Advancement on Computer and Communication. 2018:135-142.
[5]刘斌.基于Hadoop的Web日志分析系统设计[J].安徽科技学院学报,2017,31(4):67-70.
[6]陶兴宇.分布式系统日志数据采集关键技术研究与实现[D].沈阳:沈阳航空航天大学,2016.
[7] JIANG Q F,LUO Y,MANIVANNAN D. An optimistic checkpointing and message logging approach for consistent global checkpoint collection in distributed systems[J].Journal of Parallel and Distributed Computing,2008,68(12):1575-1589.
[8]王静.基于飞腾平台的容器虚拟化技术研究[D].长沙:国防科学技术大学,2013.
[9]马晓光,孙大军,吴登勇,等.容器虚拟化技术在飞腾1500A平台的应用[J].信息技术与信息化,2017(6):39-41.
[10]魏元豪.基于飞腾处理器的存储系统IO调度优化[D].长沙:国防科学技术大学,2014.
[11]孟小甫,高翔,从明,等.龙芯3A多核处理器系统级性能优化与分析[J].计算机研究与发展,2012,49(S1):137-142.
[12]孙永杰.从龙芯性能之殇看国产芯片发展之路[J].通信世界,2015(10):14.
[13]马健.龙芯高性能计算机软件系统的优化研究[D].北京:中国科学院大学,2017.
[14]张寓琛,张小芳. x86服务器虚拟化平台性能测试[J].计算机与现代化,2014(2):32-35.
[15] BLEM E,MENON J,SANKARALINGAM K. Power struggles:Revisiting the RISC vs. CISC debate on contemporary ARM and x86 architectures[C]//Proceedings of the 2013IEEE 19th International Symposium on High Performance Computer Architecture. 2013:1-12.
[16] BOETTIGER C. An introduction to Docker for reproducible research[J]. ACM SIGOPS Operating Systems Review,2015,49(1):71-79.
[17]李战,彭扬.一种基于Docker的容器云资源调度方案[DB/OL].(2017-12-22). http://www. paper. edu. cn/releasepaper/content/201712-295.
[18]王健. Docker技术将引爆容器生态系统[J].软件和集成电路,2016(S1):48-49.
[19]孙海鸣,龚崟,陆钢.容器云在运营商平台中的应用及规划研究[J].广东通信技术,2017,37(5):44-47.
[20]庞希愚,姜波,仝春玲,等.一种自适应数据变化规律的数据采集算法[J].计算机技术与发展,2013,23(2):157-161.
[21]杨军,潘宝军,原鑫. Paa S混搭云构建与节点优化配比分析[J].计算机工程与设计,2015,36(5):1396-1401.
[22]钟磊,沈伟,张建.浅析移动网络中的WAP网关流量压缩技术[J].广西通信技术,2012(4):33-36.
[23] RFC 1157,A Simple Network Management Protocol(SNMP)[S].
[2]臧建林.利用集中日志服务器管理网络[J].网管员世界,2009(1):59-60.
[3]曾建伟,黄奕维,林伟坚.防火墙日志分析的关键技术[J].教育信息技术,2018(S1):97-98.
[4] KUMAR V,THAKUR R S. Web log analysis tools:At a glance[C]//Proceedings of the 2018 International Conference on Recent Advancement on Computer and Communication. 2018:135-142.
[5]刘斌.基于Hadoop的Web日志分析系统设计[J].安徽科技学院学报,2017,31(4):67-70.
[6]陶兴宇.分布式系统日志数据采集关键技术研究与实现[D].沈阳:沈阳航空航天大学,2016.
[7] JIANG Q F,LUO Y,MANIVANNAN D. An optimistic checkpointing and message logging approach for consistent global checkpoint collection in distributed systems[J].Journal of Parallel and Distributed Computing,2008,68(12):1575-1589.
[8]王静.基于飞腾平台的容器虚拟化技术研究[D].长沙:国防科学技术大学,2013.
[9]马晓光,孙大军,吴登勇,等.容器虚拟化技术在飞腾1500A平台的应用[J].信息技术与信息化,2017(6):39-41.
[10]魏元豪.基于飞腾处理器的存储系统IO调度优化[D].长沙:国防科学技术大学,2014.
[11]孟小甫,高翔,从明,等.龙芯3A多核处理器系统级性能优化与分析[J].计算机研究与发展,2012,49(S1):137-142.
[12]孙永杰.从龙芯性能之殇看国产芯片发展之路[J].通信世界,2015(10):14.
[13]马健.龙芯高性能计算机软件系统的优化研究[D].北京:中国科学院大学,2017.
[14]张寓琛,张小芳. x86服务器虚拟化平台性能测试[J].计算机与现代化,2014(2):32-35.
[15] BLEM E,MENON J,SANKARALINGAM K. Power struggles:Revisiting the RISC vs. CISC debate on contemporary ARM and x86 architectures[C]//Proceedings of the 2013IEEE 19th International Symposium on High Performance Computer Architecture. 2013:1-12.
[16] BOETTIGER C. An introduction to Docker for reproducible research[J]. ACM SIGOPS Operating Systems Review,2015,49(1):71-79.
[17]李战,彭扬.一种基于Docker的容器云资源调度方案[DB/OL].(2017-12-22). http://www. paper. edu. cn/releasepaper/content/201712-295.
[18]王健. Docker技术将引爆容器生态系统[J].软件和集成电路,2016(S1):48-49.
[19]孙海鸣,龚崟,陆钢.容器云在运营商平台中的应用及规划研究[J].广东通信技术,2017,37(5):44-47.
[20]庞希愚,姜波,仝春玲,等.一种自适应数据变化规律的数据采集算法[J].计算机技术与发展,2013,23(2):157-161.
[21]杨军,潘宝军,原鑫. Paa S混搭云构建与节点优化配比分析[J].计算机工程与设计,2015,36(5):1396-1401.
[22]钟磊,沈伟,张建.浅析移动网络中的WAP网关流量压缩技术[J].广西通信技术,2012(4):33-36.
[23] RFC 1157,A Simple Network Management Protocol(SNMP)[S].