基于Python的新闻聚合系统网络爬虫研究
|
摘要
本文开发了一套基于Python的网络爬虫,并预留API,从而构建一个新闻聚合系统。新闻聚合系统中的新闻数据需要爬虫来获取,然而不同的网站有不同的页面布局,本研究旨在创建一个能够从不同页面布局中提取数据的开源爬虫,其中包括网络爬虫、API、网络爬虫调度器以及Socket服务器的实现等。开发过程中使用Python语言开发网络爬虫,利用Beautiful Soup作为网络爬虫的web提取工具,以Laravel为web应用程序框架,以PHP作为主要后端语言,承载CMS和API。网络爬虫可以通过利用用户创建的配置文件来适应从不同的页面布局中提取数据,并将提取的数据导出到JSON文件或数据库系统中。
This paper develops a Python-based web crawler and reserves the API to construct a news aggregation system. The news data in the news aggregation system needs to be acquired by the web crawler,but different websites have different page layouts. This study aims to create an open source crawler able to extract data from different page layouts,including the implementation of web crawler,API,web crawler scheduler and Socket server. In the development process,Python language is applied to develop the network crawler,Beautiful Soup is used as the web extraction tool of the web crawler,Laravel is adopted as the web application framework,PHP is used as the main back-end language to support CMS and API. Web crawler can adapt to different page layouts and extract data from them by using the configuration files created by users and export the extracted data to JSON file or database system.
This paper develops a Python-based web crawler and reserves the API to construct a news aggregation system. The news data in the news aggregation system needs to be acquired by the web crawler,but different websites have different page layouts. This study aims to create an open source crawler able to extract data from different page layouts,including the implementation of web crawler,API,web crawler scheduler and Socket server. In the development process,Python language is applied to develop the network crawler,Beautiful Soup is used as the web extraction tool of the web crawler,Laravel is adopted as the web application framework,PHP is used as the main back-end language to support CMS and API. Web crawler can adapt to different page layouts and extract data from them by using the configuration files created by users and export the extracted data to JSON file or database system.
引文
[1]范传辉. Python爬虫开发与项目实践[M].北京:机械工业出版社,2017:72-81.
[2]于娟.主题网络爬行研究综述[J].计算机工程与科学,2015(2):231-237.
[3]黄日强,黄勇明.网络爬虫的优化策略探略[J].现代情报,2011(10):31-35.
[4]韦伟.精通Python网络爬虫[M].北京:机械工业出版社,2017:128.
[5]安子健.基于Scrapy框架的网络爬虫实现与数据抓取分析[D].长春:吉林大学,2017.
[2]于娟.主题网络爬行研究综述[J].计算机工程与科学,2015(2):231-237.
[3]黄日强,黄勇明.网络爬虫的优化策略探略[J].现代情报,2011(10):31-35.
[4]韦伟.精通Python网络爬虫[M].北京:机械工业出版社,2017:128.
[5]安子健.基于Scrapy框架的网络爬虫实现与数据抓取分析[D].长春:吉林大学,2017.