基于云平台的分布式高性能网络爬虫的研究与设计.PDFVIP

运营技术广角 基于云平台的分布式高性能网络爬虫的研究与设计 石恩名，肖晓军，卢宇 （广州优亿信息科技有限公司，广东广州 510630 ） 摘 要：随着大数据时代的到来，数据成为最宝贵的资源，而网络爬虫技术作为外部数据采集的重要手段， 已然成为数据分析的标配。介绍了一种高性能、灵活和便捷的基于云平台的爬虫架构设计和实现。从爬虫的 整体架构、分布式设计以及各模块的设计等角度进行了详细的阐述。爬虫各模块用 Docker 封装，Kubernetes 做集群的资源调度和管理，在性能优化上采用了MD5 去重树算法、DNS 优化和异步I/O 等多种策略组合的形 式。实验表明，对比未优化的方案，爬虫在性能上具有较明显的优势。 关键词：分布式系统架构；网络爬虫；Docker ；高性能 中图分类号：TP393 文献标识码：A doi: 10.11959/j.issn.1000−0801.2017234 Research and design of distributed high-performance network reptiles based on cloud platform SHI Enming, XIAO Xiaojun, LU Yu Guangzhou Useease Information Technology Co., Ltd., Guangzhou 510630, China Abstract: With the arrival of large data age, data has become the most valuable resource. And web crawler technology as an important means of external data collection, has become a standard tool for data analysis. A high-performance, conve- nient cloud-based crawler architecture design was introduced. The overall structure of the reptile to the distributed design and the design of the sub-module was described in detail. Each module of the crawler was encapsulated in Docker, and Ku- bernetes was used as the resource scheduling and management of the cluster. In the performance of optimization, the MD5 reset tree algorithm, DNS optimization and asynchronous I/O were adopted. Experimental results show that the performance of crawler has obvious advantages compared with the UN optimized scheme. Key words: distributed system architecture, web crawler, Docker, high-performance 据采集手段，既是各类应用的外部数据来源（如 1 引言 内容聚合类应用：今日头条、百度新闻等），也是 大数据时代，数据是最丰富也是最宝贵的， 有哪些信誉好的足球投注网站引擎的重要组成部分。特别随着数据分析技 网络信息量快速增长，面对这样海量的