slides_chap12 Modern Information Retrieval 2e 英文课件.pdf

Modern Information Retrieval Chapter 12 Web Crawling with Carlos Castillo Applications of a Web Crawler Architecture and Implementation Scheduling Algorithms Crawling Evaluation Extensions Examples of Web Crawlers Trends and Research Issues Web Crawling, Modern Information Retrieval, Addison Wesley, 2010 – p. 1 Introduction and a Brief History A Web Crawler is a software for downloading pages from the Web Also known as Web Spider, Web Robot, or simply Bot Cycle of a Web crawling process The crawler start downloading a set of seed pages, that are parsed and scanned for new links The links to pages that have not yet been downloaded are added to a central queue for download later Next, the crawler selects a new page for download and the process is repeated until a stop criterion is met Web Crawling, Modern Information Retrieval, Addison Wesley, 2010 – p. 2 Introduction and a Brief History The first known web crawler was created in 1993 by Matthew Gray, an undergraduate student at MIT On June of that year, Gray sent the following message to the www-talk mailing list: “I have written a perl script that wanders the WWW collecting URLs, keeping tracking of where it’s been and new hosts that it finds. Eventually, after hacking up the code to return some slightly more useful information (currently it just returns URLs), I will produce a searchable index of this.” The project of this Web crawler was called WWWW (World Wide Web Wanderer) It was used mostly for Web characterization studies Web Craw