Efficient Crawling Through URL Ordering高效爬行通过网址订购.pptVIP

Efficient Crawling Through URL Ordering What is a crawler? Program that automatically retrieves pages from the Web. Widely used for search engines. Challenges There are many pages out on the Web. (Major search engines indexed more than 100M pages) The size of the Web is growing enormously. Most of them are not very interesting ? In most cases, it is too costly or not worthwhile to visit the entire Web space. Good crawling strategy Make the crawler visit “important pages” first. Save network bandwidth Save storage space and management cost Serve quality pages to the client application Outline Importance metrics : what are important pages? Crawling models : How is crawler evaluated? Experiments Conclusion Future work Importance metric The metric for determining if a page is HOT Similarity to driving query Location Metric Backlink count Page Rank Similarity to a driving query Importance is measured by closeness of the page to the topic (e.g. the number of the topic word in the page) Personalized crawler Importance metric The metric for determining if a page is HOT Similarity to driving query Location Metric Backlink count Page Rank Backlink-based metric Backlink count number of pages pointing to the page Citation metric Page Rank weighted backlink count weight is iteratively defined Ordering metric The metric for a crawler to “estimate” the importance of a page The ordering metric can be different from the importance metric Crawling models Crawl and Stop Keep crawling until the local disk space is full. Limited buffer crawl Keep crawling until the whole web space is visited throwing out seemingly unimportant pages. Crawl and stop model Crawling models Crawl and Stop Keep crawling until the local disk space is full. Limited buffer crawl Keep crawling until the whole web space is visited throwing out seemingly unimportant pages. Limited buffer model Architecture Experiments Backlink-based importance metric backlink count