Crawling Web Pages with Support for.pptVIP

Crawling Web Pages with Support forClient-Side Dynamism Manuel álvarez, Alberto Pan, Juan Raposo, and Justo Hidalgo Department of Information and Communications Technologies, University of A Coru?a, Denodo Technologies Inc Advances in Web-Age Information Management 7th International Conference, WAIM 2006, Hong Kong, China, June 17-19, 2006. Proceedings 林子熠 Outline Introduction The case for client-side hidden web The problem with conventional crawlers Our approach Introduction to NSEQL The crawling engine Dealing with sessions: routes Mini-browsers as crawling process System architecture/ basic functioning Algorithm for generating new routes Related work and conclusion Introduction The “Hidden Web” or “Deep Web” is usually defined as the part of WWW documents that is dynamically generated. Client-side hidden web techniques are concerned about accessing content dynamically generated in the client web browser The Problem with Conventional Crawlers Client-Side scripting languages Session maintenance mechanisms Redirections Applets and flash code Other issues Our Approach The crawling processes are based on automated “mini web browsers”, built using standard browser APIs( Web Browser Control), which understand NSEQL . To implement special algorithms to manage the process of generating new “routes to crawl” from a web page. The system uses the concept of route to a document, which can be seen as a generalization of URL, to solve the problem of session maintenance. The system also includes some functionality to access pages hidden behind forms Introduction to NSEQL NSEQL is a language to declaratively define sequences of events on the interface provided by a web browser. An example of an NSEQL program The Crawling Engine The basic idea: a list of routes The list is initialized with a list of routes A crawling process picks a route from the list, downloads its associated document and analyzes it. Obtain new routes from the document’s anchors, which are then add