robots.txt语法（机器人。txt语法）.docVIP

robots.txt语法（机器人。txt语法） Talk about the ROBOTS.TXT syntax and function We know that the search engine has its own search robot (ROBOTS), and through the ROBOTS link in the network along on the web (usually HTTP and Src link) continue to grab data to build their own database. For web site managers and content providers, sometimes there will be some of the content of the site, do not want to be ROBOTS grab and open. In order to solve this problem, the ROBOTS development community provides two ways: one is robots.txt, another is The Robots META tags. A, robots.txt 1, what is the robots.txt? Robots.txt is a text file, the website does not want to be part of the robots access, declared in this file so that the site is all or part of the content can not be included in search engines, or specify the search engine included only the specified content. When a search robot to visit a web site, it will first check whether robots.txt exists, the site root directory if it is found, the robot will search range according to the contents of a file to determine access, if the file does not exist, then the search robot crawls along the link. Robots.txt must be placed in the root directory of a site, and the file name must be all lowercase. The website URL The corresponding robots.txt URL Http:// Http:///robots.txt Http:/// Http:///robots.txt Http://:80/ Http://:80/robots.txt Http://:1234/ Http://:1234/robots.txt Http:/// Http:///robots.txt 2, the syntax of robots.txt The robots.txt file contains one or more records, these records by separate blank lines (CR, CR/NL, or, NL as the terminator), each record format is as follows: :. In this file can be used for # notes, the use of specific methods and UNIX in practice. The file records usually start with one or more rows of User-agent, followed by the number of Disallow, the details are as follows: User-agent: This value is used to describe the search engine robot name in the robots.txt file, if there are multiple User-agent records that ha