网站robots写法（Site robots writing）.docVIP

网站robots写法（Site robots writing） The writing of the website robots Webmasters job is to design a beautiful website for the public to show the website rich and colorful content. Of course, we also want well-designed web sites to achieve an ideal ranking, which requires us to study the search engine ranking rules, the greatest degree of access to the customer. However, many types of search engines, sometimes we rank to a search engine is very good, but can not get the same ranking in the search engine, the reason is not the same rules of various search engines. For this reason, someone copied the same content to deal with different search engine ranking rules, and produced a large number of duplicate pages. Once the search engine is found, it will be punished. To this end, the emergence of robots. First, the basic concepts and functions of robots Robots.txt file is a web site file, it is to search engine spiders to see. Search engine spiders crawling our website, first of all, grab this file, according to the content of the site to determine the scope of access to the document. Robots.txt can also be used in a directory. Search the scope of the files in this directory. A few points?: 1) the site must have a robots.txt file. 2) the file name is lowercase. 3) when you need to completely shield files, you need to match the robots attribute of meta. Robots has two main functions: One function is to block duplicate pages Another feature is shielding, such as grabbing pages, such as privacy. For different search engines, show different search paths. What search engines and their corresponding User-agent are there in robots? Below, I listed some for your reference. Search engine User-Agent AltaVista Scooter Baidu Baiduspider Infoseek Infoseek Hotbot Slurp AOL Search Slurp Excite ArchitextSpider Google Googlebot Goto Slurp Lycos Lycos MSN Slurp Netscape Googlebot NorthernLight Gulliver WebCrawler ArchitextSpider Iwon Slurp Fast Fast DirectHit Grabber Yahoo Web Pages Googlebot