Robots.txt

Webbsidor

Robotstxt.org
Jul 16, 2009 ... Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
www.robotstxt.org

A Standard for Robot Exclusion - The Web Robots Pages
Jul 16, 2009 ... This file must be accessible via HTTP on the local URL " /robots.txt ". The contents of this file are specified below. ...
www.robotstxt.org

Robots exclusion standard - Wikipedia, the free encyclopedia
The robots.txt standard was developed in 1994, when large-scale web indexing became popular; indexers such as Lycos and AltaVista used it. ...
en.wikipedia.org

is the robots.txt of Google - Google
User-agent: * Disallow: /search Disallow: /groups Disallow: /images Disallow: / catalogs Disallow: /catalogues Disallow: /news Allow: /news/directory ...
www.google.com