Robots.txt
The robots.txt file is a commonly supported format for managing search engine spider access to a website or web directory, by agent name. Commonly, this would be used to prevent content being spidered by search engines.
The robots.txt file can also be used to give the location of sitemaps on a website, that will be spidered by search engines supporting the robots.txt sitemap line - all the major search engines now support this.
Last updated on 13 January 2009, at 21:57.