|
||||||
Occasionally, a need arises to remove from a search engines database, parts of a website that were mistakenly or incorrectly included in the database, or that have aged.
Google's spider, Googlebot, will obey the first line of a file in which the user-agent is "Googlebot". If there is no such line, the spider will obey the first line in which the user-agent is "*".
Google added flexibility to the standard. Templates of Disallow may also include "*", which means relating to any combination of characters, and "$" at the end, which indicates the end of a name. <exp> (expression) should be replaced with the object that should be removed. User-agent:* In all the search engines
User-agent:Googlebot In Google only
User-agent:* or User-agent:
Disallow: / Removal of a complete site
User-agent:Googlebot
Disallow: /<directory name> Removal of a directory
User-agent:Googlebot
Disallow: /*.<exp>$ Removal of a particular type of file
User-agent:Googlebot
Disallow: /*? Removal of all dynamic pages
Translated by Debi Zylbermann |
||||||
