Scripting:

What is a Robots.txt File?

Robots.txt is a plain text file that Search Engines use to see if there are areas within your Website which should not be indexed.

This file must be placed in your 'www' directory in order for a search engine to see it.

The sample below tells search engines not to index pages in the specified folders.

---- robots.txt (contents follow)

User-agent: *

Disallow: /cgi-bin/

Disallow: /images

Disallow: /secret

Disallow: /members