Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to [url=http://www.webmasterworld.com/donate.htm ...

Robots.txt Generator from HowRank.com generates your robots.txt file for you. You can even include your SiteMap for better indexing.

Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.

# robots.txt for http://www.w3.org/ # # $Id: robots.txt,v 1.59 2010/01/29 15:52:50 ted Exp $ # # For use by search.w3.org. User-agent: W3C-gsa. Disallow: /Out-Of-Date

Increase your ranking with a poper robotx.txt file.

User-agent: * Disallow: / User-agent: delicious-thumbnails. Allow: / User-agent: Slurp. Allow: / Disallow: /inbox. Disallow: /subscriptions. Disallow: /network

If you care about validation, this robots.txt validator is a tester that will check your robots.txt file searching for syntax errors

robots. txt is a text file which can be used to restrict web robots to accessing your web site only in ways of which you approve. This robots. txt file blocks Google s Imagebot ...

# robots.txt, www.nytimes.com 1/21/2009 # User-agent: * Disallow: /adx/bin/ Disallow: /aponline/ Disallow: /archives/ Disallow: /auth/ Disallow: /cnet/

Interactively create a robots.txt file to indicate which robots you don't want crawling your site, and which files or directories you don't want crawled.