French | Spanish | Portuguese | Italian | German | Japanese | Chinese | Korean | Russian | Arabic

Your Robots.txt File

A robots.txt file is the first file that a search engine robot visits on your website. Like a snooty nightclub bouncer with a velvet rope, the robots.txt file decides which robots are welcome and which need to move on to that less-exclusive joint down the street. Robots.txt can admit or reject robots on a sitewide, directory-by-directory, or page-bypage basis.

SEO folks often feel a special affection for the robots.txt file because it provides a rare opportunity to communicate with a search engine robot. However, its capabilities are limited. Robots.txt files exist only to exclude indexing. Just as a bouncer can keep people out but can’t force anyone to come in, the robots.txt file can’t do anything to entice a robot to spend more time or visit more pages on your site. Also, compliance with your robots.txt file is voluntary, not mandatory. The major search engines will generally try to follow your instructions, but other, less-reputable types might not. This is why you should not rely on your robots.txt file to prevent spidering of sensitive, private, or inappropriate materials.

Do You Need a Robots.txt File?

You may not need a robots.txt file. Without one, all robots will have free access to nonpassword- protected pages on your site. To decide if you need a robots.txt file for your website, ask yourself these questions:

• Are there any pages or directories on my site that I do not want listed on the search engines, such as an intranet or internal phone list?

• Are there any specific search engines that I do not want to display my site?

• Do I know of any dynamic pages or programming features that might cause problems for spiders, like getting caught in a loop (infinitely bouncing between two pages)?

• Does my website contain pages with duplicate content?

• Are there directories on the site that contain programming scripts only, not viewable pages?

If the answers to these questions are no, then you do not need a robots.txt file. You’ve got the rest of the day off! If you have any yes answers, you’ll prepare your robots.txt file today.

Create Your Robots.txt File

Robots.txt files are very simple text files. To find a sample, go to yourseoplan.com/ robots.txt and view ours, or go to just about any other site and look for the robots.txt file in the root directory.

The robots.txt file usually looks something like this:

   User-agent: googlebot
   Disallow: /private-files/
   Disallow: /more-private-files/
   User-agent: *
   Disallow: /cgi-scripts/

In this example, Google’s spider (called Googlebot) is excluded from indexing files within the two directories called private-files and more-private-files, and all robots (signified by a wild-card asterisk, *) are excluded from indexing the directory called cgi-scripts.

There are numerous websites that will walk you through building and saving your robots.txt file. A helpful robots.txt builder can be found here: clickability.co.uk/ robotstxt.html. Answers to just about any question you could think of about robots are here: www.robotstxt.org.

If you are feeling any doubt about whether your robots.txt file is written properly, don’t post it. The last thing you want to do is inadvertently shut out the search engines.

Here’s a bonus: Robots.txt can also be used to tell search engines where to find your XML Sitemap.

Robots Meta Tags

A robots meta tag serves a similar purpose as the robots.txt file, but it is placed within individual pages on your site rather than in your root directory. A robots meta tag affects only the page it resides on. Chances are you don’t need to use this type of tag, but here’s a quick overview in case you do.

You might choose to use a robots meta tag rather than a robots.txt file because it’s easier for you to set up the exclusion using your web page template rather than the robots.txt file, or maybe you only want to do a brief, temporary exclusion. Another possible reason is that you do not have access to the root directory on your site.

To exclude the robots from a page using the robots meta tag, simply include the following code in the HTML head of the page:

   <meta name=”robots” content=”noindex, nofollow”>

This will prevent search engine robots from listing the page on which the tag resides.

Robot Exclusion for Google

If you plan to use robots exclusion to control the sharing of Google PageRank among pages on your website (for example, by excluding low-quality pages that you do not want hogging authority), you should know that Google handles the robots.txt and robots meta tag exclusions slightly differently:

• Pages excluded with either type of exclusion are allowed to accumulate PageRank authority.

• A page that is excluded with the robots.txt file may be listed in search results with a URL only, and no description. A page that is excluded with a robots meta tag will not be displayed in search results at all.

• A page that is excluded with the robots.txt file will not be crawled by Googlebot, and it will not pass PageRank to other pages to which it links.

• A page that is excluded with the robots meta tag may be crawled by Googlebot, and Google will follow links on the page.The PageRank that is accumulated by this page will be shared with pages to which it links.