Create Robots.txt File – Robots Text File

2010
03.16

Why do I need a robots.txt file?

Creating a robots.txt file will not improve your search engine positioning, but it does provide robots with information concerning which files you will not allow to be crawled and indexed in the search engines.

When a robot crawls your site it looks for the  robots.txt file. If it doesn’t find one it assumes automatically that it may crawl and index the entire site. Not having a robots.txt file can also create unnecessary 404 errors in your server logs, making it more difficult to track “real” 404 errors.

Assuming you want your entire site indexed and only want to stop the unnecessary 404 errors from occurring you have a couple of options.

  • Upload a blank robots.txt file to the root directory of your domain.
  • Upload a simple robots.txt file to the root directory of your domain.

What is a simple robots.txt file?

Please note: This will allow all robots to crawl and index all files.

This allows all robots to crawl all files.

User-agent: *
Disallow:

What if I don’t want a particular file crawled?

Please note: Disallowing a specific file to be crawled will keep it from being indexed. The file disallowed will not show up in the search engines. HOWEVER, this is only effective for friendly robots. Robots can choose to ignore your instructions.

This allows all robots to crawl all files except the images file.

User-agent: *
Disallow: /images/

This allows all robots to crawl all files except the images file and the stats file.

User-agent: *
Disallow: /images/
Disallow: /stats/

What if I want to disallow a particular robot?

Occasionally you may find that you would like to disallow specific robots from crawling your site or limit which files they may have access to.

This denies access to Googlebot-image to any files in your domain

User-agent: Googlebot-Image
Disallow: /

This specifically denies Googlebot-image to your images file

User-agent: Googlebot-Image
Disallow: /images/

For a current data base of robot names and information, visit:

http://www.robotstxt.org/wc/active/html/index.html

How do I create a robots.txt file?

Simply create a text document and save the new document as robots.txt Do not use a html editor to create the file unless is has the ability to create a plain text document (ASCII). Most computers will allow you to create a text document using notepad.

  • Right click on your desktop
  • Choose new
  • Choose text document
  • Open the document you just created
  • Insert instructions to robots
  • Click on save as
  • Save document as robots.txt

How do I know if I have done everything correctly?

Once you have uploaded the file to the root directory of your domain it’s good idea to use a robots.txt validator to confirm that everything is correct. You can search Google for free robots.txt validator or try the one listed below.

http://tool.motoricerca.info/robots-checker.phtml

What if I need more information about robots.txt files?

This page is intended to cover creating a very simple robots.txt file. If you require a more detailed robots.txt file for your website there are many help resources available on the net. Google Information for Webmasters recommends visiting:

http://www.robotstxt.org/wc/norobots.html

Comments are closed.