November 19, 2003

Back in the spring of 2003, I wrote an article on the Robots exclusion protocol that generated a lot of emails and many questions. I am still getting them, so it seems that a more extended article is warranted. Often referred to as the Robots.txt file, the Robots exclusion protocol can be a very important part of your search engine optimization program, but it needs to be carefully implemented to be really successful.

If used incorrectly, this small and “innocent looking” text file can cause a lot of problems. It can even cause your site from being excluded from the search engine’s databases, it it’s not written correctly! In this extended article, I will show you how to correctly write it and make the Robots exclusion protocol an important part of your SEO efforts in attaining good visibility in the major search engines.

How the robots exclusion protocol works
Some of you may ask what is it and why do we need it? In a nutshell, as it’s name implies, the Robots exclusion protocol is used by Webmasters and site owners to prevent search engine crawlers (or spiders) from indexing certain parts of their Web sites. It could be for a number of reasons, such as sensitive corporate information, semi-confidential data, information that needs to stay private, or to prevent certain programs or scripts from being indexed, etc.

Next: The robots.txt exclusion protocol was developed at the end of 1993 >>>

Pages:     1     2     3     4     5     6     7

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2003

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of rankforsales.com and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact sthibodeau@rankforsales.com to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit www.techblog.org

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.