November 19, 2003

User-agent: EmailWolf
Disallow: /

User-agent: ExtractorPro
Disallow: /

The user-agent is the name of the robot you want to disallow. In this example, I have chosen to disallow Titan, EmailCollector, EmailSiphon, EmailWolf and ExtractorPro. Note that many of these robots are from spam organizations or companies that collect email addresses that will be used in spam. Those unwanted robots take up unnecessary Internet bandwidth and slows down your Web server in the process. It is my experience that most of those email collectors usually obey the Robots.txt protocol.

Conclusion
Properly implementing the Robots exclusion protocol is both a simple process and takes very little time to enforce. When used as it is intended, it can ensure that the files you want indexed in your website will be indexed. Any site owner or Webmaster should really take the little time it takes to fully understand how this simple protocol works and how it can be safely implemented in any website, large or small.

It will also tell the search robots where they are not welcomed, so you can really concentrate in managing your online business in the safest way possible, away from “inquisitive minds”.

<<< Previous page

Pages:     1     2     3     4     5     6     7

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2003

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of rankforsales.com and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact sthibodeau@rankforsales.com to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit www.techblog.org

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.