November 19, 2003

Examples of a properly written Robots.txt file
In the following example, I will show you how to properly write or edit a Robots.txt file. First, never use a word processor to write or edit these files. Today’s modern word processors use special formatting and characters that will not be understood by any of the search robots and could lead to problems, or worse, it could cause them to ignore the Robots file completely. Use a simple ‘pure vanilla text’ editor of the ASCII type or any text editor of the Unix variety. Personally, I always use the Notepad utility editor that comes on any Windows operating system.

Make certain you save it as “robots.txt” (all in lower case). Remember that most Web servers today run Unix and Linux and are all case sensitive. Here is a carefully written Robots.txt file:

User-agent: Titan
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

<<< Previous page              Next: How to disallow user agents >>>

Pages:     1     2     3     4     5     6     7

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2003

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of rankforsales.com and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact sthibodeau@rankforsales.com to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit www.techblog.org

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.