October 22, 2004

Most likely, you have probably used a search engine such as Google, MSN or AltaVista, or a search directory such as Yahoo, DMOZ or Global Business Listing, in order to help you find specific information on the Internet that is of interest to you. Did you ever stop and wonder how exactly do search engines find that information?

If you are like me, you are curious in finding out more, hoping to learn additional techniques or ways that can be used in order to have your site get better visibility and popularity on the Internet.

I will explain that to you in this article. In so doing, you will begin to discover how to efficiently structure your web site to obtain maximum brand visibility and recognition from the major search engines in use today. At the same time, you will also help your users better find what it is they were looking for, which in turn should convert into more sales and more customers for you. I will also tell you what to do and what not to do in order to achieve all this.

Let's clear the misconceptions first
It's a very common misconception that when a user enters a query into any search engine, the engine interrogates the Web to find pages that match the query. While this would certainly be plausible, that is not how it works at all. The search engine searches its own copy of the Web. Every search engine actually creates its own copy of the Internet. This copy is called an index. The size of a search engine's index varies from search engine to search engine, but it is always smaller than the Web as a whole.

For example, as of November 2003, it is currently estimated that the Web presently consists of approximately Twelve Billion pages. However, Google's search index right now is estimated to be only 3.6 Billion pages, although growing everyday.

Today, most major search engines such as Google, AltaVista, Global Business Listing and the others constantly retrieve a huge list of pages to add to their current index, using special automated software programs known as crawlers or spiders. These spiders crawl across the whole Internet, adding pages they visit to the list of pages to their index.

Automated spiders are capable of reading a Web page and finding links to other pages or websites to visit. All this of course consumes a lot of Internet bandwidth on the major or monthly updates they need to do, but is all part of the normal process in getting "fresh" and updated results that will be of value to all searchers.

In this fashion, spiders can often travel across the Web finding new or updated pages to add or update their index. In the case of Google, "Freshbot" (the name of one of its most popular crawlers) visits a large number of websites every 24 or 48 hours, and many times will write the date it updated that site in its index directly on the results pages in Google.

Some time after a page has been "spidered" (visited by a crawler), the search engine's software effectively adds a copy of the new or altered pages to the search engine's index. When a user enters a query into a search engine, the search's engine software searches its index to find the pages matching the search query. It then sorts those pages into ranking order.

Each search engine uses its own "secret technology" to find and rank pages, but most base their technology on the frequency and location of the search, as well as certain parameters such as keyword density and other factors.

For example, Google uses it's patented and very popular PageRank algorithm, which it calculates as a site's importance in the number of "votes" from other on-topic, related web pages. The way it works is, the higher the Page Rank, thus the number of votes, the higher that site should be ranked in it's index.

When you think about it, it makes sense and it probably explains why Google today is the most relevant search engine on the Web.

General observations and assessments
This, as briefly as it sounds, is how search engines actually work. Of course, there can be variations of this, as most major search engines today do in fact regularly change their algorithms a lot or make important changes in the way their whole ranking system works. However, if you look at all this from a global perspective, you can safely reach the following conclusions:

     An engine may not have a copy of every page on your site
     If it does have it, that copy may not be up to date
     An engine can have a copy of a page that no longer exists on your site

Also, remember that in order for a page on your site to be listed in response to a user query, words in that query must match words in the search engine's copy of the page. The professional image of your company or your brand could be, to a certain degree, be damaged if you don't take these facts into consideration.

What I mean by this is, if the search engine looks for a page that still happens to be in its index but is no longer available on your web site, it will generate what is called an HTTP error 404, which means the page the user is looking for at your site does not exist anymore. This fully underscores the importance of updating your web site at regular intervals and making certain there are no pages missing or any broken links.

If you really have to remove certain pages, it is better to leave those pages intact, using the same filenames, but removing the body of the text and replacing it with a friendly message saying: "We are sorry, but this page no longer exists", rather than having your visitor getting those dreaded HTTP Error 404 pages which don't mean anything to your typical visitor.

New sections or new pages added
In the same line of thinking, if you add new pages or new sections to your website, it is imperative that you submit these new pages to the search engines. This will give you a "leg up" instead of waiting for the spiders to crawl your site, risking they might miss your new pages. As a whole, most major search engines today do a great job at indexing and updating old and newer web pages, but, as anything, nothing is perfect and it is always possible they may not have seen or discovered your newly created pages, so it's always a good idea to "hand submit" those new pages.

You should always try to avoid using automated software programs that claim to automatically submit your site to hundreds or "thousands" of search engines. These programs could in fact harm you more than they might help you. Also, there are no "thousands" of search engines worth submitting to in the first place. Submit to the most popular and major search engines and you will be fine.

Also, you should always stay away from those so-called "link farms" which is another risk definitely not worth taking with any site. Such link farms could have your site banned or penalized in certain engines and they will rarely provide any help to any of your users.

(Updated from my August 2001 article).

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2001

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of rankforsales.com and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact sthibodeau@rankforsales.com to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit www.techblog.org

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

        

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.