How search engines work

May 21, 2004

Most likely, you have probably used a Web search engine such as Google, Yahoo, AltaVista or MSN to find specific information on the Internet. Did you ever stop and wonder how exactly do search engines find that information?

I will explain that and more in this section. In so doing, you will begin to discover how to efficiently structure your web site to obtain maximum brand strength, utilizing the power of today's modern search engines.

First, let's eliminate the myths
It's a very common misconception that when a user enters a query into any search engine, it interrogates the Web to find pages that match the query. That is NOT how it works at all. Instead, the search engine looks at its own copy of the Web. Every search engine actually creates its own version of the Internet. This version is called an "index".

The size of a search engine's index varies from search engine to search engine, but it is always much smaller than the Web as a whole. For example, as of February 28, 2004, it is currently estimated that the whole Web presently consists of approximately 15 to 20 billion pages, whereas Google, which has the largest search engine index, has approximately 6 billion pages in it's index.

In fact, as late as October 2003, Google had only about 3.3 billion pages in its index, according to information available on its homepage at that time.

The search engine builds a list of pages to add to its index using a special piece of software known as a crawler or spider. The spider crawls across the Web, adding pages it visits to the list of pages to its index.

The spider is capable of reading text on a Web page and finding links to other pages to visit. In this fashion, the spider travels all across the Web, constantly finding new or modified pages to add or update to its current index.

Some time after a page has been "spidered" (visited by a crawler), the search engine's software effectively adds a copy of new or altered pages to the search engine's index. When a user enters a query into a search engine, the search engine's software searches its index to find the pages matching the search query.

It then sorts those pages into a specific ranking order. Each search engine uses its own search algorithm to find and rank pages, but most base their technology on the frequency and location of the search term on the page.

Apart from Google's Page Rank� algorithm, engineers at Google have also developed the Hilltop� algo, which is even more sophisticated.

The Hilltop algorithm determines the relevance and importance of a specific web page, determined by the search query or keyword used in the search box. In its basic, simplest form, instead of relying only on the PageRank� value to find �authoritative pages�, it would be more useful if that �PR value� (PageRank� value) would be more relevant by the topic or subject of that same page. Hilltop does that, plus a bit more.

Three important reminders related to SEO
This, as briefly as it sounds, is how search engines actually work. With that minimal knowledge and a little thought, and with the subject of search engine optimization in mind, you can reach the following three conclusions:

      A search engine may not have a copy of every page on your site
      If it does have it, that copy may not be up to date
      An engine can have a copy of a page that no longer exists on your site

These three reminders are really important, if you want to offer your visitors the best usability features in your site, and if you want to attain the highest visibility in the search engines.

Good housekeeping and good SEO
Also, remember that in order for a page on your site to be listed in response to a user query, words in that query must match words in the search engine's copy of the page. The professional image of your company or your brand could be, to a certain degree, damaged if you don't take these important facts into consideration.

What I mean by this is, if the search engine looks for a page that still happens to be on its index but is no longer available on your website, your Web server will generate what is called an HTTP error 404, which means the page the search engine is looking for at your site simply does not exist any longer.

This fully underscores the importance of updating your site at regular intervals and making certain there are no pages missing or any broken links.

Additionnally, keep in mind that about 85% of today's websites are hosted on servers that run the Linux operating system. Linux is very similar to the Unix operating system and is case sensitive. The letter "A" is not the same as the letter "a" in the Unix environment.

Always write all your filenames in lower case, avoiding using upper case and try to follow a consistent file naming strategy and this should go a long way in preventing most of the errors that are often encountered.

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2004

(Updated from the previous article I wrote on May 15, 2002).

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of rankforsales.com and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact sthibodeau@rankforsales.com to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit www.techblog.org

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

        

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.