October 10, 2003

They're called spiders. Scores of these digital robots crawl throug cyberspace, scouring the billions of pages in the World Wide Web.

They catalog all the words in the pages, how many times they're viewed, and which other pages they link to. These spiders power the fast-growing search industry, led by Google, Yahoo! (YHOO ) and Microsoft's MSN.

This is quickly becoming humanity's favorite tool for finding out just about anything, with 550 million Internet searches performed worldwide each day. Type in a couple words, say "sleeping" and "bag," and the engine, fueled by its spiders and scientific algorithms, promptly provide a list of Web sites, starting with the most relevant. Right?

Not so fast. In the last year, a host of search engines, including MSN, Lycos and Global Business Listing, have sprinkled growing numbers of paid corporate Web pages into the search results and accepting money each time one of these so-called paid-inclusion links is clicked. This practice is a booming business, one expected to double this year, to $200 million, and to reach $600 million by 2007, says U.S. Bancorp Piper Jaffray (USB ). But unlike the clearly marked advertised links that show up next to search results, these paid inclusions are virtually invisible to average Web surfers.

Only specialists prepared to decipher Web addresses can begin to distinguish between the paid and unpaid pages. "These are not objective oracles of information," says Beau Brendler, director of Consumer WebWatch, a project of Consumers Union.

Controversy over paid inclusion is driving a wedge through the search industry. At the heart of the issue is whether the industry's content will be swayed by advertisers. Google, the leading search engine, steers clear of paid inclusions, saying that they undermine confidence in the objectivity of search results.

Instead, Google relies entirely upon its technology for the search, and features paid ads only in clearly marked boxes near the search results. AOL uses Google technology and follows the same line. But others, including Yahoo, say that paid inclusion can provide users with better information. And they maintain that their search results are still displayed in order of relevance. That means that paid ads get no preferential treatment. "All listings are run through an algorithm that is blind to their source," says Tony Mamone, sales vice-president at LookSmart (LOOK ) Ltd., which feeds paid-inclusion results to portals like MSN.

Ticket to the top
Yet an investigation by BusinessWeek, based on more than 30 interviews and analysis of dozens of Internet searches, suggests that paying customers frequently fare better. Indeed, out of 20 advertisers and online marketing pros interviewed by BusinessWeek, 10 had experienced firsthand a boost in search-engine rankings when they signed up for paid inclusion. Says James Taylor, CEO of AICompany.com, a search-marketing company, "Paid inclusion dilutes the accuracy and relevance of a search engine."

Dennis Swanson learned the hard way. Earlier this year, the chief executive of Lamps Plus Inc., couldn't understand why his e-commerce offerings for light fixtures turned up so prominently in Google but were nearly out of sight on MSN. So he paid to have his site included with LookSmart, which feeds search results to MSN and shares with MSN revenues it gets from advertisers.

Almost instantly, Lamps Plus pages soared to near the top of MSN's search results, and Microsoft's portal became Swanson's biggest traffic driver. When he temporarily stopped his LookSmart deal, MSN's traffic spigot shut off. "If we don't have a deal with LookSmart, we don't show up in MSN's results," he says. MSN, while declining to comment on this case, says it is currently reevaluating its entire Internet search offering, including paid-inclusion listings, to put forth the most relevant results.

What gives? For starters, the paid-inclusion pages are easily engineered for high placement. When submitting paid-inclusion data, companies typically fill out a spreadsheet with information on product details, along with the search words and phrases for which they'd like to appear. The result is a rich stream of data targeted precisely to what the search engine will deem relevant. Consequently, their ratings are primed to soar.

Anecdotal examples aren't tough to turn up. Plug in "green sleeping bag" to HotBot.com and the first six results, in a recent search, featured paid-inclusion listings from Inktomi, which is now owned by Yahoo. Some weren't even for sleeping bags. The fifth listing took users to a paid-inclusion product page for a white canvas bag. Meanwhile, numerous pure results displayed lower were highly relevant product pages for green sleeping bags. A Yahoo spokesperson responds that there are just as many examples of paid-inclusion listings improving search results.

Despite criticisms, the paid-inclusion business addresses some very real needs. One biggie is the sheer number of Web pages tucked away in corporate databases, where search-engine spiders have a difficult time navigating. By letting, say, Staples.com pipe its data directly to a search engine, a user seeking "inkjet cartridges" could get a fresh listing from Staples.com that might not otherwise show up. "We use [paid inclusion] only if it's improving the relevance of our results," says Tim Cadogan, Yahoo's vice-president for search.

Still, the growth of paid inclusion could tilt the search business toward deep-pocketed companies. For instance, travel sites that want to be in LookSmart's paid-inclusion program have to pay the company a flat rate of 30 cents for each click, according to the company's Web site. Jumbo chains like Marriott International (MAR ) Inc. or Hilton Hotels (HLT ) Corp. are certainly better equipped to cover such costs than, say, a bed-and-breakfast.

Just ask Rob Spooner. The 56-year-old runs a travel site called Online Highways with informational pages about various small towns throughout the U.S. Inktomi, a search engine that now belongs to Yahoo, contacted him late last year about becoming a paid-inclusion participant. The proposal: Spooner would pay 10 cents for every visitor Inktomi passed along. Spooner did the math, figured he would lose money on anything more than 3 cents a click, and declined the offer.

Then things went downhill. Spooner's Web pages soon plunged in Inktomi's search rankings and disappeared from key sites like MSN, where Inktomi feeds its listings. After he demanded to know what happened, Spooner learned from Inktomi that his site contained editorial flaws that hurt his ranking. And he would have to become a paid-inclusion customer to learn what these flaws were. All this, while his pages remained well ranked on Google. "I lost a quarter of my traffic," says Spooner.

Yahoo declined to comment on this incident. However, it and other search engines offering paid inclusion say it serves the customers' interest. Says Tom Wilde, general manager of search services at Terra Lycos: "If you deliver a poor user experience, people leave, and they leave rapidly."

And why not label the sponsored links? Many search companies and advertisers fret that marking them would scare the public away from relevant sites. Says Glenn McLaren, e-commerce manager at Friedrich Air Conditioning: "Part of what makes it attractive is that it doesn't look like advertising." The trouble is, if paid inclusions are hidden, readers could grow to dismiss Internet search as nothing but a pile of ads.

Article by Bill Elgin
Source: Business Week Online

        

Read Serge Thibodeau's daily blogs on search engines at Serge Thibodeau Live. We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. Click here to start your keyword and key phrase research.

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.