Craigslist blocks out search engine crawlers

The Search Engine Professionals at Rank for $ales.com --- In business since 1997.

SEARCH FOR: CITY or STATE:

Search this site Join the SEO Help Forum

Craigslist blocks out search engine crawlers

January 4, 2006

Get the most dependable Linux or Windows Web hosting at the lowest cost. Domain names at only 99 cents for a whole year! Click here to learn more.

Craigslist has apparently blocked the crawling of its classified ads site from search engine spiders.

As a result of its voluntary blocking of search engine robots, Craigslist's Web pages are not showing anymore in search engine results pages.

A query in Google or Yahoo for an item in Craig�s �jobs� or �for sale� category will confirm that the content to the site has been removed entirely.

According to some observers, this could be the largest site de-indexing ever.

Why would Craigslist want to do such a drastic delisting when webmasters all over the globe are scrounging to get such valuable search traffic?

It could be an effort to deter spammers from listing bogus links in Craigslist in an effort to have those links followed by search engines.

The delisting may also be an attempt to stop scraping of the sites content via bots which access cache pages of Craigslist from search engines, then reprint the content illegally on other sites.

Remember back last year when Craigslist blocked aggregation sites and niche engines like Oodle from indexing their classified ads?

This recent block of the major search engine bots may be an extended effort to establish Craigslist as more of a destination, and not content fuel for Yahoo, Google, AskJeeves and MSN.

Source: Search Engine Journal

Increase your site traffic with a paid inclusion program

Read Serge Thibodeau's daily blogs on search engines at Serge Thibodeau Live. We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. Click here to start your keyword and key phrase research.

Pay Rank for $ales securely with your Visa, MasterCard, Discover, or American Express credit card through the secure PayPal network. (Note: PayPal is an eBay company, and maintains a net free capital of US $ 50 Million).

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.

Sponsered by Link Rent Sponsored by Avantex Traffic stats by Site Clicks�

Site design by Mtl. Web D. Sponsored by Press Broadcast Sponsored by Blog Hosting.ca