noindex snippets

Check which of your pages are in the Google supplemental index a.k.a Google hell

Tagged seo, google, robots.txt, noindex, follow  Languages 

Try this query to find pages that are in Google's supplemental index (trick invented by Bruce Clay, Inc):*

The query should list all of your pages that are in Google's supplemental index a.k.a Google hell. These pages lower your Google page rank, so you should tell Google not to bother indexing those pages. This can be done with robots.txt:

Disallow: /tags/*
Disallow: /archive/*

Another way of doing it is to add a meta tag:

<meta name="robots" content="noindex,follow"/>

This tells search engines to read the page but not index it.

And, be careful with what you put in robots.txt...