To submit your sitemap to search engines—at least Google, MSN and Yahoo support this feature—add this line to your robots.txt file:

   1  Sitemap: http://aktagon.com/sitemap.xml

This allows the search engine to find your sitemap when it visits your site, which means you don’t have to manually register it with each search engine.

Try this query to find pages that are in Google’s supplemental index (trick invented by Bruce Clay, Inc):

   1  -site:aktagon.com/* +site:aktagon.com/

The query should list all of your pages that are in Google’s supplemental index a.k.a Google hell. These pages lower your Google page rank, so you should tell Google not to bother indexing those pages. This can be done with robots.txt:

   1  User-agent:*
   2  Disallow: /tags/*
   3  Disallow: /archive/*

Another way of doing it is to add a meta tag:

   1  <meta name="robots" content="noindex,follow"/> 

This tells search engines to read the page but not index it.

And, be careful with what you put in robots.txt…