web crawler snippets

How to detect traffic from the most common search spiders with Ruby

Tagged spider, web crawler, bot, search, user agent, detect  Languages ruby

This snippet detects traffic from the following bots, which is enough for me:

The code (via):

user_agent = request.user_agent.downcase
@bot = [ 'msnbot', 'yahoo! slurp','googlebot' ].detect { |bot| user_agent.include? bot }

When the Google bot visists your site the @bot string will contain 'googlebot'.

If you need to detect more bots than these, then the user-agents.org site contains a list of various user agents for both bots and browsers.