  
<div id="snippet_221" class="snippet">
  <h2><a href="/snippets/221-How-to-detect-traffic-from-the-most-common-search-spiders-with-Ruby-" title="How to detect traffic from the most common search spiders with Ruby  - Ruby - spider, web crawler, bot, search, user agent, detect">How to detect traffic from the most common search spiders with Ruby </a></h2>
  <div class="details">
    <a style="background-color: #FFFF94;" href="http://snippets.aktagon.com/languages/124-Ruby">
      Ruby</a> posted about 1 year ago by christian
          
  </div>

  <div class="body">
    This snippet detects traffic from the following bots, which is enough for me:
	<ul>
	<li>Google &#8211; Googlebot/2.1 ( http://www.googlebot.com/bot.html)</li>
		<li>Google Image &#8211; Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)</li>
		<li> MSN  Live &#8211; msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)</li>
		<li>Yahoo &#8211; Mozilla/5.0 (compatible; Yahoo! Slurp;)</li>
	</ul>


	<p>The code (<a href="http://wiki.rubyonrails.org/rails/pages/UserAgent">via</a>):
<pre class="active4d"><span class="line-numbers">   1 </span> user_agent <span class="Operator">=</span> request.<span class="FunctionName">user_agent</span>.<span class="FunctionName">downcase</span>
<span class="line-numbers">   2 </span> <span class="Variable"><span class="Variable">@</span>bot</span> <span class="Operator">=</span> [ <span class="String"><span class="String">'</span>msnbot<span class="String">'</span></span>, <span class="String"><span class="String">'</span>yahoo! slurp<span class="String">'</span></span>,<span class="String"><span class="String">'</span>googlebot<span class="String">'</span></span> ].<span class="FunctionName">detect</span> { |<span class="Variable">bot</span>| user_agent.<span class="FunctionName">include?</span> bot }
</pre></p>


	<p>When the Google bot visists your site the @bot string will contain &#8216;googlebot&#8217;.</p>


	<p>If you need to detect more bots than these, then the <a href="http://www.user-agents.org/">user-agents.org</a> site contains a list of various user agents for both bots and browsers.</p>
  </div>

  <div style="font-size: 0.8em;margin:0.5em;">
    
      Tagged <a href="/tags/510-spider">spider</a>, <a href="/tags/511-web-crawler">web crawler</a>, <a href="/tags/334-bot">bot</a>, <a href="/tags/56-search">search</a>, <a href="/tags/512-user-agent">user agent</a>, <a href="/tags/392-detect">detect</a>
    
    
  </div>
</div>



  
<div id="snippet_130" class="snippet">
  <h2><a href="/snippets/130-A-simple-Jabber-XMPP-bot-that-uses-the-Jabber-Simple-library" title="A simple Jabber/XMPP bot that uses the Jabber:Simple library - Ruby - jabber, xmpp, gmail, gtalk, bot">A simple Jabber/XMPP bot that uses the Jabber:Simple library</a></h2>
  <div class="details">
    <a style="background-color: #FFFF94;" href="http://snippets.aktagon.com/languages/124-Ruby">
      Ruby</a> posted over 2 years ago by christian
          
  </div>

  <div class="body">
    <p>First install <a href="http://xmpp4r-simple.rubyforge.org/">Jabber::Simple</a>:</p>


	<p><pre class="active4d"><span class="line-numbers">   1 </span> <span class="Variable"><span class="Variable">$</span>sudo</span> gem install xmpp4r<span class="Operator">-</span>simple <span class="Operator">-</span>y
</pre></p>


	<p>On  OSX  you might get this error when installing xmpp4r-simple and the rdoc dependency:
<pre class="active4d"><span class="line-numbers">   1 </span> make
<span class="line-numbers">   2 </span> gcc <span class="Operator">-</span><span class="Variable">I</span>. <span class="Operator">-</span><span class="Variable">I</span><span class="Operator">/</span>usr<span class="Operator">/</span>local<span class="Operator">/</span>lib<span class="Operator">/</span>ruby<span class="Operator">/</span><span class="Number">1.8</span><span class="Operator">/</span>i686<span class="Operator">-</span>darwin8.<span class="Number">10.3</span> <span class="Operator">-</span><span class="Variable">I</span><span class="Operator">/</span>usr<span class="Operator">/</span>local<span class="Operator">/</span>lib<span class="Operator">/</span>ruby<span class="Operator">/</span><span class="Number">1.8</span><span class="Operator">/</span>i686<span class="Operator">-</span>darwin8.<span class="Number">10.3</span> <span class="Operator">-</span><span class="Variable">I</span>.  <span class="Operator">-</span>fno<span class="Operator">-</span>common <span class="Operator">-</span>g <span class="Operator">-</span><span class="Variable">O2</span>  <span class="Operator">-</span>fno<span class="Operator">-</span>common <span class="Operator">-</span>pipe <span class="Operator">-</span>fno<span class="Operator">-</span>common  <span class="Operator">-</span>c callsite.<span class="FunctionName">c</span>
<span class="line-numbers">   3 </span> gcc <span class="Operator">-</span><span class="Variable">I</span>. <span class="Operator">-</span><span class="Variable">I</span><span class="Operator">/</span>usr<span class="Operator">/</span>local<span class="Operator">/</span>lib<span class="Operator">/</span>ruby<span class="Operator">/</span><span class="Number">1.8</span><span class="Operator">/</span>i686<span class="Operator">-</span>darwin8.<span class="Number">10.3</span> <span class="Operator">-</span><span class="Variable">I</span><span class="Operator">/</span>usr<span class="Operator">/</span>local<span class="Operator">/</span>lib<span class="Operator">/</span>ruby<span class="Operator">/</span><span class="Number">1.8</span><span class="Operator">/</span>i686<span class="Operator">-</span>darwin8.<span class="Number">10.3</span> <span class="Operator">-</span><span class="Variable">I</span>.  <span class="Operator">-</span>fno<span class="Operator">-</span>common <span class="Operator">-</span>g <span class="Operator">-</span><span class="Variable">O2</span>  <span class="Operator">-</span>fno<span class="Operator">-</span>common <span class="Operator">-</span>pipe <span class="Operator">-</span>fno<span class="Operator">-</span>common  <span class="Operator">-</span>c rcovrt.<span class="FunctionName">c</span>
<span class="line-numbers">   4 </span> cc <span class="Operator">-</span>dynamic <span class="Operator">-</span>bundle <span class="Operator">-</span>undefined suppress <span class="Operator">-</span>flat_namespace  <span class="Operator">-</span><span class="Variable">L</span><span class="String"><span class="String">&quot;</span>/usr/local/lib<span class="String">&quot;</span></span> <span class="Operator">-</span>o rcovrt.<span class="FunctionName">bundle</span> callsite.<span class="FunctionName">o</span> rcovrt.<span class="FunctionName">o</span>  <span class="Operator">-</span>lruby  <span class="Operator">-</span>lpthread <span class="Operator">-</span>ldl <span class="Operator">-</span>lobjc  
<span class="line-numbers">   5 </span> <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>bin<span class="Operator">/</span>ld: <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>lib<span class="Operator">/</span>gcc<span class="Operator">/</span>i686<span class="Operator">-</span>apple<span class="Operator">-</span>darwin8<span class="Operator">/</span><span class="Number">4.0</span>.<span class="Number">1</span><span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>libpthread.<span class="FunctionName">dylib</span> unknown flags (type) of section <span class="Number">6</span> (__TEXT,__dof_plockstat) <span class="Keyword">in</span> load command <span class="Number">0</span>
<span class="line-numbers">   6 </span> <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>bin<span class="Operator">/</span>ld: <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>lib<span class="Operator">/</span>gcc<span class="Operator">/</span>i686<span class="Operator">-</span>apple<span class="Operator">-</span>darwin8<span class="Operator">/</span><span class="Number">4.0</span>.<span class="Number">1</span><span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>libdl.<span class="FunctionName">dylib</span> unknown flags (type) of section <span class="Number">6</span> (__TEXT,__dof_plockstat) <span class="Keyword">in</span> load command <span class="Number">0</span>
<span class="line-numbers">   7 </span> <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>bin<span class="Operator">/</span>ld: <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>lib<span class="Operator">/</span>gcc<span class="Operator">/</span>i686<span class="Operator">-</span>apple<span class="Operator">-</span>darwin8<span class="Operator">/</span><span class="Number">4.0</span>.<span class="Number">1</span><span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>libobjc.<span class="FunctionName">dylib</span> load command <span class="Number">9</span> unknown cmd field
<span class="line-numbers">   8 </span> <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>bin<span class="Operator">/</span>ld: <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>lib<span class="Operator">/</span>gcc<span class="Operator">/</span>i686<span class="Operator">-</span>apple<span class="Operator">-</span>darwin8<span class="Operator">/</span><span class="Number">4.0</span>.<span class="Number">1</span><span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>..<span class="Operator">/</span>libSystem.<span class="FunctionName">dylib</span> unknown flags (type) of section <span class="Number">6</span> (__TEXT,__dof_plockstat) <span class="Keyword">in</span> load command <span class="Number">0</span>
<span class="line-numbers">   9 </span> <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>bin<span class="Operator">/</span>ld: <span class="String"><span class="String">/</span></span><span class="String">usr</span><span class="String"><span class="String">/</span></span>lib<span class="Operator">/</span>libSystem.<span class="FunctionName">B</span>.<span class="FunctionName">dylib</span> unknown flags (type) of section <span class="Number">6</span> (__TEXT,__dof_plockstat) <span class="Keyword">in</span> load command <span class="Number">0</span>
<span class="line-numbers">  10 </span> collect2: ld returned <span class="Number">1</span> exit status
<span class="line-numbers">  11 </span> make: <span class="Operator">**</span><span class="Operator">*</span> [rcovrt.<span class="FunctionName">bundle</span>] <span class="Variable">Error</span> <span class="Number">1</span>
</pre></p>


	<p>Simply install XCode 3 to make the error go away, then run this code to start the bot&#8212;warning the bot will execute the message body, for example &#8220;ls -la&#8221;, on the system:
<pre class="active4d"><span class="line-numbers">   1 </span> <span class="Keyword">require</span> <span class="String"><span class="String">'</span>rubygems<span class="String">'</span></span>
<span class="line-numbers">   2 </span> <span class="Keyword">require</span> <span class="String"><span class="String">'</span>xmpp4r-simple<span class="String">'</span></span>
<span class="line-numbers">   3 </span> 
<span class="line-numbers">   4 </span> <span class="Keyword">include</span> <span class="Variable">Jabber</span>
<span class="line-numbers">   5 </span> <span class="LineComment"><span class="LineComment">#</span>Jabber::debug = true</span>
<span class="line-numbers">   6 </span> 
<span class="line-numbers">   7 </span> jid <span class="Operator">=</span> <span class="String"><span class="String">'</span>user@server.com<span class="String">'</span></span>
<span class="line-numbers">   8 </span> pass <span class="Operator">=</span> <span class="String"><span class="String">'</span>password<span class="String">'</span></span>
<span class="line-numbers">   9 </span> 
<span class="line-numbers">  10 </span> jabber <span class="Operator">=</span> <span class="LibraryClassType">Simple</span>.<span class="FunctionName">new</span>(jid, pass)
<span class="line-numbers">  11 </span> 
<span class="line-numbers">  12 </span> <span class="Keyword">loop</span> <span class="Keyword">do</span>
<span class="line-numbers">  13 </span>   messages <span class="Operator">=</span> jabber.<span class="FunctionName">received_messages</span>
<span class="line-numbers">  14 </span>   messages.<span class="FunctionName">each</span> <span class="Keyword">do </span>|<span class="Variable">message</span>| 
<span class="line-numbers">  15 </span>     body <span class="Operator">=</span> message.<span class="FunctionName">body</span> <span class="Keyword">if</span> message.<span class="FunctionName">type</span> <span class="Operator">==</span> <span class="UserDefinedConstant"><span class="UserDefinedConstant">:</span>chat</span>
<span class="line-numbers">  16 </span>     
<span class="line-numbers">  17 </span>     process <span class="Operator">=</span> <span class="Variable">IO</span>.<span class="FunctionName">popen</span>(body)
<span class="line-numbers">  18 </span>     result <span class="Operator">=</span> process.<span class="FunctionName">readlines</span>
<span class="line-numbers">  19 </span>     
<span class="line-numbers">  20 </span>     jabber.<span class="FunctionName">deliver</span>(<span class="String"><span class="String">'</span>some.user@gmail.com<span class="String">'</span></span>, result)
<span class="line-numbers">  21 </span>   <span class="Keyword">end</span>
<span class="line-numbers">  22 </span>       
<span class="line-numbers">  23 </span>   sleep <span class="Number">1</span>
<span class="line-numbers">  24 </span> <span class="Keyword">end</span>
</pre></p>


	<p>To use GTalk from another domain than gmail, you need to edit the Jabber::Simple source code&#8230;</p>
  </div>

  <div style="font-size: 0.8em;margin:0.5em;">
    
      Tagged <a href="/tags/330-jabber">jabber</a>, <a href="/tags/331-xmpp">xmpp</a>, <a href="/tags/332-gmail">gmail</a>, <a href="/tags/333-gtalk">gtalk</a>, <a href="/tags/334-bot">bot</a>
    
    
  </div>
</div>






