Scraping Google search results with Scrubyt and Ruby

2 Note that these instructions don't work with the latest Scrubyt version... Scrubyt is a Ruby library that allows you to easily scrape the contents of any site. h2. First install Scrubyt: <code> $ sudo gem install mechanize hpricot parsetree ruby2ruby scrubyt </code> You also need to install ReadLine version 3.6.3: <code> sudo gem install -v 3.6.3 RubyInline </code> If you install the wrong RubyInline version or have multiple versions installed, you'll get the following error: <code> /usr/lib/ruby/1.8/rubygems.rb:207:in `activate': can't activate RubyInline (= 3.6.3), already activated RubyInline-3.6.6] (Gem::Exception) from /usr/lib/ruby/1.8/rubygems.rb:225:in `activate' from /usr/lib/ruby/1.8/rubygems.rb:224:in `each' from /usr/lib/ruby/1.8/rubygems.rb:224:in `activate' from /usr/lib/ruby/1.8/rubygems/custom_require.rb:32:in `require' from t2:2 </code> To fix it first uninstall the latest version, and keep only version 3.6.3: <code> sudo gem uninstall RubyInline Select RubyGem to uninstall: 1. RubyInline-3.6.3 2. RubyInline-3.6.6 3. All versions > 2 </code> h2. Scraping Google search results Then run this to Scrape the first two pages of the Google results for *ruby*: <code> require 'rubygems' require 'scrubyt' # See http://scrubyt.org/example-specification-from-the-page-known-issues-and-pitfalls/ # Create a learning extractor data = Scrubyt::Extractor.define do fetch('http://www.google.com/') fill_textfield 'q', 'ruby' submit # Teach Scrubyt what we want to retrieve # In this case we want Scruby to find all search results # and "Ruby Programming Language" happens to be the first # link in the result list. Change "Ruby Programming Language" # to whatever you want Scruby to find. link do name "Ruby Programming Language" url "href", :type => :attribute end # Click next until we're on the second page. next_page "Next", :limit => 2 end # Print out what Scruby found puts data.to_xml puts "Your production scraper has been created: data_extractor_export.rb." # Export the production version of the scraper data.export(__FILE__) </code> h2. Learning Extractor vs Production extractor Note that this example uses the "Learning Extractor":http://scrubyt.org/example-specification-from-the-page-known-issues-and-pitfalls/ functionality of Scrubyt. The production extractor is generated with the last line: <code> data.export(__FILE__) </code> If you open the production extractor in an editor you'll see that it uses XPath queries to extract the content: <code> link("/html/body/div/div/div/h2", { :generalize => true }) do name("/a[1]") url("href", { :type => :attribute }) end </code> h2. Finding the correct XPath The learning mode is pretty good at finding the XPath of HTML elements, but if you have difficulties getting Scrubyt to extract exactly what you want, simply install Firebug and use the Inspect feature to select the item you want to extract the value from. Then right-click on it in the Firebug window and choose Copy XPath. Note that there's a "gotcha":http://groups.google.com/group/firebug/browse_thread/thread/b3f9b0893c1ad7e1 when copying the XPath of an element with Firebug. Firebug uses Firefox's internal and normalized DOM model, which might not match match the real-world HTML structure. For example the tbody tag is usually added by Firefox/Firebug, and should be removed if it isn't in the HTML. Another option that I haven't tried myself is to use the "XPather":https://addons.mozilla.org/en-US/firefox/addon/1192 extension. h2. Using hpricot to find the XPath If you're really having problems finding the right XPath of an element, you can also use HPricot to find it. In this example the code prints out the XPath to all table columns containing the text 51,999: <code> require 'rexml/document' require 'hpricot' require 'open-uri' url = "http://xyz" page = Hpricot(open(url, 'User-Agent' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12', 'Referer' => 'http://xyz' )) page.search( "//td:contains('51,992')" ).each do |row| puts row.xpath() end </code> The output from the above snippet looks something like this: <code> /html/body/table[2]/tr[2]/td[3] /html/body/table[2]/tr[2]/td[3]/table[4]/tr[1]/td[1] /html/body/table[2]/tr[2]/td[3]/table[4]/tr[1]/td[1]/table[1]/tr[2]/td[2] </code> Note that sometimes I find that hrpicot is easier to use than Scrubyt, so use what's best for you. h2. Miscellaneous problems The following problem can be solved by following the instructions found "here":http://agora.scrubyt.org/forums/3/topics/270: <code> Your production scraper has been created: data_extractor_export.rb. /var/lib/gems/1.8/gems/ParseTreeReloaded-0.0.1/lib/parse_tree_reloaded.rb:129:in `extend': wrong argument type Class (expected Module) (TypeError) from /var/lib/gems/1.8/gems/ParseTreeReloaded-0.0.1/lib/parse_tree_reloaded.rb:129:in `to_sexp' from /var/lib/gems/1.8/gems/ParseTreeReloaded-0.0.1/lib/parse_tree_reloaded.rb:93:in `parse_tree_for_method' from /var/lib/gems/1.8/gems/ruby2ruby-1.1.6/lib/ruby2ruby.rb:1063:in `to_sexp' </code> 0

2007-10-15T14:07:52+03:00

124

Note that these instructions don’t work with the latest Scrubyt version… Scrubyt is a Ruby library that allows you to easily scrape the contents of any site. <h2>First install Scrubyt:</h2> <pre class="active4d"> 1 $ sudo gem install mechanize hpricot parsetree ruby2ruby scrubyt </pre> You also need to install ReadLine version 3.6.3: <pre class="active4d"> 1 sudo gem install -v 3.6.3 RubyInline </pre> If you install the wrong RubyInline version or have multiple versions installed, you’ll get the following error: <pre class="active4d"> 1 /usr/lib/ruby/1.8/rubygems.rb:207:in `activate': can't activate RubyInline (= 3.6.3), already activated RubyInline-3.6.6] (Gem::Exception) 2 from /usr/lib/ruby/1.8/rubygems.rb:225:in `activate' 3 from /usr/lib/ruby/1.8/rubygems.rb:224:in `each' 4 from /usr/lib/ruby/1.8/rubygems.rb:224:in `activate' 5 from /usr/lib/ruby/1.8/rubygems/custom_require.rb:32:in `require' 6 from t2:2 </pre> To fix it first uninstall the latest version, and keep only version 3.6.3: <pre class="active4d"> 1 sudo gem uninstall RubyInline 2 3 Select RubyGem to uninstall: 4 1. RubyInline-3.6.3 5 2. RubyInline-3.6.6 6 3. All versions 7 > 2 </pre> <h2>Scraping Google search results</h2> Then run this to Scrape the first two pages of the Google results for ruby: <pre class="active4d"> 1 require 'rubygems' 2 require 'scrubyt' 3 4 # See http://scrubyt.org/example-specification-from-the-page-known-issues-and-pitfalls/ 5 6 # Create a learning extractor 7 data = Scrubyt::Extractor.define do 8 fetch('http://www.google.com/') 9 fill_textfield 'q', 'ruby' 10 submit 11 12 # Teach Scrubyt what we want to retrieve 13 # In this case we want Scruby to find all search results 14 # and "Ruby Programming Language" happens to be the first 15 # link in the result list. Change "Ruby Programming Language" 16 # to whatever you want Scruby to find. 17 link do 18 name "Ruby Programming Language" 19 url "href", :type => :attribute 20 end 21 22 # Click next until we're on the second page. 23 next_page "Next", :limit => 2 24 end 25 26 # Print out what Scruby found 27 puts data.to_xml 28 29 puts "Your production scraper has been created: data_extractor_export.rb." 30 31 # Export the production version of the scraper 32 data.export(__FILE__) </pre> <h2>Learning Extractor vs Production extractor</h2> Note that this example uses the <a href="http://scrubyt.org/example-specification-from-the-page-known-issues-and-pitfalls/">Learning Extractor</a> functionality of Scrubyt. The production extractor is generated with the last line: <pre class="active4d"> 1 data.export(__FILE__) </pre> If you open the production extractor in an editor you’ll see that it uses XPath queries to extract the content: <pre class="active4d"> 1 link("/html/body/div/div/div/h2", { :generalize => true }) do 2 name("/a[1]") 3 url("href", { :type => :attribute }) 4 end </pre> <h2>Finding the correct XPath</h2> The learning mode is pretty good at finding the XPath of HTML elements, but if you have difficulties getting Scrubyt to extract exactly what you want, simply install Firebug and use the Inspect feature to select the item you want to extract the value from. Then right-click on it in the Firebug window and choose Copy XPath. Note that there’s a <a href="http://groups.google.com/group/firebug/browse_thread/thread/b3f9b0893c1ad7e1">gotcha</a> when copying the XPath of an element with Firebug. Firebug uses Firefox’s internal and normalized DOM model, which might not match match the real-world HTML structure. For example the tbody tag is usually added by Firefox/Firebug, and should be removed if it isn’t in the HTML. Another option that I haven’t tried myself is to use the <a href="https://addons.mozilla.org/en-US/firefox/addon/1192">XPather</a> extension. <h2>Using hpricot to find the XPath</h2> If you’re really having problems finding the right XPath of an element, you can also use HPricot to find it. In this example the code prints out the XPath to all table columns containing the text 51,999: <pre class="active4d"> 1 require 'rexml/document' 2 require 'hpricot' 3 require 'open-uri' 4 5 url = "http://xyz" 6 7 page = Hpricot(open(url, 8 'User-Agent' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12', 9 'Referer' => 'http://xyz' 10 )) 11 12 page.search( "//td:contains('51,992')" ).each do |row| 13 puts row.xpath() 14 end </pre> The output from the above snippet looks something like this: <pre class="active4d"> 1 /html/body/table[2]/tr[2]/td[3] 2 /html/body/table[2]/tr[2]/td[3]/table[4]/tr[1]/td[1] 3 /html/body/table[2]/tr[2]/td[3]/table[4]/tr[1]/td[1]/table[1]/tr[2]/td[2] </pre> Note that sometimes I find that hrpicot is easier to use than Scrubyt, so use what’s best for you. <h2>Miscellaneous problems</h2> The following problem can be solved by following the instructions found <a href="http://agora.scrubyt.org/forums/3/topics/270">here</a>: <pre class="active4d"> 1 Your production scraper has been created: data_extractor_export.rb. 2 /var/lib/gems/1.8/gems/ParseTreeReloaded-0.0.1/lib/parse_tree_reloaded.rb:129:in `extend': wrong argument type Class (expected Module) (TypeError) 3 from /var/lib/gems/1.8/gems/ParseTreeReloaded-0.0.1/lib/parse_tree_reloaded.rb:129:in `to_sexp' 4 from /var/lib/gems/1.8/gems/ParseTreeReloaded-0.0.1/lib/parse_tree_reloaded.rb:93:in `parse_tree_for_method' 5 from /var/lib/gems/1.8/gems/ruby2ruby-1.1.6/lib/ruby2ruby.rb:1063:in `to_sexp' </pre>

2008-11-23T02:00:24+02:00