Code tagged with 'distributed'

-->

Using backgroundrb to execute tasks asynchronously in Rails

Ruby posted 9 months ago by christian

Draft…

Planning on using BackgroundDRB? Take a long look at the alternatives first

Ask yourself, do you really need a complex solution like BackgroundDRB? Most likely you don’t, so use a simple daemonized process instead, see this snippet about the daemons gem for more information.

Heck, even a simple Ruby script run by cron every 5 minutes will be more stable than BackgroundDRB and require less work.

Even if you really need to process a lot of data asynchronously in the background, I wouldn’t recommend BackgroundDRB, it’s riddled with bugs and unstable in production, so use the BJ plugin instead.

Anyway, continue reading if you want to use BackgroundDRB…

Installing the prerequisites:

   1  $ sudo gem install chronic packet

Installing backgroundrb

   1  $ cd rails_project
   2  $ git clone git://gitorious.org/backgroundrb/mainline.git vendor/plugins/backgroundrb

You can also get the latest stable version from the Subversion repository:

   1  svn co http://svn.devjavu.com/backgroundrb/trunk  vendor/plugins/backgroundrb

Setup backgroundrb

   1  rake backgroundrb:setup

Create a worker

   1  ./script/generate worker feeds_worker

   1  class FeedsWorker < BackgrounDRb::MetaWorker
   2    set_worker_name :feeds_worker
   3    
   4    def create(args = nil)
   5      # this method is called, when worker is loaded for the first time
   6      logger.info "Created feeds worker"
   7    end
   8    
   9    def update(data)
  10      logger.info "Updating #{Feed.count} feeds."
  11      
  12      seconds = Benchmark.realtime do
  13        thread_pool.defer do
  14          Feed.update_all()
  15        end
  16      end
  17  
  18      logger.info "Update took #{'%.5f' % seconds}."
  19    end
  20  end

Starting backgroundrb

First configure backgroundrb by opening config/backgroundrb.yml in your editor:

   1  :backgroundrb:
   2    :ip: 0.0.0.0
   3  
   4  :development:
   5    :backgroundrb:
   6      :port: 11111     # use port 11111
   7      :log: foreground # foreground mode,print log messages on console
   8  
   9  :production:
  10    :backgroundrb:
  11      :port: 22222      # use port 22222

Next, start backgroundrb in development mode:

   1  ./script/backgroundrb -e development &

Call your worker

From the command line:

   1  $ script/console
   2  Loading development environment (Rails 2.0.2)
   3  >> MiddleMan.worker(:feeds_worker).update()

When things go wrong

Asynchronous programming is complex, so expect bugs…

Rule #1 know who you’re calling.

If you give your MiddleMan the wrong name of your worker, he’ll just spit this crap at you:

   1  You have a nil object when you didn't expect it!
   2  The error occurred while evaluating nil.send_request
   3  /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in `ask_worker'
   4  /Users/christian/Documents/Projects/xxx/vendor/plugins/backgroundrb/server/lib/master_worker.rb:104:in `process_work'
   5  /Users/christian/Documents/Projects/xxx/vendor/plugins/backgroundrb/server/lib/master_worker.rb:35:in `receive_data'
   6  /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in `call'
   7  /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in `extract'
   8  /Users/christian/Documents/Projects/xxx/vendor/plugins/backgroundrb/server/lib/master_worker.rb:31:in `receive_data'

So for example this command would generate the above mentioned error:

   1  MiddleMan.worker(:illegal_worker).update()

It’s always nice to see a cryptic error messages such as this, it really deserves an award.

Check for bugs and bug fixes

git mainline commits

Going to production

Starting the daemon:

   1  ./script/backgroundrb -e production start

Configuring your task to run periodically

The following example makes backgroundrb call the FeedsWorker’s update method once every 15 minutes:

   1  :production:
   2    :backgroundrb:
   3      :port: 22222      # use port 22222
   4      :lazy_load: true  # do not load models eagerly
   5      :debug_log: false # disable log workers and other logging
   6  # Cron based scheduling
   7  :schedules:
   8    :feeds_worker:
   9      :update:
  10        :trigger_args: * */15 * * * *
  11        :data: "Hello world"

At the time of writing, the cron scheduler seems to be broken, so I prefer hard-coding the interval in the worker’s create method:

   1  def create
   2             add_periodic_timer(15.minutes) { update }
   3           end

If using Vlad or Capistrano, it’s also a good idea to fix script/backgroundrb by changing these lines:

   1  pid_file = "#{RAILS_HOME}/../../shared/pids/backgroundrb_#{CONFIG_FILE[:backgroundrb][:port]}.pid"
   2  SERVER_LOGGER = "#{RAILS_HOME}/../../shared/log/backgroundrb_server_#{CONFIG_FILE[:backgroundrb][:port]}.log"