God is a monitoring framework written in Ruby that can be used for monitoring, for example, mongrel processes.

Installing god

Install god with the following command:

   1  sudo gem install god

Configuring god

To configure god, first create a master configuration script by saving the following in /etc/god/god.rb:

   1  # load in all god configs
   2  God.load "/etc/god/conf/*.rb"

Now, save this configuration in /etc/god/conf/site.com.rb:

   1  #
   2  # Test this configuration file by executing:  
   3  #   god -c /path_to_this_file -D
   4  # 
   5  require 'yaml'
   6  
   7  
   8  #
   9  # Change these to match your project setup
  10  #
  11  APPLICATION  = "xxx.com"
  12  ROOT         = "/var/www/#{APPLICATION}" # deployment directory
  13  RAILS_ROOT   = ROOT + '/current'         # current release directory
  14  MONGREL_CONF = ROOT + '/shared/mongrel_cluster.conf' # mongrel_cluster.conf file
  15  
  16  # Read in mongrel_conf
  17  OPTIONS      = YAML.load_file(MONGREL_CONF)   # Read mongrel configuration
  18  
  19  #
  20  # TODO This can be simplified
  21  #
  22  def ports(port, servers)
  23    ports = []
  24    
  25    start_port = port
  26    end_port   = start_port + servers - 1
  27    
  28    for port in start_port..end_port do
  29      ports << port
  30    end
  31    
  32    ports
  33  end
  34  
  35  PORTS        = ports(OPTIONS['port'].to_i, OPTIONS['servers'].to_i)
  36  
  37  #
  38  # Returns path of mongrel pid or log file:
  39  #
  40  #   mongrel_path "/tmp/mongrel.pid", 9000 => "/tmp/mongrel.9000.pid"
  41  #
  42  def mongrel_path(file_path, port)
  43      file_ext = File.extname(file_path)
  44      file_base = File.basename(file_path, file_ext)
  45      file_dir = File.dirname(file_path)
  46      file = [file_base, port].join(".") +  file_ext
  47      
  48      File.join(file_dir, file)
  49  end
  50  
  51  #
  52  # Returns the mongrel_rails start, stop or restart command depending on command parameter
  53  #
  54  def mongrel_rails(command, port)
  55    raise "Unsupported command '#{command}'" if !['start', 'stop', 'restart'].include?(command)
  56  
  57    argv = [ "mongrel_rails" ]
  58    argv << command
  59    argv << "-d" if command != 'stop'
  60    argv << "-e #{OPTIONS['environment']}" if OPTIONS['environment'] && command != 'stop'
  61    argv << "-a #{OPTIONS['address']}"  if OPTIONS['address'] && command != 'stop'
  62    argv << "-c #{OPTIONS['cwd']}" if OPTIONS['cwd']
  63    argv << "-f #{OPTIONS['force']}" if OPTIONS['force'] && command == 'stop'
  64    argv << "-o #{OPTIONS['timeout']}" if OPTIONS['timeout'] && command != 'stop'
  65    argv << "-t #{OPTIONS['throttle']}" if OPTIONS['throttle'] && command != 'stop'
  66    argv << "-m #{OPTIONS['mime_map']}" if OPTIONS['mime_map'] && command != 'stop'
  67    argv << "-r #{OPTIONS['docroot']}" if OPTIONS['docroot'] && command != 'stop'
  68    argv << "-n #{OPTIONS['num_procs']}" if OPTIONS['num_procs'] && command != 'stop'
  69    argv << "-B" if OPTIONS['debug'] && command != 'stop'
  70    argv << "-S #{OPTIONS['config_script']}" if OPTIONS['config_script'] && command != 'stop'
  71    argv << "--user #{OPTIONS['user']}" if OPTIONS['user'] && command != 'stop'
  72    argv << "--group #{OPTIONS['group']}" if OPTIONS['group'] && command != 'stop'
  73    argv << "--prefix #{OPTIONS['prefix']}" if OPTIONS['prefix'] && command != 'stop'
  74    argv << "-p #{port}" if command != 'stop'
  75    argv << '-P ' + mongrel_path(OPTIONS['pid_file'], port)
  76    argv << '-l ' + mongrel_path(OPTIONS['log_file'], port) if command != 'stop'
  77  
  78    cmd = argv.join " "
  79  
  80    return cmd
  81  end
  82  
  83  PORTS.each do |port|
  84    God.watch do |w|
  85      w.name          = "#{APPLICATION}-#{port}"
  86      w.group         = "mongrels"
  87      w.interval      = 30.seconds
  88      w.start         = mongrel_rails('start', port)
  89      w.stop          = mongrel_rails('stop', port)
  90      w.restart       = mongrel_rails('restart', port)
  91      w.start_grace   = 10.seconds
  92      w.restart_grace = 10.seconds
  93      w.pid_file      = File.join(RAILS_ROOT, "/tmp/pids/mongrel.#{port}.pid")
  94          
  95      w.behavior(:clean_pid_file)
  96  
  97      w.start_if do |start|
  98        start.condition(:process_running) do |c|
  99          c.interval = 5.seconds
 100          c.running  = false
 101        end
 102      end
 103      
 104      w.restart_if do |restart|
 105        restart.condition(:memory_usage) do |c|
 106          c.above = 150.megabytes
 107          c.times = [3, 5] # 3 out of 5 intervals
 108        end
 109      
 110        restart.condition(:cpu_usage) do |c|
 111          c.above = 50.percent
 112          c.times = 5
 113        end
 114      end
 115      
 116      # lifecycle
 117      w.lifecycle do |on|
 118        on.condition(:flapping) do |c|
 119          c.to_state     = [:start, :restart]
 120          c.times        = 5
 121          c.within       = 5.minute
 122          c.transition   = :unmonitored
 123          c.retry_in     = 10.minutes
 124          c.retry_times  = 5
 125          c.retry_within = 2.hours
 126        end
 127      end
 128    end
 129  end

Add a script for each site you want to monitor.

Starting god

To start god execute:

   1  god -c /etc/god/god.rb

For a list of available commands run god with the help switch:

   1  $ god --help
   2    Usage:
   3      Starting:
   4        god [-c <config file>] [-p <port> | -b] [-P <file>] [-l <file>] [-D]
   5        
   6      Querying:
   7        god <command> <argument> [-p <port>]
   8        god <command> [-p <port>]
   9        god -v
  10        god -V (must be run as root to be accurate on Linux)
  11        
  12      Commands:
  13        start <task or group name>         start task or group
  14        restart <task or group name>       restart task or group
  15        stop <task or group name>          stop task or group
  16        monitor <task or group name>       monitor task or group
  17        unmonitor <task or group name>     unmonitor task or group
  18        remove <task or group name>        remove task or group from god
  19        load <file>                        load a config into a running god
  20        log <task name>                    show realtime log for given task
  21        status                             show status of each task
  22        quit                               stop god
  23        terminate                          stop god and all tasks
  24        check                              run self diagnostic
  25        
  26      Options:
  27      -c, --config-file CONFIG         Configuration file
  28      -p, --port PORT                  Communications port (default 17165)
  29      -b, --auto-bind                  Auto-bind to an unused port number
  30      -P, --pid FILE                   Where to write the PID file
  31      -l, --log FILE                   Where to write the log file
  32      -D, --no-daemonize               Don't daemonize
  33      -v, --version                    Print the version number and exit
  34      -V                               Print extended version and build information
  35          --log-level LEVEL            Log level [debug|info|warn|error|fatal]
  36          --no-syslog                  Disable output to syslog
  37          --attach PID                 Quit god when the attached process dies
  38          --no-events                  Disable the event system
  39          --bleakhouse                 Enable bleakhouse profiling

Surviving reboots

Save the following in /etc/init.d/god:

   1  #!/bin/bash
   2  #
   3  # God
   4  #
   5  
   6  RETVAL=0
   7  
   8  case "$1" in
   9      start)
  10        god -c /etc/god/god.rb -P /var/run/god.pid -l /var/log/god.log
  11        RETVAL=$?
  12        echo "God started"
  13    ;;
  14      stop)
  15        kill `cat /var/run/god.pid`
  16        RETVAL=$?
  17        echo "God stopped"
  18    ;;
  19      restart)
  20        kill `cat /var/run/god.pid`
  21        god -c /etc/god/god.rb -P /var/run/god.pid -l /var/log/god.log
  22        RETVAL=$?
  23        echo "God restarted"
  24    ;;
  25      status)
  26        RETVAL=$?
  27    ;;
  28      *)
  29        echo "Usage: god {start|stop|restart|status}"
  30        exit 1
  31    ;;
  32  esac
  33  
  34  exit $RETVAL

Make the file executable with chmod:

   1  chmod +x /etc/init.d/god

Tell Debian to run the script at startup:

   1  sudo /usr/sbin/update-rc.d -f god defaults

This is a draft…

Installing Vlad the Deployer

   1  gem install vlad

Configuring Vlad the Deployer

Add this to the end of RakeFile:

   1  begin
   2    require 'rubygems'
   3    require 'vlad'
   4    Vlad.load :scm => :git
   5  rescue LoadError => e
   6    puts "Unable to load Vlad #{e}."
   7  end

Note that we’re telling Vlad to use git. This snippet- gives you a quick introduction on how to use git with Rails.

Creating the deployment recipe

If you’re uncertain what these variables mean, have a look at the docs. This folder is also worth a look, and don’t forget to take a peek at the vlad source code.

   1  #
   2  # General configuration
   3  #
   4  set :ssh_flags,             '-p 666'
   5  set :application,           'xxx.com'
   6  set :domain,                '127.0.01'
   7  set :deploy_to,             '/var/www/xxx.com'
   8  set :repository,            '/var/lib/git/repositories/xxx.com/.git/'
   9  
  10  
  11  #
  12  # Mongrel configuration
  13  #
  14  set :mongrel_clean,         true
  15  set :mongrel_command,       'sudo mongrel_rails'
  16  set :mongrel_group,         'www-data'
  17  set :mongrel_port,          9000
  18  set :mongrel_servers,       3
  19  
  20  #set :mongrel_address,       '127.0.0.1'
  21  #set(:mongrel_conf)          { '#{shared_path}/mongrel_cluster.conf' }
  22  #set :mongrel_config_script, nil
  23  #set :mongrel_environment,   'production'
  24  #set :mongrel_log_file,      nil
  25  #set :mongrel_pid_file,      nil
  26  #set :mongrel_prefix,        nil
  27  #set :mongrel_user,          'mongrel'
  28  
  29  #
  30  # Customize Vlad to our needs
  31  #
  32  namespace :vlad do
  33    #
  34    # Add an after_update hook
  35    #
  36    remote_task :update do
  37      Rake::Task['vlad:after_update'].invoke
  38    end
  39  
  40    #
  41    # The after_update hook, which is run after vlad:update
  42    #
  43    remote_task :after_update do
  44    # Link to shared resources, if you have them in .gitignore
  45    #  run "ln -s #{deploy_to}/shared/system/database.yml #{deploy_to}/current/config/database.yml"
  46    end
  47  
  48    #
  49    # Deploys a new version of your application
  50    #
  51    remote_task :deploy => [:update, :migrate, :start_app]
  52  end

Setup the server

   1  $ rake vlad:setup

This will create the necessary folders and mongrel_cluster configuration file.

Deploy the application

Now deploy the application with vlad:deploy, which is a custom rake task that we added to the deployment recipe:

   1  $ rake vlad:deploy

Copying your SSH public key to the remote server

Vlad uses ssh for executing commands on the remotely, and rsync for copying the build to your server, which means you’ll quickly grow tired of typing your password each time a command is run.

This problem is solved by copying your public SSH keys to the remote server, this snippet- explains how to do exactly that.

DRAFT …

Install RubyGems

   1  http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
   2  
   3  tar zxvf rubygems-1.0.1.tgz
   4  
   5  cd rubygems-1.0.1
   6  
   7  ruby setup.rb

Install Rails

   1  gem install rails

Install sqlite3 (optional)

   1  apt-get install sqlite3 libsqlite3-dev
   2  gem install sqlite3-ruby

Install mongrel and mongrel_cluster

   1  $ gem install mongrel mongrel_cluster
   2  
   3  $ mongrel_rails cluster::configure -e production \
   4    -p 8000 \
   5    -a 127.0.0.1 \
   6    -N 3 \
   7    -c /var/www/xyz/current
   8  
   9  
  10  $ mongrel_rails cluster::start
  11  
  12  
  13  $ useradd -g www-data -d /var/www mongrel

Surviving reboots

   1  sudo mkdir /etc/mongrel_cluster
   2  
   3  sudo ln -s /var/www/xyz/config/mongrel_cluster.yml /etc/mongrel_cluster/xyz.yml
   4  
   5  sudo cp /usr/local/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/resources/mongrel_cluster /etc/init.d/
   6  
   7  sudo chmod +x /etc/init.d/mongrel_cluster
   8  
   9  sudo /usr/sbin/update-rc.d -f mongrel_cluster defaults
  10  
  11  mongrel_cluster_ctl status

Stale pids

If your mongrels crash or if you kill them, mongrel_cluster won’t start your mongrels because mongrel_cluster believes the processes are still running, instead mongrel_cluster complains and does nothing:

   1  ** !!! PID file tmp/pids/mongrel.8000.pid already exists.  Mongrel could be running already.  Check your log/mongrel.8000.log for errors.
   2  ** !!! Exiting with error.  You must stop mongrel and clear the .pid before I'll attempt a start.

To fix this simply add the —clean switch to the /usr/local/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/resources/mongrel_cluster startup script:

   1  mongrel_cluster_ctl start -c $CONF_DIR --clean

I received the following error when installing Mongrel from the gem repository:

   1  marko@x61s:$ sudo gem install mongrel
   2  Updating metadata for 281 gems from http://gems.rubyforge.org
   3  complete
   4  Building native extensions.  This could take a while...
   5  ERROR:  Error installing mongrel:
   6  	ERROR: Failed to build gem native extension.
   7  
   8  /usr/bin/ruby1.8 extconf.rb install mongrel
   9  extconf.rb:1:in `require': no such file to load -- mkmf (LoadError)
  10  	from extconf.rb:1

The fix is to install the ruby development package:

   1  sudo apt-get install ruby1.8-dev

If you’re using Edge Rails and mongrel you might have seen this error:

   1  warning: already initialized constant OPTIONS

There’s a bug report here that explains this problem in detail.

The solution for me was to rerun the freeze command:

   1  rake rails:freeze:edge

You might also get “warning: already initialized constant OPTIONS ” error if you haven’t installed a gem your software relies on. In the following example openssl should be installed:

   1  /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require’: no such file to load—openssl (MissingSourceFile)

ActiveResource might also be the problem:

   1  gem install activeresource --source http://gems.rubyonrails.org