Register now and start sharing your code snippets.
-->
How to get monit to start mongrel_rails properly
Shell Script (Bash) posted 16 days ago by christian
Mongrel_rails and monit are not the best of friends. It’s difficult to get them to work together.
For example, this is the error I got in my monit logs when switching to a new mongrel_rails command that cleans up stale pids:
1 'mongrel_1' process is not running 2 'mongrel_2' trying to restart 3 'mongrel_3' start: /usr/local/bin/mongrel_rails 4 'mongrel_4' failed to start
To fix it I added the following start_command to the monit configuration:
1 /usr/bin/env PATH=/usr/local/bin/:$PATH mongrel_rails cluster::start -C /var/www.... --clean --only 8000
The problem is that monit overrides the PATH environment variable, so it won’t find mongrel_rails unless you tell it where to find it.
Monit also contains a bug which doesn’t tell you why it can’t start mongrel_rails, but that’s another story…
Note that I’m using the —clean switch which will startup the mongrels even if a stale pid exists.
In fact I got so tired of the whole mess I wrote a plugin that generates a working monit configuration for mongrel_rails from one or more mongrel_cluster.yml configuration files.
How to use god to monitor a pack of mongrels
Ruby posted 8 months ago by christian
God is a monitoring framework written in Ruby that can be used for monitoring, for example, mongrel processes.
Installing god
Install god with the following command:
1 sudo gem install god
Configuring god
To configure god, first create a master configuration script by saving the following in /etc/god/god.rb:
1 # load in all god configs 2 God.load "/etc/god/conf/*.rb"
Now, save this configuration in /etc/god/conf/site.com.rb:
1 # 2 # Test this configuration file by executing: 3 # god -c /path_to_this_file -D 4 # 5 require 'yaml' 6 7 8 # 9 # Change these to match your project setup 10 # 11 APPLICATION = "xxx.com" 12 ROOT = "/var/www/#{APPLICATION}" # deployment directory 13 RAILS_ROOT = ROOT + '/current' # current release directory 14 MONGREL_CONF = ROOT + '/shared/mongrel_cluster.conf' # mongrel_cluster.conf file 15 16 # Read in mongrel_conf 17 OPTIONS = YAML.load_file(MONGREL_CONF) # Read mongrel configuration 18 19 # 20 # TODO This can be simplified 21 # 22 def ports(port, servers) 23 ports = [] 24 25 start_port = port 26 end_port = start_port + servers - 1 27 28 for port in start_port..end_port do 29 ports << port 30 end 31 32 ports 33 end 34 35 PORTS = ports(OPTIONS['port'].to_i, OPTIONS['servers'].to_i) 36 37 # 38 # Returns path of mongrel pid or log file: 39 # 40 # mongrel_path "/tmp/mongrel.pid", 9000 => "/tmp/mongrel.9000.pid" 41 # 42 def mongrel_path(file_path, port) 43 file_ext = File.extname(file_path) 44 file_base = File.basename(file_path, file_ext) 45 file_dir = File.dirname(file_path) 46 file = [file_base, port].join(".") + file_ext 47 48 File.join(file_dir, file) 49 end 50 51 # 52 # Returns the mongrel_rails start, stop or restart command depending on command parameter 53 # 54 def mongrel_rails(command, port) 55 raise "Unsupported command '#{command}'" if !['start', 'stop', 'restart'].include?(command) 56 57 argv = [ "mongrel_rails" ] 58 argv << command 59 argv << "-d" if command != 'stop' 60 argv << "-e #{OPTIONS['environment']}" if OPTIONS['environment'] && command != 'stop' 61 argv << "-a #{OPTIONS['address']}" if OPTIONS['address'] && command != 'stop' 62 argv << "-c #{OPTIONS['cwd']}" if OPTIONS['cwd'] 63 argv << "-f #{OPTIONS['force']}" if OPTIONS['force'] && command == 'stop' 64 argv << "-o #{OPTIONS['timeout']}" if OPTIONS['timeout'] && command != 'stop' 65 argv << "-t #{OPTIONS['throttle']}" if OPTIONS['throttle'] && command != 'stop' 66 argv << "-m #{OPTIONS['mime_map']}" if OPTIONS['mime_map'] && command != 'stop' 67 argv << "-r #{OPTIONS['docroot']}" if OPTIONS['docroot'] && command != 'stop' 68 argv << "-n #{OPTIONS['num_procs']}" if OPTIONS['num_procs'] && command != 'stop' 69 argv << "-B" if OPTIONS['debug'] && command != 'stop' 70 argv << "-S #{OPTIONS['config_script']}" if OPTIONS['config_script'] && command != 'stop' 71 argv << "--user #{OPTIONS['user']}" if OPTIONS['user'] && command != 'stop' 72 argv << "--group #{OPTIONS['group']}" if OPTIONS['group'] && command != 'stop' 73 argv << "--prefix #{OPTIONS['prefix']}" if OPTIONS['prefix'] && command != 'stop' 74 argv << "-p #{port}" if command != 'stop' 75 argv << '-P ' + mongrel_path(OPTIONS['pid_file'], port) 76 argv << '-l ' + mongrel_path(OPTIONS['log_file'], port) if command != 'stop' 77 78 cmd = argv.join " " 79 80 return cmd 81 end 82 83 PORTS.each do |port| 84 God.watch do |w| 85 w.name = "#{APPLICATION}-#{port}" 86 w.group = "mongrels" 87 w.interval = 30.seconds 88 w.start = mongrel_rails('start', port) 89 w.stop = mongrel_rails('stop', port) 90 w.restart = mongrel_rails('restart', port) 91 w.start_grace = 10.seconds 92 w.restart_grace = 10.seconds 93 w.pid_file = File.join(RAILS_ROOT, "/tmp/pids/mongrel.#{port}.pid") 94 95 w.behavior(:clean_pid_file) 96 97 w.start_if do |start| 98 start.condition(:process_running) do |c| 99 c.interval = 5.seconds 100 c.running = false 101 end 102 end 103 104 w.restart_if do |restart| 105 restart.condition(:memory_usage) do |c| 106 c.above = 150.megabytes 107 c.times = [3, 5] # 3 out of 5 intervals 108 end 109 110 restart.condition(:cpu_usage) do |c| 111 c.above = 50.percent 112 c.times = 5 113 end 114 end 115 116 # lifecycle 117 w.lifecycle do |on| 118 on.condition(:flapping) do |c| 119 c.to_state = [:start, :restart] 120 c.times = 5 121 c.within = 5.minute 122 c.transition = :unmonitored 123 c.retry_in = 10.minutes 124 c.retry_times = 5 125 c.retry_within = 2.hours 126 end 127 end 128 end 129 end
Add a script for each site you want to monitor.
Starting god
To start god execute:
1 god -c /etc/god/god.rb
For a list of available commands run god with the help switch:
1 $ god --help 2 Usage: 3 Starting: 4 god [-c <config file>] [-p <port> | -b] [-P <file>] [-l <file>] [-D] 5 6 Querying: 7 god <command> <argument> [-p <port>] 8 god <command> [-p <port>] 9 god -v 10 god -V (must be run as root to be accurate on Linux) 11 12 Commands: 13 start <task or group name> start task or group 14 restart <task or group name> restart task or group 15 stop <task or group name> stop task or group 16 monitor <task or group name> monitor task or group 17 unmonitor <task or group name> unmonitor task or group 18 remove <task or group name> remove task or group from god 19 load <file> load a config into a running god 20 log <task name> show realtime log for given task 21 status show status of each task 22 quit stop god 23 terminate stop god and all tasks 24 check run self diagnostic 25 26 Options: 27 -c, --config-file CONFIG Configuration file 28 -p, --port PORT Communications port (default 17165) 29 -b, --auto-bind Auto-bind to an unused port number 30 -P, --pid FILE Where to write the PID file 31 -l, --log FILE Where to write the log file 32 -D, --no-daemonize Don't daemonize 33 -v, --version Print the version number and exit 34 -V Print extended version and build information 35 --log-level LEVEL Log level [debug|info|warn|error|fatal] 36 --no-syslog Disable output to syslog 37 --attach PID Quit god when the attached process dies 38 --no-events Disable the event system 39 --bleakhouse Enable bleakhouse profiling
Surviving reboots
Save the following in /etc/init.d/god:
1 #!/bin/bash 2 # 3 # God 4 # 5 6 RETVAL=0 7 8 case "$1" in 9 start) 10 god -c /etc/god/god.rb -P /var/run/god.pid -l /var/log/god.log 11 RETVAL=$? 12 echo "God started" 13 ;; 14 stop) 15 kill `cat /var/run/god.pid` 16 RETVAL=$? 17 echo "God stopped" 18 ;; 19 restart) 20 kill `cat /var/run/god.pid` 21 god -c /etc/god/god.rb -P /var/run/god.pid -l /var/log/god.log 22 RETVAL=$? 23 echo "God restarted" 24 ;; 25 status) 26 RETVAL=$? 27 ;; 28 *) 29 echo "Usage: god {start|stop|restart|status}" 30 exit 1 31 ;; 32 esac 33 34 exit $RETVAL
Make the file executable with chmod:
1 chmod +x /etc/init.d/god
Tell Debian to run the script at startup:
1 sudo /usr/sbin/update-rc.d -f god defaults