First, tell your application to use syslog

#
# Example in config/production.rb:
#
#   config.logger = LogHelper.open
#   config.log_tags = [ :uuid, :remote_ip ]
#
require 'syslog/logger'

class LogHelper
  PROGRAM_NAME = "rails"
  LEVEL = Logger::DEBUG

  # Maps syslog severity to a string
  SEVERITY_MAP = {
    0 => 'debug',
    1 => 'info',
    2 => 'warn',
    3 => 'error',
    nil => ''
  }.freeze

  def self.open
    # There can only be one program name
    l = Syslog::Logger.new(PROGRAM_NAME)
    l.formatter = proc do |severity, datetime, progname, msg|
      "[#{SEVERITY_MAP[severity].center(5)}] #{msg}\n"
    end
    l.level = LEVEL
    # Add tags to log messages from thread local context, e.g., request.uuid
    ActiveSupport::TaggedLogging.new(l)
  end
end

Example:

logger = LogHelper.open
logger.info("test")

Install and configure the rsyslog server

Next install rsyslog on the server:

$ sudo apt-get install rsyslog

Uncomment the TCP module:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Comment out the IncludeConfig directive:

# $IncludeConfig /etc/rsyslog.d/*.conf

We’ll use a DynaFile to simplify the rsyslog configuration:

$template DynaFile,"/var/log/hosts/%HOSTNAME%/%PROGRAMNAME%.log"
*.* -?DynaFile

This tells rsyslog to write to a file named after the host and program that sent the message, e.g., /var/log/hosts/prod-1/rails.log.

Install and configure rsyslog clients

The naming of rsyslog configuration files IS important. Rules are processed in alphabetic order according to file name:

$ ls /var/rsyslog.d/
/etc/rsyslog.d/10-app.conf
/etc/rsyslog.d/50-default.conf

First, tell syslog to forward everything to the remote rsyslog server by editing /etc/rsyslog.conf:

#
# Send all logs to centralized log server over TCP
#
*.*       @@192.168.0.0

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

You can configure application logs to be written to a local file in addition to the default (remote server):

if $programname startswith 'rails' then {
  /var/log/rails.log
  # Uncomment this to disable forwarding:
  # & stop
}

Configure Haproxy

Add log-send-hostname to the haproxy configuration.

Haproxy only seems to work with UDP logging, so enable it in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

Restarting rsyslog

Remember to restart rsyslog after checking the configuration is valid:

$ rsyslogd -N1
$ sudo service rsyslog restart

Tested with rsyslog version 8.

Gotchas

• rsyslog’s default rate-limiting configuration will drop messages
• make sure to check rsyslog’s own logs for warnings
• you probably want to use monit to monitor that rsyslog is writing log messages to disk