haproxy snippets

How to fix "SSL client CA chain cannot be verified" haproxy error

Tagged haproxy, openssl, ssl  Languages bash

Check which client certificates CA names the server accepts

$ openssl s_client -connect 127.0.0.1:443 -servername xxx.com

Acceptable client certificate CA names
/C=FI/ST=Finland/O=Vaestorekisterikeskus TEST/OU=Terveydenhuollon testiammattivarmenteet/CN=VRK TEST CA for Healthcare Professionals
/C=FI/ST=Finland/O=Vaestorekisterikeskus TEST/OU=Testivarmenteet/CN=VRK CA for Test Purposes

Export client certificate public key to a file

If needed, export the client certificate's public key to a file, e.g. xyz.pem.

Check who has issued the client certificate:

$ openssl x509 -in xyz.pem -text

Issuer: C=FI, ST=Finland, O=Vaestorekisterikeskus TEST, OU=Terveydenhuollon testiammattivarmenteet, CN=VRK TEST CA for Healthcare Professionals

Is the issuer one of the CAs listed in step #1?

Verify client certificate

If yes, verify the client certificate against haproxy's ca-file:

cat xyz.pem | openssl verify -CAfile /etc/ssl/certs/haproxy-ca-file.pem

If validation fails, you probably need to add some root or intermediate certificates to /etc/ssl/certs/haproxy-ca-file.pem.

Using rsyslog for remote logging (Rails and Haproxy)

Tagged imfile, rsyslog, haproxy, rails  Languages bash, ruby

First, consider using rsyslog’s imfile module to send log files to rsyslog instead of configuring Rails/Ruby to send log messsages to rsyslog: http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html

If you still want to Ruby to send the log messages to rsyslog, you need to tell your application to send log messages to syslog:

#
# Example in config/production.rb:
#
#   config.logger = LogHelper.open
#   config.log_tags = [ :uuid, :remote_ip ]
#
require 'syslog/logger'

class LogHelper
  PROGRAM_NAME = "rails"
  LEVEL = Logger::DEBUG

  # Maps syslog severity to a string
  SEVERITY_MAP = {
    0 => 'debug',
    1 => 'info',
    2 => 'warn',
    3 => 'error',
    nil => ''
  }.freeze

  def self.open
    # There can only be one program name
    l = Syslog::Logger.new(PROGRAM_NAME)
    l.formatter = proc do |severity, datetime, progname, msg|
      "[#{SEVERITY_MAP[severity].center(5)}] #{msg}\n"
    end
    l.level = LEVEL
    # Add tags to log messages from thread local context, e.g., request.uuid
    ActiveSupport::TaggedLogging.new(l)
  end
end

Example:

logger = LogHelper.open
logger.info("test")

Install and configure the rsyslog server

Next install rsyslog on the server:

$ sudo apt-get install rsyslog

Uncomment the TCP module:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Comment out the IncludeConfig directive:

# $IncludeConfig /etc/rsyslog.d/*.conf

We’ll use a DynaFile to simplify the rsyslog configuration:

$template DynaFile,"/var/log/hosts/%HOSTNAME%/%PROGRAMNAME%.log"
*.* -?DynaFile

This tells rsyslog to write to a file named after the host and program that sent the message, e.g., /var/log/hosts/prod-1/rails.log.

Install and configure rsyslog clients

The naming of rsyslog configuration files IS important. Rules are processed in alphabetic order according to file name:

$ ls /var/rsyslog.d/
/etc/rsyslog.d/10-app.conf
/etc/rsyslog.d/50-default.conf

First, tell syslog to forward everything to the remote rsyslog server by editing /etc/rsyslog.conf:

#
# Send all logs to centralized log server over TCP
#
*.*       @@192.168.0.0

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

You can configure application logs to be written to a local file in addition to the default (remote server):

if $programname startswith 'rails' then {
  /var/log/rails.log
  # Uncomment this to disable forwarding:
  # & stop
}

Configure Haproxy

Add log-send-hostname to the haproxy configuration.

Haproxy only seems to work with UDP logging, so enable it in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

Restarting rsyslog

Remember to restart rsyslog after checking the configuration is valid:

$ rsyslogd -N1
$ sudo service rsyslog restart

Tested with rsyslog version 8.

Gotchas

  • You will lose log messages because of rsyslog’s rate limiting and maximum message size configuration
  • Ruby’s syslog module only supports one program name
  • Make sure to check rsyslog’s own logs for warnings
  • You probably want to use monit to monitor that rsyslog is writing log messages to disk

Haproxy configuration template with SSL

Tagged ssl, template, haproxy  Languages 
# HAProxy documentation:
# http://cbonte.github.io/haproxy-dconv/configuration-1.7.html
#
# Inspiration:
# https://gist.github.com/nateware/3987720
# https://serversforhackers.com/using-ssl-certificates-with-haproxy
# https://developers.livechatinc.com/blog/speeding-up-our-api/
#
global
  # syslog
  log               /dev/log local0
  log               127.0.0.1 local1 notice
  # run as haproxy
  user              haproxy
  group             haproxy
  # total number of allowed open connections
  maxconn           50000
  pidfile           /var/run/haproxy.pid
  # random health checks
  spread-checks     5
  # run in background
  daemon
  # SSL certificates are found here
  ca-base           /etc/ssl/certs
  crt-base          /etc/ssl/private
  # SSL hardening, see https://www.ssllabs.com/ssltest/analyze.html
  tune.ssl.default-dh-param   2048
  ssl-default-bind-options    no-sslv3
  ssl-default-bind-ciphers    ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  ssl-default-server-options  no-sslv3
  ssl-default-server-ciphers  ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  # uncomment to debug
  #debug

# defaults apply to all servers
defaults
  log               global
  # requests use HTTP protocol
  mode              http
  # log HTTP requests
  option            httplog
  # keep alive connections between client and balancer. Close connections between balancer and backend
  option            http-server-close
  # X-Forwarded-For header
  option            forwardfor
  # Client closed connection, abort request
  option            abortonclose
  # if request fails, resend request to up to 2 servers
  retries           3
  # request can be handled by any server in case of failure
  option            redispatch
  # total number of allowed open connections per server
  maxconn           25000
  # health check fails it takes longer than this to respond
  timeout check     5s
  timeout client    30s
  timeout connect   30s
  timeout server    30s

#
# Define frontends (haproxy)
#
frontend http
  bind            *:80
  # redirect HTTP to HTTPS
  redirect scheme https if !{ ssl_fc }
  default_backend http-backend

frontend https
  bind            *:443 ssl crt www.xxx.com.pem
  default_backend http-backend

#
# Define backends (Rails, Go, Elixir, etc)
#
backend http-backend
  balance         roundrobin
  # health check is done by fetching /
  option          httpchk HEAD / HTTP/1.1
  # Define two backend servers
  server          http1 10.0.0.1:9000 check #inter 5s rise 18 fall 2
  server          http2 10.0.0.2:9001 check #inter 5s rise 18 fall 2
  # Set HTTP headers
  http-request    set-header X-Forwarded-Port %[dst_port]
  http-request    add-header X-Forwarded-Proto https if { ssl_fc }