haproxy snippets

How to fix "SSL client CA chain cannot be verified" haproxy error

Tagged haproxy, ssl, openssl  Languages bash

Check which client certificates CA names the server accepts

$ openssl s_client -connect 127.0.0.1:443 -servername xxx.com

Acceptable client certificate CA names
/C=FI/ST=Finland/O=Vaestorekisterikeskus TEST/OU=Terveydenhuollon testiammattivarmenteet/CN=VRK TEST CA for Healthcare Professionals
/C=FI/ST=Finland/O=Vaestorekisterikeskus TEST/OU=Testivarmenteet/CN=VRK CA for Test Purposes

Export client certificate public key to a file

If needed, export the client certificate's public key to a file, e.g. xyz.pem.

Check who has issued the client certificate:

$ openssl x509 -in xyz.pem -text

Issuer: C=FI, ST=Finland, O=Vaestorekisterikeskus TEST, OU=Terveydenhuollon testiammattivarmenteet, CN=VRK TEST CA for Healthcare Professionals

Is the issuer one of the CAs listed in step #1?

Verify client certificate

If yes, verify the client certificate against haproxy's ca-file:

cat xyz.pem | openssl verify -CAfile /etc/ssl/certs/haproxy-ca-file.pem

If validation fails, you probably need to add some root or intermediate certificates to /etc/ssl/certs/haproxy-ca-file.pem.

Using rsyslog for remote logging (Rails and Haproxy)

Tagged rsyslog, haproxy, rails  Languages bash, ruby

First, tell your application to use syslog

#
# Example in config/production.rb:
#
#   config.logger = LogHelper.open
#   config.log_tags = [ :uuid, :remote_ip ]
#
require 'syslog/logger'

class LogHelper
  PROGRAM_NAME = "rails"
  LEVEL = Logger::DEBUG

  # Maps syslog severity to a string
  SEVERITY_MAP = {
    0 => 'debug',
    1 => 'info',
    2 => 'warn',
    3 => 'error',
    nil => ''
  }.freeze

  def self.open
    # There can only be one program name
    l = Syslog::Logger.new(PROGRAM_NAME)
    l.formatter = proc do |severity, datetime, progname, msg|
      "[#{SEVERITY_MAP[severity].center(5)}] #{msg}\n"
    end
    l.level = LEVEL
    # Add tags to log messages from thread local context, e.g., request.uuid
    ActiveSupport::TaggedLogging.new(l)
  end
end

Example:

logger = LogHelper.open
logger.info("test")

Install and configure the rsyslog server

Next install rsyslog on the server:

$ sudo apt-get install rsyslog

Uncomment the TCP module:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Comment out the IncludeConfig directive:

# $IncludeConfig /etc/rsyslog.d/*.conf

We’ll use a DynaFile to simplify the rsyslog configuration:

$template DynaFile,"/var/log/hosts/%HOSTNAME%/%PROGRAMNAME%.log"
*.* -?DynaFile

This tells rsyslog to write to a file named after the host and program that sent the message, e.g., /var/log/hosts/prod-1/rails.log.

Install and configure rsyslog clients

The naming of rsyslog configuration files IS important. Rules are processed in alphabetic order according to file name:

$ ls /var/rsyslog.d/
/etc/rsyslog.d/10-app.conf
/etc/rsyslog.d/50-default.conf

First, tell syslog to forward everything to the remote rsyslog server by editing /etc/rsyslog.conf:

#
# Send all logs to centralized log server over TCP
#
*.*       @@192.168.0.0

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

You can configure application logs to be written to a local file in addition to the default (remote server):

if $programname startswith 'rails' then {
  /var/log/rails.log
  # Uncomment this to disable forwarding:
  # & stop
}

Configure Haproxy

Add log-send-hostname to the haproxy configuration.

Haproxy only seems to work with UDP logging, so enable it in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

Restarting rsyslog

Remember to restart rsyslog after checking the configuration is valid:

$ rsyslogd -N1
$ sudo service rsyslog restart

Tested with rsyslog version 8.

Gotchas

  • rsyslog’s default rate-limiting configuration will drop messages
  • make sure to check rsyslog’s own logs for warnings
  • you probably want to use monit to monitor that rsyslog is writing log messages to disk

Haproxy configuration template with SSL

Tagged haproxy, ssl, template  Languages 
# HAProxy documentation:
# http://cbonte.github.io/haproxy-dconv/configuration-1.7.html
#
# Inspiration:
# https://gist.github.com/nateware/3987720
# https://serversforhackers.com/using-ssl-certificates-with-haproxy
# https://developers.livechatinc.com/blog/speeding-up-our-api/
#
global
  # syslog
  log               /dev/log local0
  log               127.0.0.1 local1 notice
  # run as haproxy
  user              haproxy
  group             haproxy
  # total number of allowed open connections
  maxconn           50000
  pidfile           /var/run/haproxy.pid
  # random health checks
  spread-checks     5
  # run in background
  daemon
  # SSL certificates are found here
  ca-base           /etc/ssl/certs
  crt-base          /etc/ssl/private
  # SSL hardening, see https://www.ssllabs.com/ssltest/analyze.html
  tune.ssl.default-dh-param   2048
  ssl-default-bind-options    no-sslv3
  ssl-default-bind-ciphers    ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  ssl-default-server-options  no-sslv3
  ssl-default-server-ciphers  ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  # uncomment to debug
  #debug

# defaults apply to all servers
defaults
  log               global
  # requests use HTTP protocol
  mode              http
  # log HTTP requests
  option            httplog
  # keep alive connections between client and balancer. Close connections between balancer and backend
  option            http-server-close
  # X-Forwarded-For header
  option            forwardfor
  # Client closed connection, abort request
  option            abortonclose
  # if request fails, resend request to up to 2 servers
  retries           3
  # request can be handled by any server in case of failure
  option            redispatch
  # total number of allowed open connections per server
  maxconn           25000
  # health check fails it takes longer than this to respond
  timeout check     5s
  timeout client    30s
  timeout connect   30s
  timeout server    30s

#
# Define frontends (haproxy)
#
frontend http
  bind            *:80
  # redirect HTTP to HTTPS
  redirect scheme https if !{ ssl_fc }
  default_backend http-backend

frontend https
  bind            *:443 ssl crt www.xxx.com.pem
  default_backend http-backend

#
# Define backends (Rails, Go, Elixir, etc)
#
backend http-backend
  balance         roundrobin
  # health check is done by fetching /
  option          httpchk HEAD / HTTP/1.1
  # Define two backend servers
  server          http1 10.0.0.1:9000 check #inter 5s rise 18 fall 2
  server          http2 10.0.0.2:9001 check #inter 5s rise 18 fall 2
  # Set HTTP headers
  http-request    set-header X-Forwarded-Port %[dst_port]
  http-request    add-header X-Forwarded-Proto https if { ssl_fc }