How to use fswatch to run a command when a file is changed

Tagged fswatch, rspec  Languages bash

This example executes the the rspec command when a Ruby file is modified in the current directory:

# watch .rb files in spec directory and execute rspec <file>
fswatch -0 -e ".*" -i ".rb" . | xargs -0 -n 1 -I {} rspec {}

Note that fswatch will execute test concurrently if the same test file is changed again before the previous test has time to complete.

Haproxy configuration template with SSL

Tagged haproxy, ssl, template  Languages 
# HAProxy documentation:
# http://cbonte.github.io/haproxy-dconv/configuration-1.7.html
#
# Inspiration:
# https://gist.github.com/nateware/3987720
# https://serversforhackers.com/using-ssl-certificates-with-haproxy
# https://developers.livechatinc.com/blog/speeding-up-our-api/
#
global
  # syslog
  log               /dev/log local0
  log               127.0.0.1 local1 notice
  # run as haproxy
  user              haproxy
  group             haproxy
  # total number of allowed open connections
  maxconn           50000
  pidfile           /var/run/haproxy.pid
  # random health checks
  spread-checks     5
  # run in background
  daemon
  # SSL certificates are found here
  ca-base           /etc/ssl/certs
  crt-base          /etc/ssl/private
  # SSL hardening, see https://www.ssllabs.com/ssltest/analyze.html
  tune.ssl.default-dh-param   2048
  ssl-default-bind-options    no-sslv3
  ssl-default-bind-ciphers    ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  ssl-default-server-options  no-sslv3
  ssl-default-server-ciphers  ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  # uncomment to debug
  #debug

# defaults apply to all servers
defaults
  log               global
  # requests use HTTP protocol
  mode              http
  # log HTTP requests
  option            httplog
  # keep alive connections between client and balancer. Close connections between balancer and backend
  option            http-server-close
  # X-Forwarded-For header
  option            forwardfor
  # Client closed connection, abort request
  option            abortonclose
  # if request fails, resend request to up to 2 servers
  retries           3
  # request can be handled by any server in case of failure
  option            redispatch
  # total number of allowed open connections per server
  maxconn           25000
  # health check fails it takes longer than this to respond
  timeout check     5s
  timeout client    30s
  timeout connect   30s
  timeout server    30s

#
# Define frontends (haproxy)
#
frontend http
  bind            *:80
  # redirect HTTP to HTTPS
  redirect scheme https if !{ ssl_fc }
  default_backend http-backend

frontend https
  bind            *:443 ssl crt www.xxx.com.pem
  default_backend http-backend

#
# Define backends (Rails, Go, Elixir, etc)
#
backend http-backend
  balance         roundrobin
  # health check is done by fetching /
  option          httpchk HEAD / HTTP/1.1
  # Define two backend servers
  server          http1 10.0.0.1:9000 check #inter 5s rise 18 fall 2
  server          http2 10.0.0.2:9001 check #inter 5s rise 18 fall 2
  # Set HTTP headers
  http-request    set-header X-Forwarded-Port %[dst_port]
  http-request    add-header X-Forwarded-Proto https if { ssl_fc }

Using rsyslog for remote logging (Rails and Haproxy)

Tagged rsyslog, haproxy, rails  Languages bash, ruby

First, tell your application to use syslog

#
# Example in config/production.rb:
#
#   config.logger = LogHelper.open
#   config.log_tags = [ :uuid, :remote_ip ]
#
require 'syslog/logger'

class LogHelper
  PROGRAM_NAME = "rails"
  LEVEL = Logger::DEBUG

  # Maps syslog severity to a string
  SEVERITY_MAP = {
    0 => 'debug',
    1 => 'info',
    2 => 'warn',
    3 => 'error',
    nil => ''
  }.freeze

  def self.open
    # There can only be one program name
    l = Syslog::Logger.new(PROGRAM_NAME)
    l.formatter = proc do |severity, datetime, progname, msg|
      "[#{SEVERITY_MAP[severity].center(5)}] #{msg}\n"
    end
    l.level = LEVEL
    # Add tags to log messages from thread local context, e.g., request.uuid
    ActiveSupport::TaggedLogging.new(l)
  end
end

Example:

logger = LogHelper.open
logger.info("test")

Install and configure the rsyslog server

Next install rsyslog on the server:

$ sudo apt-get install rsyslog

Uncomment the TCP module:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Comment out the IncludeConfig directive:

# $IncludeConfig /etc/rsyslog.d/*.conf

We’ll use a DynaFile to simplify the rsyslog configuration:

$template DynaFile,"/var/log/hosts/%HOSTNAME%/%PROGRAMNAME%.log"
*.* -?DynaFile

This tells rsyslog to write to a file named after the host and program that sent the message, e.g., /var/log/hosts/prod-1/rails.log.

Install and configure rsyslog clients

The naming of rsyslog configuration files IS important. Rules are processed in alphabetic order according to file name:

$ ls /var/rsyslog.d/
/etc/rsyslog.d/10-app.conf
/etc/rsyslog.d/50-default.conf

First, tell syslog to forward everything to the remote rsyslog server by editing /etc/rsyslog.conf:

#
# Send all logs to centralized log server over TCP
#
*.*       @@192.168.0.0

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

You can configure application logs to be written to a local file in addition to the default (remote server):

if $programname startswith 'rails' then {
  /var/log/rails.log
  # Uncomment this to disable forwarding:
  # & stop
}

Configure Haproxy

Add log-send-hostname to the haproxy configuration.

Haproxy only seems to work with UDP logging, so enable it in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

Restarting rsyslog

Remember to restart rsyslog after checking the configuration is valid:

$ rsyslogd -N1
$ sudo service rsyslog restart

Tested with rsyslog version 8.

Gotchas

  • rsyslog’s default rate-limiting configuration will drop messages
  • make sure to check rsyslog’s own logs for warnings
  • you probably want to use monit to monitor that rsyslog is writing log messages to disk

Annotated Plug and Elixir Example

Tagged cowboy, elixir, erlang, genserver, otp, plug  Languages elixir, bash

This is an annotated example of a web application written using Elixir and Plug.

Create mix.exs:

#
# Mix is a build tool that provides tasks for creating, compiling, and testing
# Elixir projects, managing its dependencies, and more.
# https://github.com/hexpm/hex_web/blob/master/mix.exs
#
defmodule PlugExample.Mixfile do
  # In order to configure Mix, a developer needs to use Mix.Project in a module
  # and define a function named project that returns a keyword list with
  # configuration.
  # http://elixir-lang.org/docs/stable/mix/Mix.Project.html
  use Mix.Project

  # Configure project.
  def project do
    [app: :Router,
      version: "0.0.1",
      elixir: "~> 1.2",
      deps: deps]
  end

  #
  # Returns applications that should be started.
  #
  # In OTP, application denotes a component implementing some specific
  # functionality, that can be started and stopped as a unit, and which can be
  # re-used in other systems as well.
  # http://erlang.org/doc/man/application.html
  #
  def application do
    [
      applications: [:logger, :plug], # 3rd-party applications
      mod: {PlugExample.Supervisor, []} # own application
    ]
  end

  #
  # Returns project dependencies:
  # https://hex.pm/
  #
  def deps do
    [
      {:cowboy, "~> 1.0.0"}, # Erlang web server
      {:plug, "~> 1.0"} # Elixir's rack (Ruby), wsgi (Python), servlet (Java) API
    ]
  end
end

Create lib/router.exs

defmodule PlugExample.Router do
  #
  # Plug ships with a router that allows developers to quickly match on
  # incoming requests and perform some action:
  # https://hexdocs.pm/plug/readme.html#the-plug-router
  #
  use Plug.Router
  #
  # This module defines a Plug.Conn struct and the main functions for working with Plug connections.
  # https://hexdocs.pm/plug/Plug.Conn.html
  #
  import Plug.Conn
  # Logs request to STDOUT
  plug Plug.Logger
  # Matches request to a route
  plug :match
  # Dispatches request to a route
  plug :dispatch

  #
  # Start a GenServer, i.e., a server instance:
  #
  # http://elixir-lang.org/getting-started/mix-otp/genserver.html
  # https://blog.drewolson.org/understanding-gen-server/
  #
  def start_link do
    # Cowboy adapter:
    # https://hexdocs.pm/plug/Plug.Adapters.Cowboy.html
    {:ok, _} = Plug.Adapters.Cowboy.http PlugExample.Router, []
  end

  #
  # HTTP GET /
  #
  # Use conn.private to access route options:
  #   conn.private[:protected]
  #
  get "/", private: %{protected: false} do
    conn |> send_resp(200, "Hello world")
  end

  get "/favicon.ico" do
    conn |> send_resp(200, "LOL")
  end
  
  # Forward /about to another router's get "/" route
  # forward "/about", to: PlugExample.About

  #
  # A catch-all route
  #
  match _ do
    conn |> send_resp(404, "Not found")
  end
end

Create lib/supervisor.ex

#
# A supervisor is a process which supervises other processes, called child
# processes. Supervisors are used to build a hierarchical process structure
# called a supervision tree, a nice way to structure fault-tolerant applications.
#
# http://elixir-lang.org/docs/stable/elixir/Supervisor.html
#
defmodule PlugExample.Supervisor do
  #
  # In Elixir (actually, in Erlang/OTP), an application is a component
  # implementing some specific functionality, that can be started and stopped as a
  # unit, and which can be re-used in other systems.
  # http://elixir-lang.org/docs/stable/elixir/Application.html
  # http://erlang.org/doc/man/application.html
  # http://erlang.org/doc/design_principles/applications.html
  #
  use Application

  #
  # The type argument passed to start/2 is usually :normal unless in a
  # distributed setup where application takeovers and failovers are configured.
  #
  # start/2 typically returns {:ok, pid} or {:ok, pid, state} where pid
  # identifies the supervision tree and state is the application state.
  #
  def start(_type, _args) do
    #
    # Convenience functions for defining a supervision specification.
    # http://elixir-lang.org/docs/stable/elixir/Supervisor.Spec.html
    #
    import Supervisor.Spec, warn: false
    # A list of children (workers or supervisors) to supervise
    children = [
      worker(PlugExample.Router, [])
    ]
    #
    # If a child process terminates, only that process is restarted:
    # http://elixir-lang.org/docs/stable/elixir/Supervisor.html
    #
    opts = [strategy: :one_for_one, name: PlugExample.Supervisor]
    #
    # Start supervisor by passing the children to start_link/2.
    # You may want to use a module-based supervisor.
    Supervisor.start_link(children, opts)
  end
end

Start the server:

$ mix deps.get
$ iex -S mix

Continue reading: How to deploy an Elixir web application.

Elixir Deployment

Tagged deployment, elixir, exrm, deploy, rollback, upgrade  Languages bash

Setup

Development environment:

  1. Sign up for a Gitlab account, or use your favorite git server
  2. Create a (private) repository
  3. Create a deploy key that allows read-only access to your private repository (optional, if public)
  4. Commit code to the repository

Deployment

Production environment:

  1. Clone project
$ cd /var/www
$ git clone git@gitlab.com:christianhellsten/aktagon-snippets.git
  1. Create a release
$ MIX_ENV=prod mix do deps.get, clean, compile
$ MIX_ENV=prod mix release --verbosity=verbose
  1. Create a monit start script
check process snippets MATCHING "snippets/bin/snippets"
    start program = "/bin/su - deploy -c '/var/www/aktagon-snippets/rel/snippets/bin/snippets start'"
    stop program = "/bin/su - deploy -c '/var/www/aktagon-snippets/rel/snippets/bin/snippets stop'"
  1. Start server
$ sudo monit start snippets

Deploying a new version

How to perform hot upgrades:

$ git pull
$ MIX_ENV=prod mix do compile, release
$ rel/snippets/bin/snippets upgrade "0.0.2"

Rolling back to a previous version

How to perform hot downgrades:

$ rel/snippets/bin/snippets downgrade "0.0.1"

See https://exrm.readme.io/docs/upgradesdowngrades

Alternatives

Pagination with Elixir and Ecto

Tagged ecto, elixir, pagination  Languages elixir
defmodule Pagination do
  import Ecto.Query
  alias Snippets.Repo
  #
  # ## Example
  #
  #    Snippets.Snippet
  #    |> order_by(desc: :inserted_at)
  #    |> Pagination.page(page: 0, per_page: 10)
  #
  def page(query, page: page, per_page: per_page) do
    count = per_page + 1
    result = query
              |> limit(^count)
              |> offset(^(page*per_page))
              |> Repo.all
    %{ has_next?: (length(result) == count),
       has_prev?: page > 0,
       list: Enum.slice(result, 0, count-1) }
  end
end

Postgres Performance Tuning

Tagged database, performance, postgres, tuning  Languages sql

Tuning memory

To tune memory it helps to understand how much data fits into memory:

SELECT 
  sum(heap_blks_read) as heap_read,
  sum(heap_blks_hit)  as heap_hit,
  sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM 
  pg_statio_user_tables;

If the cache hit rate is below 99%, try giving PG more memory.

Finding slow queries

Using the slow-query feature is one option. Telling Postgres to store statistics is easier:

CREATE extension pg_stat_statements;

-- View slowest queries
SELECT 
  (total_time / 1000 / 60) as total_minutes, 
  (total_time/calls) as average_time, 
  query 
FROM pg_stat_statements 
ORDER BY 1 DESC 
LIMIT 100;

Finding tables that are not indexed

SELECT 
  relname, 
  100 * idx_scan / (seq_scan + idx_scan) percent_of_times_index_used, 
  n_live_tup rows_in_table
FROM 
  pg_stat_user_tables
WHERE 
    seq_scan + idx_scan > 0 
ORDER BY 
  n_live_tup DESC;

Reference:

How to monitor database backups with monit

Tagged backup, monit, monitoring, restore, sql  Languages 

“It’s not the backup, it’s the restore”

We want to know when our database backup process, including restore, is broken. We can use monit to monitor that our database:

  • is being backed up, i.e., that the backup script is being run periodically
  • can be restored from a backup, i.e., that the backup is not corrupted

More specifically we want to monitor the following:

We use monit to monitor the script that restores the database from a backup. For this we need:

  • a script, e.g. written in Ruby, that restores the database from a fresh backup

    • the script should write to a log file, e.g. /var/log/db-restore.log, after the database has been successfully restored
    • the script should check facts such as, e.g., number of rows
  • a monit script that checks the script has been run successfully within, e.g., the last 24 hours

Logging to a file with Elixir

Tagged elixir, file, logging  Languages elixir

mix.exs:

  def application do
    [
      applications: [:logger, :logger_file_backend, ...],
      mod: {Snippets, []}
    ]
  end

  def deps do
    [
      {:cowboy, "~> 1.0.0"}, # web server
      {:logger_file_backend, "~> 0.0.7"} # writes log messages to a file, LOL
      ....
    ]
   end

config/dev.exs:

config :logger, :error_log,
  path: "log/app.log",
  level: :debug