configuration snippets

Change nginx configuration on the fly

Tagged nginx, reload, configuration  Languages bash

First make your changes to nginx.conf.

Then run the following command to test the new configuration:

# nginx -t -c /etc/nginx/nginx.conf 
2007/10/18 20:55:07 [info] 3125#0: the configuration file /etc/nginx/nginx.conf syntax is ok
2007/10/18 20:55:07 [info] 3125#0: the configuration file /etc/nginx/nginx.conf was tested successfully

Next, look for the process id of the master nginx process:

# ps -ef|grep nginx
root      1911     1  0 18:00 ?        00:00:00 nginx: master process /usr/sbin/nginx
www-data  1912  1911  0 18:00 ?        00:00:00 nginx: worker process

Lastly, tell nginx to reload the configuration and restart the worker processes:

# kill -HUP 1911

Sphinx configuration file template

Tagged sphinx, template, configuration  Languages 
source feed_items
{
        type                    = mysql

        sql_host                = 127.0.0.1
        sql_user                = root
        sql_pass                =
        sql_db                  = xxx_production
        sql_port                = 3306  # optional, default is 3306
        sql_sock                = /var/run/mysqld/mysqld.sock

        sql_query_pre           = SET NAMES utf8
        #sql_query_pre          = SET SESSION query_cache_type=OFF

    # Unique ID should be first column
        sql_query               = \
                SELECT i.id, i.title, i.link, f.link, f.title FROM feed_items i LEFT JOIN feeds f ON f.id = i.feed_id
}


index feed_items
{
        source                  = feed_items
        path                    = /var/sphinx/xxx
        morphology              = libstemmer_sv
        charset_type            = utf-8
}


indexer
{
        mem_limit               = 32M
}

searchd
{
        address                 = 127.0.0.1
        port                    = 3312
        log                     = /var/log/sphinx/searchd.log
        query_log               = /var/log/sphinx/query.log
        pid_file                = /var/log/searchd.pid
        max_matches             = 1000
}

Awstats LogFormat configuration for nginx LogFormat

Tagged awstats, nginx, logformat, configuration  Languages 

This is my nginx LogFormat configuration:

log_format main '$remote_addr - $remote_user [$time_local] $status '
                    '"$request" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "http_x_forwarded_for"';

And this is my Awstats LogFormat configuration:

LogFormat = "%host - %host_r %time1 %code %methodurl %bytesd %refererquot %uaquot %otherquot"

My .vimrc configuration

Tagged vimrc, vim, configuration  Languages 
set guifont=Monaco:h12

autocmd FileType ruby,eruby set omnifunc=rubycomplete#Complete
autocmd FileType ruby,eruby let g:rubycomplete_rails = 1

let leader = ","

set grepprg=ack
set grepformat=%f:%l:%m

set hidden
set history=1000
set tabstop=2
set smarttab
set shiftwidth=2
set autoindent
set expandtab
set backspace=start,indent
set ignorecase
set smartcase
set scrolloff=3
set backupdir=~/.vim-tmp,~/.tmp,~/tmp,/var/tmp,/tmp
set directory=~/.vim-tmp,~/.tmp,~/tmp,/var/tmp,/tmp
set ruler
set number
set hlsearch
set incsearch " ...dynamically as they are typed.
set visualbell

nnoremap <C-e> 3<C-e>
nnoremap <C-y> 3<C-y>

runtime macros/matchit.vim

autocmd FileType make     set noexpandtab
autocmd FileType python   set noexpandtab

syntax on
filetype on
filetype plugin on
filetype indent on

let g:fuzzy_ignore = "*.log"
let g:fuzzy_matching_limit = 70

map ,d :execute 'NERDTreeToggle ' . getcwd()<CR>
map ,t :FuzzyFinderTextMate<CR>
map ,b :FuzzyFinderBuffer<CR>
map ,f gg=G

set columns=160                                                     "set width of window off open
set lines=50                                                        "height of window off open

set gcr=a:blinkwait500-blinkon1300-blinkoff150                      "tune the blinking of the cursor, short on, long off.
set gcr=i:hor10                                                     "underline cursor. not too thick not too thin. goldielocks style
set gcr=v:block                                                     "selecting should cover the text

map ,c :ruby finder.rescan!

set fuopt+=maxhorz                      " grow to maximum horizontal width on entering fullscreen mode

A simple configuration framework for storing a Rails app's configuration in YAML

Tagged yaml, rails, configuration, ostruct  Languages ruby

Create config/initializers/configuration.rb:

require 'ostruct'

# From https://www.ruby-forum.com/topic/137104
class NestedOstruct
   def self.new(hash)
     OpenStruct.new(hash.inject({}){|r,p| r[p[0]] = p[1].kind_of?(Hash) ?  NestedOstruct.new(p[1]) : p[1]; r })
   end
end

config = YAML.load_file(Rails.root.join('config', 'application.yml'))[Rails.env].symbolize_keys
AppConfig = NestedOstruct.new(config)

Create config/application.yml:

development:
  twitter:
    api_key: xxx
    api_secret: xxx
production:
...

Use it in your application:

AppConfig.twitter.api_key

How to Secure an nginx Server with Fail2Ban

Tagged nginx, fail2ban, filters, configuration, security, throttling  Languages apacheconf

Our Security Requirements

  • Block anyone trying to run scripts (.pl, .cgi, .exe, etc)
  • Block anyone trying to use the server as a proxy
  • Block anyone failing to authenticate using nginx basic authentication
  • Block anyone failing to authenticate using our application's log in page
  • Block bad bots
  • Limit the number of connections per session

We could use ModSecurity to support these requirements, but it's not compatible with nginx. We want a lightweight and easy-to-use solution. We can fulfill all these requirements with fail2ban and nginx.

Configuring Fail2ban

All except the last requirement of connection throttling is supported by Fail2Ban. To start blocking unwanted guests, put this in Fail2Ban's jail.conf file:

[nginx-auth]
enabled = true
filter = nginx-auth
action = iptables-multiport[name=NoAuthFailures, port="http,https"]
logpath = /var/log/nginx*/*error*.log
bantime = 600 # 10 minutes
maxretry = 6

[nginx-login]
enabled = true
filter = nginx-login
action = iptables-multiport[name=NoLoginFailures, port="http,https"]
logpath = /var/log/nginx*/*access*.log
bantime = 600 # 10 minutes
maxretry = 6
 
[nginx-badbots]
enabled  = true
filter = apache-badbots
action = iptables-multiport[name=BadBots, port="http,https"]
logpath = /var/log/nginx*/*access*.log
bantime = 86400 # 1 day
maxretry = 1
 
[nginx-noscript]
enabled = true
action = iptables-multiport[name=NoScript, port="http,https"]
filter = nginx-noscript
logpath = /var/log/nginx*/*access*.log
maxretry = 6
bantime  = 86400 # 1 day
 
[nginx-proxy]
enabled = true
action = iptables-multiport[name=NoProxy, port="http,https"]
filter = nginx-proxy
logpath = /var/log/nginx*/*access*.log
maxretry = 0
bantime  = 86400 # 1 day

Filter configuration

The following filter configuration files are stored in /etc/fail2ban/filter.d/:

# Proxy filter /etc/fail2ban/filter.d/nginx-proxy.conf:
#
# Block IPs trying to use server as proxy.
#
# Matches e.g.
# 192.168.1.1 - - "GET http://www.something.com/
#
[Definition]
failregex = ^<HOST> -.*GET http.*
ignoreregex =
 
# Noscript filter /etc/fail2ban/filter.d/nginx-noscript.conf:
#
# Block IPs trying to execute scripts such as .php, .pl, .exe and other funny scripts.
#
# Matches e.g.
# 192.168.1.1 - - "GET /something.php
#
[Definition]
failregex = ^<HOST> -.*GET.*(\.php|\.asp|\.exe|\.pl|\.cgi|\scgi)
ignoreregex =
 
#
# Auth filter /etc/fail2ban/filter.d/nginx-auth.conf:
#
# Blocks IPs that fail to authenticate using basic authentication
#
[Definition]
 
failregex = no user/password was provided for basic authentication.*client: <HOST>
            user .* was not found in.*client: <HOST>
            user .* password mismatch.*client: <HOST>
 
ignoreregex =
#
# Login filter /etc/fail2ban/filter.d/nginx-login.conf:
#
# Blocks IPs that fail to authenticate using web application's log in page
#
# Scan access log for HTTP 200 + POST /sessions => failed log in
[Definition]
failregex = ^<HOST> -.*POST /sessions HTTP/1\.." 200
ignoreregex =

Testing fail2ban Rules

You should test your fail2ban rules with the fail2ban-regex command:

# Test against a log file
# fail2ban-regex <log file> <filter configuration>
fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-login.conf 

# Test using strings
# fail2ban-regex <e.g. row from log file> <failure regex from filter configuration>
fail2ban-regex "127.1.1.1 - - [21/Feb/2012:10:31:08 +0200] \"POST /sessions HTTP/1.1\" 200 1532 \"https://xxx.com/sessions\" \"Mozilla/5.0 Safari/535.11\" \"-\"" "^<HOST> -.*POST /sessions HTTP/1\..\" 200"

Limiting the number of connections

We limit the number of connections to 2 per second and allow a burst of 50 with the following nginx configuration:

http {
    limit_req_zone  $binary_remote_addr  zone=app:10m   rate=2r/s; 
    ... 
    server {
         ... 
        location / {
            limit_req   zone=app burst=50;
        }

Note that you could also use Rack::Attack to implement many of these requirements.

References

http://wiki.nginx.org/HttpLimitReqModule http://www.fail2ban.org/wiki/index.php/Main_Page http://serverfault.com/questions/307575/fail2ban-doesnt-process-jail-even-though-regex-matches

Where should I store my application configuration in Sinatra?

Tagged configuration, sinatra, settings  Languages ruby

This is one option:

require 'yaml'
require 'ostruct'
require 'sinatra/base'

# Usage:
#
# config/application.yml:
#   development:
#     username: ...
#   production:
#     username: ...
#   url: http://...
#
# app.rb:
# class App < Sinatra::Base                    
#   register Sinatra::Configuration            
#   load_configuration 'config/application.yml'
#   get '/' do
#     App.configuration.username
#   end
#
module Sinatra
  module Configuration
    # From https://www.ruby-forum.com/topic/137104
    class NestedOstruct
      def self.new(hash={})
        OpenStruct.new(hash.inject({}){|r,p| r[p[0]] = p[1].kind_of?(Hash) ?  NestedOstruct.new(p[1]) : p[1]; r })
      end

      def self.to_ostruct(object)
        return case object
        when Hash
          object = object.clone
          object.each do |key, value|
            object[key] = to_ostruct(value)
          end
          OpenStruct.new(object)
        when Array
          object = object.clone
          object.map! { |i| to_ostruct(i) }
        else
          object
        end
      end
    end

    def self.registered(base)
      base.set :environments, %w[test development staging production]
    end

    def load_configuration(file, namespace = :configuration)
      set namespace, parse_configuration(YAML.load_file(file)) || {}
    end

    private

    def parse_configuration(hash)
      base = hash.except(*environments)
      base.merge!(hash[environment.to_s])
      NestedOstruct.to_ostruct(base)
    end
  end

  register Configuration
  Delegator.delegate :load_configuration
end

Git Configuration

Tagged git, configuration  Languages bash

Makes git output look better:

git config format.pretty oneline
git config color.ui true

Dot notation for Ruby configuration hash

Tagged ruby, configuration, dot-notation  Languages ruby

Convert a normal hash into a hash that allows e.g. this:

CONFIG['session.ttl'] => 60
CONFIG['session.tt'] => KeyError

Code:

def to_namespace_hash(object, prefix = nil)
  if object.is_a? Hash
    object.map do |key, value|
      if prefix
        to_namespace_hash value, "#{prefix}.#{key}"
      else
        to_namespace_hash value, "#{key}"
      end
    end.reduce(&:merge)
  else
    { prefix => object }
  end
end

CONFIG = ... # Load hash from YAML/JSON/database/etc
CONFIG = to_namespace_hash(CONFIG).freeze

def CONFIG.[](key)
  fetch(key)
rescue KeyError => e
  possible_keys =keys.map { |x| x if x.match /.*?#{key}.*?/i }.delete_if(&:blank?).join("\n")
  fail KeyError, "Key '#{key}' not found. Did you mean one of:\n#{possible_keys}"
end