Ruby's as_json in Python

Tagged as_dict, as_json  Languages python
class Document():
    def as_dict(self, without=[]):
        res = {}
        for prop in vars(self):
            if prop in without:
                continue
            res[prop] = getattr(self, prop)
        return res

You could also simply implement and use dict, if without is not needed.

URL join for Python

Tagged join, python, uri, url  Languages python
def url_join(*parts):
    # This does not work with many parts:
    # import urllib.parse
    # urllib.parse.urljoin(*parts)
    return os.path.join(*parts)

Ban invalid HTTP requests with fail2ban

Tagged fail2ban, hex, nginx, https, ssl  Languages bash

The goal is to use fail2ban to block invalid HTTP requests sent to nginx from Iran:

"45.132.170.81 - - [04/Jan/2021:02:46:12 +0000] "\x10\xD2\xE9\xA68\x8A\x98\xB3\x00\xB9mO\xD7\xC9\xAA]"

The request is a HEX encoded string instead of the normal “GET /“.

First, configure the jail:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Add the following at the end of the file:

[nginx-idiots]
enabled  = true
port     = http,https
filter   = nginx-idiots
logpath  = /var/log/nginx/access.log
# NOTE: Docker will override fail2ban's rules, so good luck with iptables:
# logpath = /var/lib/docker/containers/*/*.log
bantime  = 86400
findtime = 86400
maxretry = 2

Add a filter to block the idiots:

sudo vim /etc/fail2ban/filter.d/nginx-idiots.conf
[Definition]
failregex = ^.* <HOST> .*".*\\x.*".*$
ignoreregex =

Test the regex:

sudo fail2ban-regex -v --print-all-missed /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-idiots.conf

Restart fail2ban.

View banned IPs with:

sudo fail2ban-client status nginx-idiots

Note that this could also be a real idiot that is sending SSL traffic to the non-SSL port, or someone who has configured SSL traffic to be sent to the non-SSL port.

Customizing Rails SQL dumps

Tagged dump, rails, sql, structure  Languages ruby

config/application.rb:

require_relative 'boot'

require 'rails/all'

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

module YourApp
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 6.0
    # Set to :all to dump all schemas
    config.active_record.dump_schemas = :all
    # Don't dump in Docker
    config.active_record.dump_schema_after_migration = !File.exist?('/.dockerenv')
    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration can go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded after loading
    # the framework and any gems in your application.
    config.active_record.schema_format = :sql
  end
end

If all else fails, override the rake task in Rakefile:

Rake::Task["db:schema:dump"].clear
Rake::Task["db:structure:dump"].clear

namespace :db do
  namespace :schema do
    task :dump do
      Kernel.system "pg_dump -s -x -O -f db/structure.sql --schema=xxx xxx_development && sed do_xyz_with_dump"
    end
  end
end

How to use PostgreSQL advisory locks with SQLAlchemy and Python

Tagged python, advisory lock, pg_try_advisory_lock, sqlalchemy  Languages python

advisory_lock.py

from models import Session
from sqlalchemy import func, select


def execute(session, lock_fn, lock_id, scope):
    """
    Executes the lock function
    """
    return session.execute(select([lock_fn(lock_id, scope)])).scalar()


def obtain_lock(session, lock_id, scope):
    """
    Obtains the advisory lock
    """
    lock_fn = func.pg_try_advisory_lock
    return execute(session, lock_fn, lock_id, scope)


def release_lock(session, lock_id, scope):
    """
    Releases the advisory lock
    """
    lock_fn = func.pg_advisory_unlock
    return execute(session, lock_fn, lock_id, scope)


def with_lock(my_func, lock_id, scope=1):
    """
    Executes my_func if the lock can be obtained.
    """
    session = Session()
    obtained_lock = False
    try:
        obtained_lock = obtain_lock(session, lock_id, scope)
        if obtained_lock:
            my_func()
    finally:
        if obtained_lock:
            release_lock(session, lock_id, scope)

Usage:

from advisory_lock import with_lock

def run():
    print("It runs")

if __name__ == '__main__':
    with_lock(run, 300000)

UFW + Docker = No firewall

Tagged docker, elasticsearch, gotcha, iptables, ufw, wtf  Languages 

TLDR: Docker can and will override existing iptable rules and expose your services to the Internet

This means you have to think twice when installing Docker on a machine that is only protected by an iptables-based firewall such as UFW. You might think you are protected by your firewall, but you very likely are not. This is probably one of the more common reasons why Elasticsearch servers, which are unprotected by default, are exposed to the internet.

For details, see: https://github.com/docker/for-linux/issues/690

Solution 1: External firewall

One solution is to use a firewall provided by the hosting provider (DO, AWS, GCP, etc.).

Solution 2: Disable Docker’s iptables “feature”

Disable iptables in Docker by adding the following switch:

--iptables=false

Solution 2: Listen on private IPs

This is perhaps the easiest to implement and easiest to forget: expose your containers and services on one of the following private IP address ranges:

  • 10.0.0.0 to 10.255.255.255
  • 172.16.0.0 to 172.31.255.255
  • 192.168.0.0 to 192.168.255.255

Note that binding to 127.0.0.1 will not work with Docker Swarm.