mDNS and DNS-SD example

Tagged dns, dns-sd, mdns, service discovery  Languages bash

To advertise the existence of an HTTP service with mDNS, use:

dns-sd -R "Service X" _http._tcp . 80 path=/

To find the HTTP services on the local network, run:

dns-sd -B _http._tcp

Browsing for _http._tcp
DATE: ---Wed 03 Mar 2021---
17:48:45.207  ...STARTING...
Timestamp     A/R    Flags  if Domain               Service Type         Instance Name
17:48:45.207  Add        2   9 local.               _http._tcp.          Service X

See /etc/services for a list of valid services.

Namespaced singular routes for uncountables in Rails

Tagged singular_route_key, namespace  Languages ruby

Correct(ing) singular route for an uncountable namespaced model in Rails:

Problem:

$ rails console
> Namespace::SMS.model_name.singular_route_key => namespace_sm
> app.polymorphic_path(Namespace::SMS.first) => NoMethodError namespace_sm_path

Solution:

ActiveSupport::Inflector.inflections(:en) do |inflect|
  inflect.uncountable %w(namespace_sms)
end

Voila:

$ rails console
> Namespace::SMS.model_name.singular_route_key => namespace_sms

Table-driven Programming

Tagged table-driven, programming  Languages python

You can use if statements to run a function when the state of your application matches specific criteria.

However, if statements don’t scale as they are hard to understand and maintain:

if state_x and state_y and state_z:
    do_xyz()

if state_x and state_z:
    do_xz()

if state_x and state_y and not state_z:
    do_xy_not_z()

Table-driven programming is an alternative that sometimes is easier to maintain:

rules = (
    # x, y, z, function
    (True, True, True, do_xyz),
    (True, True, False, do_xy_not_z),
    (True, False, True, do_xz),
    (True, True, True, do_xz),
)
for rule in rules:
    rule_x = rule[0]
    rule_y = rule[1]
    rule_z = rule[2]
    doit = rule[3]
    if rule_x == state_x and rule_y == state_y and rule_z == state_z:
        doit()
    else:
        print("Skipping")

Or, more succinctly:

def matching_rules(rules, params):
    for criterion, func in rules:
        if all(params[ix] == criteria for ix, criteria in enumerate(criterion)):
            yield func

# The table of rules
rules = (
    # x, y, z, function
    ((True, True, True), do_xyz),
    ((True, True, False), do_xy_not_z),
    ((True, False, True), do_xz),
    (True, True, True, do_xz),
)
params = (state_x, state_y, state_z)
for func in matching_rules(rules, params):
    func()

In summary, a function is run only when the criteria match.

Pattern matching is also an alternative: https://www.python.org/dev/peps/pep-0635/

However, the first-to-match rule requires the order to be correct and prevents multiple function calls.

State machines and Prolog are also options…

SQLAlchemy's yield_for with raw SQL and models

Tagged sqlalchemy, yield_for, raw sql  Languages python
from sqlalchemy import text

session = ...
q = session.query(Report).from_statement(text("""
SELECT
    id
FROM
    reports
WHERE
    id = '51812'
ORDER BY
    id DESC;
""")).yield_per(100)

Python's etree and namespaces

Tagged python, namespaces, etree  Languages python

How to parse XML with Python’s etree when XML has namespaces, even a default namespace:

from lxml import etree
doc = etree.XML(bytes(bytearray(xml, encoding='utf-8')))
ns = {}
for k in doc.nsmap.keys():
   ns[k] = doc.nsmap[k]
doc.find('.//tag', ns)
doc.findtext('./periodOfReport', namespaces=ns)

LOL, someone needs to clean up the API.

How to set the environment variables in a cron script (Docker)

Tagged env, cron, docker, crontab, environment, docker-compose  Languages bash

To give your cron script access to the same environment variables as the Docker container, you can read and export the environment variables for the PID 1 process in your script:

crontab -l
* * * * * /app/run.sh jobs.xxx

/app/run.sh

#!/usr/bin/env bash
# Read the environment variables for the main process (PID 1) running in the Docker container:
export $(xargs -0 -a "/proc/1/environ")

python3 -m $1

Ruby's to_json in Python

Tagged as_json, to_json, __dict__  Languages python
from json import JSONEncoder

class MyEncoder(JSONEncoder):
    def default(self, o):
         if isinstance(obj, XY):
             return { 'x': obj.x, 'y': obj.y }
        return o.__dict__

def to_json2(value):
    return MyEncoder(indent=2).encode(value)

URL join for Python

Tagged join, python, uri, url  Languages python
def url_join(*parts):
    # This does not work with many parts:
    # import urllib.parse
    # urllib.parse.urljoin(*parts)
    return os.path.join(*parts)

Ban invalid HTTP requests with fail2ban

Tagged https, ssl, fail2ban, hex, nginx  Languages bash

The goal is to use fail2ban to block invalid HTTP requests sent to nginx from Iran:

"45.132.170.81 - - [04/Jan/2021:02:46:12 +0000] "\x10\xD2\xE9\xA68\x8A\x98\xB3\x00\xB9mO\xD7\xC9\xAA]"

The request is a HEX encoded string instead of the normal “GET /“.

First, configure the jail:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Add the following at the end of the file:

[nginx-idiots]
enabled  = true
port     = http,https
filter   = nginx-idiots
logpath  = /var/log/nginx/access.log
# NOTE: Docker will override fail2ban's rules, so good luck with iptables:
# logpath = /var/lib/docker/containers/*/*.log
bantime  = 86400
findtime = 86400
maxretry = 2

Add a filter to block the idiots:

sudo vim /etc/fail2ban/filter.d/nginx-idiots.conf
[Definition]
failregex = ^.* <HOST> .*".*\\x.*".*$
ignoreregex =

Test the regex:

sudo fail2ban-regex -v --print-all-missed /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-idiots.conf

Restart fail2ban.

View banned IPs with:

sudo fail2ban-client status nginx-idiots

Note that this could also be a real idiot that is sending SSL traffic to the non-SSL port, or someone who has configured SSL traffic to be sent to the non-SSL port.

Customizing Rails SQL dumps

Tagged sql, structure, rails, dump  Languages ruby

config/application.rb:

require_relative 'boot'

require 'rails/all'

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

module YourApp
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 6.0
    # Set to :all to dump all schemas
    config.active_record.dump_schemas = :all
    # Don't dump in Docker
    config.active_record.dump_schema_after_migration = !File.exist?('/.dockerenv')
    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration can go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded after loading
    # the framework and any gems in your application.
    config.active_record.schema_format = :sql
  end
end

If all else fails, override the rake task in Rakefile:

Rake::Task["db:schema:dump"].clear
Rake::Task["db:structure:dump"].clear

namespace :db do
  namespace :schema do
    task :dump do
      Kernel.system "pg_dump -s -x -O -f db/structure.sql --schema=xxx xxx_development && sed do_xyz_with_dump"
    end
  end
end