Converting a string to a number with CRC

Tagged string, number, crc, conversion  Languages ruby

CRC can be used to convert a string into a number:

require 'zlib'
integer = Zlib.crc32('snippet')
=> 2518453461
=> 2656977832

The CRC algorithm is normally used to detect changes or errors in large chunks of data.

Waiting for Socket (IO) to be readable / writable in Ruby

Tagged writable, io-wait, readable, ruby, socket  Languages ruby

Option 1: Use

Use with read_nonblock, write_nonblock, and connect_nonblock.

Option 2: Use io-wait

require 'io/wait'
r, w = IO.pipe

Example from

# 30 second wait[mysock],[mysock], nil, 30)
# as opposed to (60 second wait)
require 'io/wait'
mysock.wait_readable(30) && mysock.wait_writable(30)

How to simulate TCP read, write, and connect timeouts

Tagged tcp, timeout, write, connect, read  Languages 

How to simulate TCP read, write, and connect timeouts:

  • Connect timeout

Option 1: Drop all SYN packets with firewall or iptables rules

Option 2: Try connecting to a non-routable IP, e.g.,

  • Read timeout

Read from a socket to which the client or server is not writing while keeping the socket open on both ends.

  • Write timeout

You’re (almost) out of luck. There are OS-level buffers, so the write timeout might never happen.

Option 1: Use and LD_PRELOAD to mock syscalls


Advisory locks in Linux

Tagged flock, lock, advisory  Languages bash

Use Linux advisory locks to prevent concurrent execution of scripts:

flock -nx /var/lock/scriptx

REST API thead-safety feature

Tagged request id, safe, rest, thread-safety  Languages 

REST API safety feature:

  1. Include a request ID in the request and response.
  2. Check that the response contains the request ID.


  • You will be able to notice any concurrency issues in your own code and 3rd party code, which might cause the wrong response to be returned
  • Easier to debug requests and responses

Ruby TCP socket with read, write, and connect timeout

Tagged write, connect, read, ruby, socket, tcp, timeout  Languages ruby

See Ruby’s Net::Protocol implementation for an example on how to implement connect, read, and write timeouts:

Or, write your own custom TCP socket implementation that supports read, write, and connect timeouts (warning, not tested):

require 'socket'
require 'timeout'

class TCPSocketWithTimeout
  class Error < StandardError; end
  class Timeout < StandardError; end
  class ReadTimeout < Timeout; end
  class WriteTimeout < Timeout; end
  attr_reader :host, :port, :tls, :read_timeout, :write_timeout, :connect_timeout, :socket

  def initialize(host:, port:, tls: false, connect_timeout: 5, read_timeout: 5, write_timeout: 5)
    @tls = tls
    @host = host
    @port = port
    @write_timeout = write_timeout
    @read_timeout = read_timeout
    @connect_timeout = connect_timeout
  def ssl_context
    ctx =
    ctx.ssl_version = :TLSv1_2
    ctx.ca_file = ca_file if ca_file
    ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER

  # NOTE: Upgrading Ruby 2.7 might change the Ruby's socket API
  def init_socket
    if tls
      sock =, port), # opens connection to server
      # Close both socket and encrypted layer
      sock.sync_close = true
      sock =, :STREAM, 0)
    sock.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)

  def connect(timeout: CONNECT_TIMEOUT)
    @socket = init_socket
    deadline = + timeout
    non_blocking(socket, deadline) do
      # NOTE: different method arity for non-SSL
      if tls
        socket_address = Socket.pack_sockaddr_in(port, host)
  rescue Errno::EISCONN
    # Connection established
  rescue Timeout, Errno::ETIMEDOUT => e
    raise ConnectTimeout, "Connection timeout after #{timeout} seconds trying to connect to '#{host}:#{port}': #{e.class}: #{e.message}"
  rescue SystemCallError, IOError => e
    raise Error, "Connection failure while connecting to '#{host}:#{port}': #{e.class}: #{e.message}"

  def read(timeout: READ_TIMEOUT, length: 1024)
    deadline = + timeout
    non_blocking(socket, deadline) do
  rescue Timeout
    raise ReadTimeout, "Timeout after #{timeout}s while reading data from #{host}:#{port}"
  rescue SystemCallError, IOError => e
    raise Error, "Connection error while reading data from #{host}:#{port} #{e.class}: #{e.message}"

  def write(data, timeout: WRITE_TIMEOUT)
    deadline = + timeout
    length = data.bytesize
    total_count = 0
    non_blocking(socket, deadline) do
      loop do
        count = socket.write_nonblock(data)
        total_count += count
        return total_count if total_count >= length

        data = data.byteslice(count..-1)
  rescue Timeout
    raise WriteTimeout, "Timeout after #{timeout}s while writing data to #{host}:#{port}"
  rescue SystemCallError, IOError => e
    raise Error, "Connection error while writing data to #{host}:#{port} #{e.class}: #{e.message}"

  def disconnect

  def non_blocking(socket, deadline)
    raise Error, "Socket #{host}:#{port} is closed" if closed?

  rescue IO::WaitReadable => e
    time_remaining = calculate_remaining_time(deadline)
    raise Timeout, e unless[socket], nil, nil, time_remaining)

  rescue IO::WaitWritable => e
    time_remaining = calculate_remaining_time(deadline)
    raise Timeout, e unless, [socket], nil, time_remaining)


  def calculate_remaining_time(deadline)
    time_remaining = deadline -
    raise Timeout if time_remaining.negative?


sock = 'localhost', port: 8888)
sock.write "HELLO"
puts "Writing done"


A basic TCP server that can be used to test the client:

require 'socket'

server =

while client = server.accept
  puts "Accepted"
  puts "Received #{}"
  puts "Wrote #{client.write('Hello back')}"


Use non-blocking methods and to implement timeouts using non-blocking methods.

Use TCPSocket with TLS connections.

Use Socket with SSL connections.

TCPSocket in Ruby version 3 and greater includes a connect_timeout parameter in the constructor:

Ruby’s socket API is a work in progress…


Socket connection timeout in Ruby:

Working with TCP sockets:

Ruby’s socket API and related documentation is not that great, so you might need to read the source:

Celery tips

Tagged celery, tips  Languages 
  1. Set a global task timeout

The default is no timeout.


  1. Use autoretry_for

By default, no exceptions will be retried.

  1. Set max_retries at task or global level

The default is 3 retries. None means unlimited retries.

  1. Use queues

Otherwise, low priority tasks might prevent higher priority tasks from being executed.

@app.task(base=SqlAlchemyTask, queue=’medium’)

  1. Use Flower for monitoring

  1. Separate slow tasks from fast tasks

Slow tasks will monopolizes workers, even if you have separate queues.

  1. Use late_ack=True

Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.

  1. Make tasks idempotent

  2. There’s no exactly-once

Exactly-once is a lie.


Escape and sanitize input and output?

Tagged escape, input, output, sanitize  Languages 

Thinking about escaping and sanitizing input and output? Read this:


  • Always sanitize output

By sanitizing the output, you can fix issues caused by bugs in the input sanitizer without touching the database.

  • Sanitize input, if it makes sense:

For example, if there’s a risk that input becomes invalid after sanitization then it’s better to store the data raw.

This is also a good idea, if there are multiple output formats and sanitizing the input might break something.