Celery tips

Tagged tips, celery  Languages 
  1. Set a global task timeout

The default is no timeout.


  1. Use autoretry_for

By default, no exceptions will be retried.

  1. Set max_retries at task or global level

The default is 3 retries. None means unlimited retries.

  1. Use queues

Otherwise, low priority tasks might prevent higher priority tasks from being executed.

@app.task(base=SqlAlchemyTask, queue=’medium’)

  1. Use Flower for monitoring


  1. Separate slow tasks from fast tasks

Slow tasks will monopolizes workers, even if you have separate queues.

  1. Use late_ack=True

Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.

  1. Make tasks idempotent

  2. There’s no exactly-once

Exactly-once is a lie.


Escape and sanitize input and output?

Tagged input, sanitize, output, escape  Languages 

Thinking about escaping and sanitizing input and output? Read this: https://security.stackexchange.com/questions/95325/input-sanitization-vs-output-sanitization


  • Always sanitize output

You can fix bugs in the input sanitizer without touching the database by sanitizing the output.

  • Sanitize input, if it makes sense:

For example, if there’s a risk that input becomes invalid after sanitization then it’s better to store the data raw. Also, when there are multiple output formats sanitizing input might break something.

faiss-cpu Dockerfile

Tagged dockerfile, faiss  Languages 


FROM continuumio/anaconda3

RUN conda install faiss-cpu -c pytorch


RUN pip install --upgrade pip
ADD requirements.txt /app/
RUN pip install -r requirements.txt



gid, uid in Dockerfile

Tagged dockerfile, gid, uid  Languages bash
# Debian
RUN addgroup --gid 10000 app && adduser --gid 10000 --uid 10001 app && chown -R app /app/ && chgrp -R app /app/ && chmod -R 740 /app/

# Alpine
RUN addgroup --gid 10000 app && adduser -u 10001 app -D -G app && chown -R app /app/ && chgrp -R app /app/ && chmod -R 740 /app/

Add the user to the host system too.

ActiveRecord after_create/after_save and associations

Tagged after_save, rails, association, after_create  Languages ruby

The order of ActiveRecord callbacks matter when, for example, using associations (e.g. has_many).

For example, the calculate_doors hook will be called before the associations are saved to the database if you use car.doors.build(name: :backdoor):

class Car < ApplicationRecord
  after_create :calculate_doors
  has_many :doors

This can lead to subtle errors.

Changing the order will fix the issue:

class Car < ApplicationRecord
  has_many :doors
  after_create :create_doors

Tested with Rails 6.1. Other versions might work differently.

See https://api.rubyonrails.org/classes/ActiveRecord/AutosaveAssociation.html for details:

Placing your callbacks after associations is usually a good practice.