Celery tips

Tagged tips, celery  Languages 
  1. Set a global task timeout

The default is no timeout.

CELERYD_TASK_SOFT_TIME_LIMIT = 30 CELERYD_TASK_TIME_LIMIT = 60

  1. Use autoretry_for

By default, no exceptions will be retried.

  1. Set max_retries at task or global level

The default is 3 retries. None means unlimited retries.

  1. Use queues

Otherwise, low priority tasks might prevent higher priority tasks from being executed.

@app.task(base=SqlAlchemyTask, queue=’medium’)

  1. Use Flower for monitoring

https://docs.celeryproject.org/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor

  1. Separate slow tasks from fast tasks

Slow tasks will monopolizes workers, even if you have separate queues.

  1. Use late_ack=True

Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.

  1. Make tasks idempotent

  2. There’s no exactly-once

Exactly-once is a lie.

Reference:

Escape and sanitize input and output?

Tagged output, input, escape, sanitize  Languages 

Thinking about escaping and sanitizing input and output? Read this: https://security.stackexchange.com/questions/95325/input-sanitization-vs-output-sanitization

Rules:

  • Always sanitize output

You can fix bugs in the input sanitizer without touching the database by always sanitizing output.

  • Sanitize input, if it makes sense:

For example, if there’s a risk that input becomes invalid after sanitization it’s better to store data raw. Also, when there are multiple output formats sanitizing might break something.

faiss-cpu Dockerfile

Tagged faiss, dockerfile  Languages 

Dockerfile:

FROM continuumio/anaconda3

RUN conda install faiss-cpu -c pytorch

WORKDIR /app

RUN pip install --upgrade pip
ADD requirements.txt /app/
RUN pip install -r requirements.txt

requirements.txt:

...

gid, uid in Dockerfile

Tagged dockerfile, gid, uid  Languages bash
# Debian
RUN addgroup --gid 10000 app && adduser --gid 10000 --uid 10001 app && chown -R app /app/ && chgrp -R app /app/ && chmod -R 740 /app/

# Alpine
RUN addgroup --gid 10000 app && adduser -u 10001 app -D -G app && chown -R app /app/ && chgrp -R app /app/ && chmod -R 740 /app/

Add the user to the host system too.

ActiveRecord after_create/after_save and associations

Tagged rails, after_create, association, after_save  Languages ruby

The order of ActiveRecord callbacks matter when for example they need to access associations (e.g. has_many).

For example, the calculate_doors will be called before the associations are saved to the database if you use car.doors.build(name: :backdoor):

class Car < ApplicationRecord
  after_create :calculate_doors
  has_many :doors

This can lead to subtle errors.

Changing the order will fix the issue:

class Car < ApplicationRecord
  has_many :doors
  after_create :create_doors

Tested with Rails 6.1. Other versions might work differently.

See https://api.rubyonrails.org/classes/ActiveRecord/AutosaveAssociation.html for details:

Placing your callbacks after associations is usually a good practice.