tips snippets

Celery tips

Tagged celery, tips  Languages 
  1. Set a global task timeout

The default is no timeout.


  1. Use autoretry_for

By default, no exceptions will be retried.

  1. Set max_retries at task or global level

The default is 3 retries. None means unlimited retries.

  1. Use queues

Otherwise, low priority tasks might prevent higher priority tasks from being executed.

@app.task(base=SqlAlchemyTask, queue=’medium’)

  1. Use Flower for monitoring

  1. Separate slow tasks from fast tasks

Slow tasks will monopolizes workers, even if you have separate queues.

  1. Use late_ack=True

Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.

  1. Make tasks idempotent

  2. There’s no exactly-once

Exactly-once is a lie.