Celery tips

Tagged celery, tips  Languages 
  1. Set a global task timeout

The default is no timeout.

CELERYD_TASK_SOFT_TIME_LIMIT = 30 CELERYD_TASK_TIME_LIMIT = 60

  1. Use autoretry_for

By default, no exceptions will be retried.

  1. Set max_retries at task or global level

The default is 3 retries. None means unlimited retries.

  1. Use queues

Otherwise, low priority tasks might prevent higher priority tasks from being executed.

@app.task(base=SqlAlchemyTask, queue=’medium’)

  1. Use Flower for monitoring

https://docs.celeryproject.org/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor

  1. Separate slow tasks from fast tasks

Slow tasks will monopolizes workers, even if you have separate queues.

  1. Use late_ack=True

Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.

  1. Make tasks idempotent

  2. There’s no exactly-once

Exactly-once is a lie.

Reference: