Celery tips
- Set a global task timeout
The default is no timeout.
CELERYD_TASK_SOFT_TIME_LIMIT = 30 CELERYD_TASK_TIME_LIMIT = 60
- Use autoretry_for
By default, no exceptions will be retried.
- Set max_retries at task or global level
The default is 3 retries. None means unlimited retries.
- Use queues
Otherwise, low priority tasks might prevent higher priority tasks from being executed.
@app.task(base=SqlAlchemyTask, queue=’medium’)
- Use Flower for monitoring
- Separate slow tasks from fast tasks
Slow tasks will monopolizes workers, even if you have separate queues.
- Use late_ack=True
Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.
Make tasks idempotent
- There’s no exactly-once
Exactly-once is a lie.
Reference: