celery snippets

Celery tips

Tagged celery, tips  Languages 
  1. Set a global task timeout

The default is no timeout.

CELERYD_TASK_SOFT_TIME_LIMIT = 30 CELERYD_TASK_TIME_LIMIT = 60

  1. Use autoretry_for

By default, no exceptions will be retried.

  1. Set max_retries at task or global level

The default is 3 retries. None means unlimited retries.

  1. Use queues

Otherwise, low priority tasks might prevent higher priority tasks from being executed.

@app.task(base=SqlAlchemyTask, queue=’medium’)

  1. Use Flower for monitoring

https://docs.celeryproject.org/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor

  1. Separate slow tasks from fast tasks

Slow tasks will monopolizes workers, even if you have separate queues.

  1. Use late_ack=True

Setting late_ack to True means we acknowledge the message after the task has run, not when received. This increases reliability.

  1. Make tasks idempotent

  2. There’s no exactly-once

Exactly-once is a lie.

Reference:

Troubleshooting Python's Celery

Tagged celery, python  Languages python

Task celery.chord_unlock[38d5105a-12f2-4119-80e5-184167998f4b] retry: Retry in 1.0s

The notes https://docs.celeryproject.org/en/latest/userguide/canvas.html#chords:

If you’re using chords with the Redis result backend and also overriding the Task.after_return() method, you need to make sure to call the super method or else the chord callback won’t be applied.

TypeError: task() argument after ** must be a mapping, not list

missing 3 required positional arguments

If you get this error:

TypeError: after_return() missing 3 required positional arguments: ‘args’, ‘kwargs’, and ‘einfo’

You might have specified the arguments incorrectly, for example:

job = job_task.subtask(1, 2, 3)

Use a list to fix the error:

job = job_task.subtask((1, 2, 3))