worker snippets

Checklist for how to write background jobs/workers

Tagged at-least-once, at-most-once, order, sidekiq, checklist, worker  Languages 


  1. Idempotent

    Jobs should be idempotent. If a job is performed multiple times the work should only be performed once, or at least the end-result should be the same.

  1. Optimistic/pessimistic/distributed locking

    You need to use different types of strategies to prevent code from being executed more than once in a distributed environment. Jobs can be executed twice under many circumstances, e.g. because of:

Solving these issues can be done with, e.g.: optimistic locking, pessimistic locking, and distributed locking, e.g. with redis-semaphore.

  1. Retry Set retry to zero for jobs that don’t need to or shouldn’t be retried.
  1. Priority Specify a queue and a queue priority for jobs that need a higher or lower priority.
  1. Transactions Does the job need database transactions? Use transactions e.g. when a job does more than one insert, update, delete, or create, otherwise you will end up having data duplication/inconsistency issues.
  1. Messaging Sidekiq is not a messaging queue. At-least-once execution is 100% guaranteed by Sidekiq Pro. I don’t trust that guarantee 100%, but that is another story. The question is: what happens if the job fails (dead job queue, dev null)? What you usually need is “job successfully executed at-least-once” not “job executed at-least once”.

Ensuring jobs are performed at-least-once and at-most-once

You can’t rely on Sidekiq, or any other messaging or job queue, to execute your job (business logic) once and exactly once. All work needs to be checked for consistency.

For example, messages/jobs can be lost under many circumstances, and the same job (business logic) can be executed an unlimited amount of times unless your application takes the necessary steps to ensure that the job is performed correctly. A background processing/messaging server should be considered unreliable.


Use a reconsiliation step, e.g. a separate worker that runs once per day, to make sure jobs are executed at least once:

  1. Databases: At-least-once is achieved when you have the data in your database and are polling to see if the job has been performed. Locking => performance issues.
  2. Sidekiq: At-least-once is achieved when you monitor the dead job queue 100% using available eyeballs.
  3. Messaging queues: At-least-once is achieved through message acknowledgment and/or through monitoring the dead letter queue 100% using available eyeballs.


Use a flag, nonce, or a guid to ensure a job is only run once, i.e. do nothing if a job is queued twice for some reason. Avoid locking database tables when possible. Use optimistic/pessimistic/distributed locks when locking is needed.


  • Should jobs be executed in order?
  • What happens if job is executed, e.g., 1 month too late?


Don’t trust your messaging and background processing server to do its job without either manual supervision (eyeballs) or automated supervision (at-least-once and at-most-once).

Golang Worker Pattern Example

Tagged channel, golang, worker, go, pattern  Languages go

An example of the worker pattern in Go.

Executes, e.g., 20 units of work with 20 workers, taking 2 seconds each, in 2 seconds.

package main

// $ go build worker.go && WORKERS=20 JOBS=20 ./worker
import (

// A unit of work
type Rabbit struct {
 ID   int
 Name string

// The worker accepting a unit of work
type RabbitWork struct {
 Rabbit Rabbit

func main() {
 jobCount, _ :=     strconv.Atoi(os.Getenv("JOBS"))
 workerCount, _ :=  strconv.Atoi(os.Getenv("WORKERS"))
 start := time.Now()
 workers := make(chan RabbitWork, workers)
 var wg sync.WaitGroup

 // Add  workers to our startup "Icarus"
 for i := 0; i < workerCount; i++ {
  go func() {
   for work := range workers {
    // do some work on data
    var rabbit = work.Rabbit
    fmt.Println("Capturing", rabbit.Name, "(", rabbit.ID, ")")
    time.Sleep(2 * time.Second)
    fmt.Println("Skinned", rabbit.Name, "(", rabbit.ID, ")")

 // Give some work to the workers
 for i := 1; i <= jobCount; i++ {
  workers <- RabbitWork{
   Rabbit: Rabbit{Name: "Toffe", ID: i},

 // Wait for workers to do their job
 fmt.Println("Work took", time.Since(start).Seconds(), "seconds")


  1. WaitGroup waits for all work to finish
  2. wg.Wait() blocks until wg.Done() is called 20 times
  3. workers <- RabbitWork will block if all 20 workers are busy working
  4. for data := range tasks iterates over incoming work sent through the tasks channel
  5. close(workers) closes the channel
  6. 40 units of work with 20 workers will complete in 4 seconds
  7. also see: