What belongs in a queue
- Email and notifications
- Calls to external systems
- Report and document generation
- Bulk operations across many records
- Anything taking more than a moment
Sending email during a request is the most common cause of a slow form submission and of a form that appears to fail when it actually succeeded.
Task design
- Pass identifiers, not objects — the record may change before the task runs
- Idempotent — a retry must not double the effect
- Bounded retries with backoff
- Explicit timeouts
- Logged, with the originating request identifier
Watch out for transactions
A task queued inside a database transaction may start before the transaction commits, and then fail to find the record it was given.
Queue tasks after the transaction commits. Django provides a hook for exactly this, and it prevents a class of intermittent failure that is difficult to diagnose.
Monitor the queue
| Signal | Means |
|---|---|
| Depth growing | Workers stopped or too few |
| Failed queue growing | Something systematically wrong |
| Zero tasks processed | Workers not running |
| Duration increasing | Something degrading |
| Retries increasing | An external dependency struggling |
The failed queue needs an owner
Most applications have a dead letter queue built during development and never checked. Work fails, lands there, and stays there indefinitely.
It should alert when it grows and somebody should review it regularly. Otherwise it is a silent hole where work disappears.