What belongs in a queue
- External API calls, which can be slow and can fail
- Email and notifications
- Report and document generation
- Data processing of any size
- Anything the user does not need to wait for
A web request should do the minimum required to respond. Everything else belongs behind a queue, where it can be retried without the user knowing.
Task design
- Idempotent — running twice must not double the effect
- Small arguments — pass identifiers, not whole objects
- Bounded retries, with backoff
- Explicit timeouts, so a stuck task does not hold a worker forever
- Logged, with a correlation identifier from the originating request
Monitor the queue, not just the workers
| Signal | Means |
|---|---|
| Queue depth growing | Workers stopped or too few |
| Task duration increasing | Something is degrading |
| Failed queue growing | Something is systematically wrong |
| Zero tasks processed | Workers are not running |
| Retries increasing | An external dependency is struggling |
Queue depth is the single most useful monitor in a queued system, and it is frequently the one nobody set up.
The failed task queue needs an owner
Most systems have a dead letter queue that was built during development and has never been checked. Work fails, lands there, and stays.
Somebody should look at it regularly, and it should alert when it grows. Otherwise it is a silent hole where work disappears.
Workers need supervision
A worker process that exits leaves work accumulating with no error anywhere visible. It needs a process supervisor to restart it and monitoring to notice if it does not.
That is basic operational plumbing and it is what separates a system that recovers from one that quietly stops.