The request should be fast
A user submitting a form should not wait while an email is sent, a PDF is generated and three external systems are notified. Each of those can fail, and each makes the request slower.
Sending email during a web request is the most common cause of slow form submissions, and the most common reason a form appears to fail when it actually succeeded.
What belongs in a queue
- Email and notifications
- Calls to external APIs
- Report and document generation
- Image and file processing
- Bulk data operations
- Anything that takes more than a moment
What queues give you
- Fast responses, because the work happens afterwards
- Retries when something transient fails
- Durability — work survives a restart
- Rate control for external services with limits
- Visibility into what failed and why
Design jobs to be retryable
| Requirement | Why |
|---|---|
| Idempotent | A retry must not duplicate the effect |
| Small payload | Pass an identifier, not the whole object |
| Bounded attempts | Failing jobs must not retry forever |
| Failure handling | A failed job queue somebody actually checks |
| Logged | So a missing outcome can be traced |
The failed job queue is the part most commonly built and never monitored. Work fails silently and nobody knows.
Workers need supervision
A queue worker that stops leaves work piling up with no error anywhere. It needs a process supervisor to restart it and monitoring to alert when the queue is growing.
Alerting on queue depth is the single most useful monitor in a queued system.