The short answer
Complete success and complete failure are both easy. The difficult case is a process that did three of its five steps and then stopped, leaving inconsistent state behind.
Design so repeating a step is harmless, and so a person can always finish or unwind the work.
Make steps safe to repeat
- Give each piece of work an identifier that travels with it.
- Check whether a step has already been done before doing it.
- Record completion of each step, not just of the whole process.
- Never let a retry send a second message or create a second record.
- Test by deliberately running the same work twice.
Point four is where the visible damage happens. Duplicate invoices, duplicate emails and duplicate orders are what customers notice from a retry loop.
Retry sensibly
| Failure | Response |
|---|---|
| Temporary network problem | Retry with increasing gaps |
| Upstream system busy | Back off, do not hammer it |
| Invalid data | Do not retry, route to a person |
| Credentials rejected | Stop and alert, retrying will not help |
| Unknown error | Retry a few times, then escalate |
The third and fourth rows matter. Retrying something that can never succeed wastes capacity and buries the real alert among repeated failures.
Keep a manual route
- A way to see exactly where a piece of work stopped
- A way to complete the remaining steps by hand
- A way to mark work as handled outside the system
- A way to unwind steps already taken, where that is possible
- Someone who knows how to do all of the above
The last point is the one that lapses. A manual route nobody has used in a year is not a route, and the incident is a poor time to learn it.
Make stuck work visible
Work that failed and is waiting needs to appear somewhere a person looks daily. A failure queue nobody opens is the same as losing the work.
Count the items in it and alert when the count grows, because the individual failures matter less than the fact that nobody is clearing them.