The short answer
Automations rarely announce their own failure. They stop running, or they run and produce nothing, and the first sign is a consequence somebody notices later.
Alerting on absence rather than only on errors is the fix, and it is straightforward.
How they fail
| Failure | Why it is silent |
|---|---|
| Scheduled run stops firing | No error, just nothing |
| Upstream format changes | Processes zero records successfully |
| Credentials expire | Fails, alert goes nowhere |
| Filter matches nothing | Runs cleanly, does nothing |
| Downstream system changes | Sends into a void |
The fourth row is the most insidious. Everything reports success and the work simply is not happening.
Alert on absence and on zero
- Alert if a scheduled run does not happen.
- Alert if a run processes zero records unexpectedly.
- Alert if volume is far outside the normal range.
- Send alerts to a person, not an unwatched mailbox.
- Check the alerting itself works, periodically.
Point five is worth scheduling. Alerting that has itself broken is a common and thoroughly unhelpful discovery during an incident.
Give every automation an owner
An automation nobody owns stops working and stays stopped. Record who owns each one, and reassign when people change roles.
- A named owner, reviewed when people move
- A note of what it does and why it exists
- What depends on it working
- What to do when it fails
- Whether it is still needed at all
The last point is worth revisiting annually. Automations outlive the processes they served, consuming resources and occasionally causing confusion.
Make failure visible to the people affected
If an automation produces something people rely on, they should be able to see when it last ran successfully. Otherwise they trust stale output.
A last-updated timestamp on the output is usually enough and costs almost nothing.