Silent failures cost the most
An application that is down gets noticed within the hour. One where the nightly sync stopped, or the email stopped sending, can run wrong for weeks.
Ask how you would find out if your application stopped sending email today. For most businesses the honest answer is a customer complaint, eventually.
Four things to watch
- Errors — aggregated, deduplicated, with the context to diagnose
- Uptime — checked frequently, alerting a person
- Queue depth and job failures, if you have background work
- One business metric — orders, submissions, records processed
The fourth is the one that catches silent failures. If the normal daily count is forty and today it is zero, something is wrong regardless of what the error log says.
Alert on absence
- Zero activity where activity is normal
- A scheduled job that did not report completion
- A queue that has stopped draining
- An integration that has sent nothing today
Route alerts to someone who can act
| Alert | Goes to |
|---|---|
| Application down | Whoever can restart it, immediately |
| Error rate spike | Whoever can diagnose it |
| Business metric at zero | The business owner of the process |
| Certificate expiring | Whoever manages hosting |
| Weekly summary | Everyone, so the system stays visible |
Prune the noise
An alert channel that fires forty times a day is a channel people mute. Any alert that has fired repeatedly without anyone acting is either wrongly tuned or unnecessary.
Ten minutes a quarter reviewing what fired and what was acted on keeps the alerts meaningful.