Silent failures cost the most
An application that is down gets noticed. One where the nightly job stopped, or email stopped sending, can run wrong for weeks.
Ask how you would find out if your application stopped sending email today. For most businesses the honest answer is a customer complaint, eventually.
Four things to watch
- Errors — aggregated, with context to diagnose
- Response times — percentiles, not averages
- Queue depth and failed tasks
- One business metric — records created, orders placed, jobs completed
The fourth catches what the others miss. If yesterday's count is zero and forty is normal, something is wrong regardless of the error log.
Percentiles, not averages
An average response time hides the slow requests. A page averaging four hundred milliseconds with a small proportion taking ten seconds is a page some users find unusable.
Watch the ninety-fifth and ninety-ninth percentile, which is where the user experience actually lives.
Route alerts to someone who can act
| Alert | Goes to |
|---|---|
| Application down | Whoever can restart it |
| Error rate spike | Whoever can diagnose |
| Business metric at zero | The business owner |
| Queue growing | Whoever manages workers |
| Weekly summary | Everyone, for visibility |
Prune the noise
An alert channel firing forty times a day is one people mute. Any alert that has fired repeatedly without anyone acting is either wrongly tuned or unnecessary.
Ten minutes a quarter keeps it meaningful, which is what makes the important alert get read.