Knowing When Something Has Gone Wrong
Last updated:
Silent failures cost the most
An application that is down gets noticed. One where the nightly job stopped, or email stopped sending, can run wrong for weeks.
Ask how you would find out if your application stopped sending email today. For most businesses the honest answer is a customer complaint, eventually.
Four things to watch
- Errors — aggregated, with context to diagnose
- Response times — percentiles, not averages
- Queue depth and failed tasks
- One business metric — records created, orders placed, jobs completed
The fourth catches what the others miss. If yesterday's count is zero and forty is normal, something is wrong regardless of the error log.
Percentiles, not averages
An average response time hides the slow requests. A page averaging four hundred milliseconds with a small proportion taking ten seconds is a page some users find unusable.
Watch the ninety-fifth and ninety-ninth percentile, which is where the user experience actually lives.
Route alerts to someone who can act
| Alert | Goes to |
|---|---|
| Application down | Whoever can restart it |
| Error rate spike | Whoever can diagnose |
| Business metric at zero | The business owner |
| Queue growing | Whoever manages workers |
| Weekly summary | Everyone, for visibility |
Prune the noise
An alert channel firing forty times a day is one people mute. Any alert that has fired repeatedly without anyone acting is either wrongly tuned or unnecessary.
Ten minutes a quarter keeps it meaningful, which is what makes the important alert get read.
Frequently asked questions
What error tracking should we use?
How do we monitor a business metric?
Should we alert on every error?
Do we need an on-call rota?
Would you know if your application stopped working properly?
Not stopped — stopped working properly. That is the failure worth monitoring for.
Related services
What we build for problems like this one