Telling Someone When Something Needs Attention

Automation without alerting is a liability

An automated process that runs unattended and fails silently is worse than a manual one, because at least a person notices when they have not done something.

The purpose of alerting is not to record that something failed. It is to make sure somebody who can act finds out while it still matters.

Alert on absence first

• The job did not run
• The file did not arrive
• Zero records processed where there are normally hundreds
• The queue has stopped draining
• The integration has sent nothing today

Every one of those looks like success to a system that only alerts on errors, and every one is a genuine failure.

Three tiers of urgency

Most alerts belong in the middle tier and get built as the first, which is how teams learn to ignore an alert channel.

Name the next action

“Import failed” is a fact. “Import failed — check whether the supplier changed the file format, then reprocess from the quarantine folder” is an alert somebody can act on.

Putting the first steps of the runbook into the message means the person receiving it does not have to find the documentation at seven in the morning.

Prune quarterly

Any alert that has fired repeatedly without anyone acting is either wrongly tuned or unnecessary. Fix it or remove it.

Ten minutes a quarter keeps the channel meaningful, which is what makes the important alert get read.

Frequently asked questions