Tuesday's data, on Friday
The stock import runs at 2am every night. Except on Tuesday it did not, and nobody noticed until Friday, when a customer ordered something that had sold out days ago. Or the nightly database backup has been failing since a password change last month. Or the queue worker that sends invoices crashed over the weekend, and a pile of invoices is still sitting in it.
Each time, someone logs in to the server, reruns the job by hand and hopes it catches up properly. Nobody can say which other jobs might be failing in the same quiet way right now.
Why scheduled jobs fail without a sound
A scheduled job has no user watching it. When it fails, there is no error page and no angry customer at the time, only a gap that shows up later.
| Cause | What happens |
|---|---|
| Output goes nowhere | Cron or Task Scheduler runs the job and its errors are discarded or emailed to an unread local mailbox |
| The job never starts | A server restart, time zone change or edited schedule stops it, and nothing is expecting it |
| Credentials changed | A rotated password or expired token makes every run fail at the first step |
| Worker process died | A queue worker crashes and nothing restarts it, so jobs pile up unprocessed |
| Overlapping runs | A slow run is still going when the next starts, and they collide |
| Half-finished runs | The job stops partway, leaving data partly updated |
The key point is the second row. Most monitoring looks for errors. A job that never started produces no error at all, so the only way to catch it is to expect a run and notice when it does not arrive.
What silent failures cost
- Stock, prices or orders out of step between systems, leading to overselling or wrong quotes.
- Backups that stopped weeks ago, discovered on the day you need one.
- Invoices, reminders or reports that were never sent.
- Staff making decisions from yesterday's or last week's numbers without knowing it.
- Hours spent working out what was missed and rerunning jobs in the right order.
The cleanup is often the expensive part. When a job has missed several runs, working out which data is stale and catching up safely is slower than the job itself.
How we make every run visible
- Find every scheduled job. We list crontabs, systemd timers, Windows Task Scheduler entries, queue workers, framework schedulers and cloud schedulers on each server, with what each job does and who relies on it.
- Give each job an owner and an expected schedule: when it should run and roughly how long it should take.
- Record every run. Each job reports its start, finish, result and key counts (rows imported, files backed up) to a central log rather than a file on the server.
- Alert on missing runs as well as failures, using heartbeat monitoring through tools such as Healthchecks.io, Cronitor, Better Stack or your cloud provider's monitoring. If a run does not check in on time, a named person hears about it.
- Supervise workers. Queue workers run under a process supervisor that restarts them, and alerts fire when a queue grows beyond normal.
- Make jobs safe to rerun, with locks that prevent overlapping runs and logic that can pick up where a failed run stopped, so recovery is just running the job again.
- Write a short note per job: what it does, how to check it ran, and how to rerun it.
This is the server side of the problem. If your scheduled work lives in no-code tools like Zapier or Make, the same principles apply but the tools are different, and we look at those too.
Mornings after the change
If a job fails or does not run, someone knows that morning, not days later. There is a single place to see what ran overnight and what it did. Queue workers restart themselves and warn when they fall behind. And when a rerun is needed, it is a known, safe step rather than an anxious manual repair.
Does this happen to you?
- You have found out days later that an import, sync or backup did not run.
- Nobody has a full list of what is scheduled on your servers.
- Job errors go to a log file or mailbox nobody reads.
- Queue workers sometimes stop and have to be restarted by hand.
- Rerunning a failed job is risky because it might duplicate data.