Silent failure is the default
A scheduled job that stops running produces no error. The report is not sent, the sync does not happen, and nobody notices until a number looks wrong days later.
Alert on completion, not just failure. A job that never started looks identical to a job with nothing to do.
What every scheduled job needs
- Idempotency — safe to rerun
- Overlap protection — a slow run must not be joined by the next
- A completion record that is monitored
- A time limit, so a stuck job does not run forever
- Logging of what was processed, not just that it ran
Design for restart
- Process in batches with progress recorded
- A restarted job resumes rather than starting again
- One bad record does not stop the run
- Failed records quarantined with the reason
A nightly job that fails at ninety per cent and restarts from nothing will eventually stop completing at all.
Watch the clock
| Issue | Guard |
|---|---|
| Daylight saving | Schedule in UTC |
| Jobs overlapping | Locking |
| Duration growing | Monitor and investigate |
| Several heavy jobs at once | Stagger the schedule |
| Month-end spikes | Test at realistic volume |
Duration growing steadily over months usually means a query scanning a growing table, and it is worth investigating before it exceeds its window.
Make manual runs possible
There will be occasions when a job needs running for a specific date, or rerunning after a fix. Building that in from the start is trivial; adding it under pressure is not.
A command that accepts a date range covers most of these situations.