Silent failure is the norm
A scheduled task that stops running produces no error anywhere. The report is not sent, the sync does not happen, and nobody notices until someone asks why a number looks wrong.
Alert on completion, not just on failure. A job that never started looks identical to a job with nothing to do, and only a completion alert distinguishes them.
What every scheduled task needs
- Idempotency — running twice must not double the effect
- Overlap protection — a slow run must not be joined by the next one
- Logging of start, finish and what was processed
- A completion signal that is monitored
- A time limit, so a stuck job does not run forever
Use the framework's scheduler
Modern frameworks provide a scheduler defined in code, driven by a single system cron entry. That keeps the schedule in version control and visible to developers.
A dozen separate crontab entries edited on the server is the alternative, and it is invariably out of date and undocumented.
Design for restartability
- Process in batches with progress recorded
- A restarted job should resume, not restart from the beginning
- Failures on one record should not stop the run
- Failed records quarantined with the reason
A nightly job that fails at ninety per cent and has to start again from nothing will eventually fail to complete at all.
Watch the clock
| Issue | Guard |
|---|---|
| Daylight saving changes | Schedule in UTC |
| Jobs overlapping | Locking |
| Jobs running longer over time | Duration monitoring |
| Several heavy jobs at once | Stagger the schedule |
| Month-end volume spikes | Test at realistic volume |