No space left on device
The site goes down without warning. Pages return errors, or the database refuses to accept new orders, or uploads fail with a message nobody on the office side understands. A developer logs in, finds the disk at full capacity, deletes a pile of old files to make some room and restarts things. Everything works again.
A few months later it happens again, and the same developer does the same clean-up. Nobody has worked out why the disk fills up in the first place, so each time it is treated as bad luck.
What quietly eats the space
Disks rarely fill because of one big file. They fill because several things grow a little every day and nothing ever removes the old parts.
| Source | Why it grows |
|---|---|
| Application and web server logs | Written every request, never rotated or deleted |
| Debug logging left on | A setting used while fixing something once, still writing detail on every call |
| Uploads and generated files | Invoices, exports, images and reports saved to the server and never archived |
| Local backups | Nightly database dumps stored on the same disk, kept forever |
| Database growth | Session, audit and log tables that nobody trims |
| Temporary and cache files | Left behind by failed jobs or old software versions |
| Container images | Old Docker images and volumes piling up after each deployment |
The local backups row causes double trouble. It fills the disk, and a backup on the same disk as the data does not protect you if that disk fails.
What a full disk costs
When a disk fills, it rarely fails politely. Databases can stop accepting writes or, worse, corrupt a table partway through. Queues stop. Emails that should have gone out do not. Logs stop being written at the exact moment you need them to understand what went wrong.
There is also the cost of the clean-up itself. Deleting files in a hurry on a production server is risky. It is easy to remove something that mattered, like the only copy of an export a customer needs or a file the application still expects to find.
And because it recurs, it costs the same person the same evening every few months, which is time that could be spent fixing the cause.
How we stop the disk filling up
- Find what is using the space. We measure disk usage by directory, by log file, by database table and by container, and track how quickly each one grows.
- Set log rotation and retention for every log source, using the operating system's tools (logrotate on Linux, or the equivalent on Windows) and the application's own settings, with debug logging turned off in production.
- Send logs somewhere built for them. Where logs need to be kept and searched, we ship them to a central service such as AWS CloudWatch, Azure Monitor or a log platform, with a retention period you agree, instead of keeping them on the web server.
- Move growing files to object storage. Uploads, exports and generated documents go to Amazon S3, Azure Blob Storage or similar, with lifecycle rules that archive or delete old items.
- Move backups off the server to separate storage, with their own retention rules, so they neither fill the disk nor share its fate.
- Trim database tables that only grow, such as sessions and old audit entries, on a schedule, archiving anything that must be kept.
- Clean up after deployments, removing old releases, container images and temporary files automatically.
- Alert early. Disk usage and growth rate are monitored, and a named person gets an alert well before the disk is full, not when it already is.
Where the server simply needs more space for good reasons, we say so and resize it, but only after the waste is gone, so the extra space is not just filled by the same logs.
What changes afterwards
Disk usage stays level instead of climbing. Logs are kept for as long as they are useful, in a place where they can be searched. Files and backups live in storage designed for them. And if something does start growing unexpectedly, someone hears about it while there is still plenty of room to deal with it calmly.
Is your server doing this?
- Your site or database has stopped because the disk was full.
- Someone periodically logs in and deletes old files to free space.
- Log files on the server are very large or date back years.
- Backups are stored on the same server as the live data.
- Nobody gets an alert when disk space runs low.