Day 28 of 100
The Storage Full Incident
🔥 Problem Statement
A production application suddenly stops being able to write files — not crashing outright, but every write operation starts failing. CPU and memory both look completely normal on the dashboard, which is usually where anyone would look first, and both come back clean.
Nobody's checked disk usage, because nothing on the current monitoring setup surfaces it as something worth checking by default.
🏗️ Environment
- Host: Linux VM running a production application
- CPU/memory: both normal per monitoring
- Application: suddenly failing to write output files
- No disk-usage alerting currently configured
🤔 Your Challenge
With CPU and memory both clean, what else would you check to explain writes suddenly failing?
- What system resource, beyond CPU and memory, could cause write failures specifically when exhausted?
- Is 'disk full' the only way storage can run out — could something else about the filesystem be exhausted instead?
- What commands would show you disk space and inode usage directly?
- What's commonly responsible for unexpected disk growth on a long-running host?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
Write failures with normal CPU and memory point toward storage almost by elimination — those are the three resources most commonly monitored, and two of them are cleared. "Disk full" is the obvious first guess, and it's usually right, but it's not the only storage-related exhaustion that produces exactly this symptom: a filesystem can also run out of inodes (the structures that track files) while still showing free space in bytes, if a huge number of small files have accumulated.
Step 2 — Identify the likely bottleneck
On a long-running host with no cleanup automation, the most common causes of unexpected disk growth are logs that were never rotated or capped, temporary files created by the application or its dependencies that are never cleaned up, or an accumulation of old build artifacts, cached data, or core dumps from past incidents that nobody thought to remove. Any of these, left unchecked over enough uptime, will eventually consume all available space or all available inodes, and the failure mode is abrupt — everything's fine until the moment it isn't.
Step 3 — Investigation
df -hChecks overall disk space usage per filesystem — the fastest way to confirm or rule out "disk is full" directly.
df -iChecks inode usage specifically — a filesystem can show plenty of free space in df -h while still being unable to create new files if inodes are exhausted, which is easy to miss if only checking space.
From there:
- Identify what's actually consuming the space —
du -sh /*from root (or targeted at likely directories like/var/log,/tmp) to find the largest consumers. - Check log file sizes specifically — an unrotated log file that's grown unbounded is one of the most common single causes of this exact scenario.
- Check for accumulated temporary files the application creates but doesn't clean up, especially under any recent change in behavior or load.
- Check when this last had headroom if any historical disk-usage data exists, to understand the growth rate and how much runway existed before this became urgent.
Step 4 — Recommended action
Immediately free enough space to restore write capability — remove or archive whatever the investigation identifies as the largest safe-to-remove consumer, most often old logs or temp files. Then address the underlying cause so it doesn't recur: configure log rotation with a retention cap, add cleanup automation for temporary files, and set up centralized logging so logs don't need to accumulate indefinitely on the host itself.
Just as important: add disk space and inode usage to standard monitoring and alerting, with a threshold well before 100% (80-85% is a common starting point) — this incident happened with zero warning specifically because nothing was watching this resource, and a threshold alert would have surfaced it while there was still time to act calmly instead of during an active outage.
Step 5 — Engineering lesson
CPU and memory get monitored by default almost everywhere; disk space and inodes often don't, simply because they change more slowly and fail less often — until a host has been running long enough for slow accumulation to become a hard limit. "Storage is fine" isn't the same claim as "storage is being watched," and this incident is exactly what the gap between those two looks like in practice.
🧠 Today’s Takeaway
Disk capacity and disk health are different signals.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

