Skip to content

Day 29 of 100

The Log That Filled the Disk

LinuxObservabilityIntermediate15–20 min

🔥 Problem Statement

Disk usage on a production host has been climbing fast — not over days, over hours — since the last deploy. That deploy included a configuration change meant to be temporary: a verbose debug logging flag, turned on to help investigate an earlier issue, that never got turned back off before this release shipped.

With no log rotation configured, every log line — and there are now a lot more of them per request than before — just keeps accumulating in the same growing file.

🏗️ Environment

  • Application: recently redeployed with a debug logging flag left enabled
  • Logging: writes to local disk, no rotation configured
  • Centralized logging: not yet in place for this service
  • Symptom: disk usage climbing rapidly since the last deploy

🤔 Your Challenge

How would you identify this as the cause quickly, and what would you actually do about it?

  • What's the fastest way to confirm logging specifically, versus some other cause, is behind the disk growth?
  • Why would a debug logging level cause dramatically more disk usage than a normal logging level?
  • What's the difference between fixing this incident and fixing the fact that it could happen again next release?
  • What role does log rotation play here, and why was its absence part of the problem?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

Disk usage climbing sharply, specifically starting right at the last deploy, is a strong timing correlation worth checking first before looking anywhere else — and log files are one of the most common, fastest-growing culprits when something changes right at deploy time. Debug-level logging is often dramatically more verbose than normal operational logging — every function entry, every intermediate value, every routine step that a production log level would skip — so even moderate traffic can generate an outsized volume of log data per request under debug settings.

Step 2 — Identify the likely bottleneck

Two things compounded here: the debug flag left on massively increased log volume per request, and the complete absence of log rotation meant nothing was capping how large the resulting log file could grow. Either one alone would likely have been manageable — verbose logging with rotation caps the damage; normal logging without rotation grows slowly enough to notice; both together produce exactly this rapid, unbounded growth.

Step 3 — Investigation

Bash
du -sh /var/log/*   # or wherever this service writes logs

Quickly identifies whether a specific log file is the actual disk consumer, and how large it's gotten.

Bash
ls -lh <logfile>

Confirms file size directly and its last-modified time, useful for correlating growth rate against the deploy timestamp.

From there:

  • Check the logging configuration deployed in the last release specifically for the debug flag or verbosity level — confirm it's actually set higher than intended.
  • Estimate growth rate — comparing file size now against an earlier point (or extrapolating from current write rate) gives a concrete sense of how much runway remained before this would have caused a full outage, not just a warning.
  • Confirm no rotation policy exists for this log path — logrotate configuration, or its equivalent, and whether this service was ever included in it.

Immediately revert the debug logging flag to its intended production level and free disk space by rotating or truncating the existing oversized log file. Then close both gaps that allowed this: add log rotation with a sane retention policy so no single log file can grow unbounded again, regardless of verbosity level, and treat "leave a debug flag on" as something that shouldn't be able to ship to production silently — a checklist item, a linter rule, or a config validation step that flags non-default logging levels before a release goes out.

Longer term, moving this service's logs to centralized logging (rather than writing indefinitely to local disk) removes this entire category of failure — log volume becomes the logging platform's problem to manage with its own retention and cost controls, not a local disk capacity risk on every host.

Step 5 — Engineering lesson

Logging is a debugging tool that has its own resource cost, and treating it as free — turn on verbose logging, forget to turn it back off, no cap on how much accumulates — turns the tool meant to help diagnose problems into the cause of an outage instead. A "temporary" debug setting that ships to production without expiring is one of the most avoidable ways a well-intentioned diagnostic step turns into an incident of its own.

🧠 Today’s Takeaway

Logs are useful until they become the outage.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.