Skip to content

Day 35 of 100

The Secret Rotation Incident

SecurityCloudAdvanced20–25 min

🔥 Problem Statement

A routine, scheduled secret rotation runs exactly as planned — a database credential gets rotated in the secret store, following standard security policy. Within minutes, the application that depends on it starts failing every database authentication attempt.

Nothing about the rotation itself failed. The new credential is valid. The application is just still using the old one, because it read the credential once at startup and has been holding onto that same value in memory ever since.

🏗️ Environment

  • Secret: database credential, rotated per routine security policy
  • Application: reads credential once at startup, caches it in memory
  • Rotation process: updates the secret store, does not notify running instances
  • Symptom: application starts failing to authenticate immediately after rotation

🤔 Your Challenge

How can secret rotation — a routine security practice — be performed without causing exactly this kind of outage?

  • If an application only reads a secret once at startup, what happens to it when that secret changes elsewhere?
  • What's the difference between the secret store having the new value and the running application actually using it?
  • Would restarting every instance simultaneously introduce its own risk, versus a more gradual approach?
  • What would 'both old and new credentials valid during a transition window' change about this scenario?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

The rotation process updated the secret store correctly — that's confirmed, the new credential is valid. The gap is entirely on the consuming side: an application that reads a secret once at process startup and caches it has no mechanism to notice that the secret store's value changed underneath it. From the running application's point of view, nothing happened; it's still using the value it read hours or days ago, which the database no longer accepts.

Step 2 — Identify the likely bottleneck

This is a design gap between two systems that were never actually coordinated: the rotation process assumes consumers will pick up the new value, and the application was built assuming secrets don't change during its lifetime. Neither side is wrong in isolation — reading a secret once at startup is simple and often reasonable — but the two assumptions directly conflict the moment rotation actually happens against a long-running process.

Step 3 — Investigation

  • Confirm exactly when the application last read the credential versus when rotation occurred — this establishes the mechanism definitively rather than leaving it inferred.
  • Check whether the old credential was immediately invalidated on rotation, or whether there was any overlap window where both old and new were valid — this determines how much time existed to react before this became a hard failure.
  • Check how many other services consume secrets the same way (read-once-at-startup, no refresh mechanism) — if this pattern is common across the environment, every one of them is exposed to the same risk on their next rotation.
  • Check what the rotation process itself currently does or doesn't do to notify or coordinate with consumers — confirming there's no signal sent anywhere is what establishes the actual root cause.

Immediate fix: restart the affected application instances so they read the newly rotated credential — this resolves the outage, but doesn't fix the underlying gap for the next rotation.

The durable fix is treating secret rotation as a coordinated process, not a fire-and-forget update to a secret store:

  • Dual-credential support during rotation: keep the old credential valid for a defined overlap window after the new one is issued, so consumers reading either value continue working until they've all picked up the new one.
  • Live secret refresh: where feasible, have applications watch for secret changes and refresh their cached value without requiring a full restart — many secret management platforms support this natively.
  • Coordinated rolling restart as part of the rotation process itself: if live refresh isn't feasible, make instance restart an explicit step of rotation, not an assumption that it'll happen on its own — and roll it out gradually rather than all at once, so a bad credential doesn't take down every instance simultaneously.

Step 5 — Engineering lesson

Rotating a secret changes state in a store; it doesn't automatically propagate that change to everything depending on the old value. Treating rotation as "update the secret store, done" skips the half of the process that actually matters for a long-running system — getting every consumer safely onto the new value without a gap where nothing is valid, and without every instance restarting at once and risking its own kind of outage.

🧠 Today’s Takeaway

Secret rotation should be designed as a deployment process.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.