Skip to content

Day 49 of 100

The Terraform State Lock

TerraformAdvanced20–25 min

🔥 Problem Statement

An engineer runs `terraform plan` against the production workspace and immediately hits a state lock error — someone else already has it locked. Rather than wait, they consider just force-unlocking it and proceeding with their own run. The lock is, after all, just getting in the way of what they need to do right now.

🏗️ Environment

  • Terraform: remote backend with state locking (e.g. S3 + DynamoDB)
  • Workspace: shared production workspace
  • Two engineers: both attempting a run against the same workspace
  • Symptom: second engineer's run is blocked by an active state lock

🤔 Your Challenge

Should the engineer force-unlock the state and proceed? What should they check first?

  • What is a state lock actually protecting against?
  • What happens if two Terraform runs modify the same state concurrently without a lock?
  • How would you find out who currently holds the lock and whether their run is still active?
  • What's the safest sequence of actions here, rather than the fastest one?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

A state lock exists specifically to prevent two concurrent Terraform runs from modifying the same state file at the same time — without it, two simultaneous applies could interleave their writes, corrupting the state file or causing one run's changes to be silently lost or overwritten by the other's. The lock being active right now means, most likely, someone else's run is genuinely in progress — the system is doing exactly what it's designed to do.

Step 2 — Identify the likely bottleneck

There isn't a bug here — there's a temptation. Force-unlocking is available specifically for the case where a lock is genuinely stale (a previous run crashed or was killed without releasing it cleanly), not as a general-purpose way to skip a wait. Force-unlocking while another run is actually still active is exactly the scenario the lock exists to prevent, and doing it anyway reintroduces the concurrent-write risk the lock was put there to close off.

Step 3 — Investigation

Before touching the lock at all:

  • Check the lock's metadata — most backends record who acquired it and when (terraform plan and apply failures typically show this in the error output directly). This alone often answers whether it's a colleague's active run or a stale leftover.
  • Reach out directly to whoever holds the lock, if identifiable — a quick message confirming whether their run is still in progress is faster and safer than guessing from the lock's age alone.
  • Check how long the lock has been held relative to how long a typical run for this workspace takes — a lock held for a duration far exceeding a normal run's length is a much stronger (though still not definitive) signal that it may be stale.
  • Check CI/CD run history if applies happen through a pipeline — a corresponding run still showing as "in progress" (or having failed without cleanup) directly confirms whether this is active or abandoned.

If the lock owner is identified and confirms their run is still active, wait — that's the lock functioning correctly, protecting exactly the scenario it exists for. If investigation confirms the lock is genuinely stale (the owning run crashed, was killed, or CI shows it failed without releasing the lock), force-unlock deliberately, with that confirmation in hand, not as a default reaction to being blocked.

Either way, this is worth a brief note to the team if it happens on a shared production workspace — even a stale-lock cleanup is worth being visible about, so nobody else is confused by state changing unexpectedly.

Step 5 — Engineering lesson

A blocking lock is inconvenient by design — that inconvenience is the entire point, trading a short wait for protection against a state file corruption that could take far longer to recover from and diagnose. The instinct to force past a lock because it's in the way of what you need to do right now is exactly the instinct the lock exists to check; the right response to being blocked is finding out why, not clearing the obstruction and hoping nothing was actually in progress.

🧠 Today’s Takeaway

Never remove a lock before understanding who holds it and why.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.