Skip to content

Day 23 of 100

The Deployment With No Rollback Plan

DevOpsRelease EngineeringIntermediate15–20 min

🔥 Problem Statement

A major release is planned for production — new functionality, a database schema change, the kind of deployment that touches enough of the system that it's clearly not routine. There's a deployment plan: steps, order of operations, who's doing what. It's thorough.

There's no rollback plan. Nobody's written down what happens if something goes wrong after this ships — not because anyone decided it wasn't needed, but because the deployment plan itself just never got a "what if this fails" section.

🏗️ Environment

  • Change: major production release, includes a database schema change
  • Deployment plan: documented, reviewed, scheduled
  • Rollback plan: does not exist
  • Stakeholders: product team eager to ship, no explicit sign-off on risk

🤔 Your Challenge

What should exist before a production deployment like this ships, beyond the deployment plan itself?

  • If this deployment fails halfway through, what state is the system actually in — and does anyone currently know?
  • Why does a database schema change make rollback specifically harder than a stateless code deploy?
  • What's the difference between 'we can roll back the code' and 'we can actually recover from this specific failure'?
  • Who needs to explicitly approve shipping without a rollback plan, if that's the decision being made?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

A deployment plan describes how to move forward. A rollback plan describes how to move backward if forward goes wrong — and those are genuinely different documents requiring different thinking, not two halves of the same checklist. The absence here isn't a gap in execution, it's a gap in planning: nobody asked "what does recovery look like" before asking "how do we ship this," which means if something does go wrong mid-deployment, the response will be improvised under pressure instead of prepared in advance.

Step 2 — Identify the likely bottleneck

The database schema change is what makes this specifically risky, more than a typical stateless deploy would be. Code can usually be rolled back cleanly — redeploy the previous version, done. A schema change is often one-directional in practice: if the new code has already written data in a new shape, or a migration has already run, "rolling back" isn't just redeploying old code — old code may not even work correctly against a changed schema, and reversing the schema change itself might mean data loss or a much more complex recovery than anyone planned for.

Step 3 — Investigation

Before deployment, not after, this is worth working through explicitly:

  • What does the schema change actually do — is it additive (new column, new table, backward-compatible) or does it modify/remove something the old code depends on? Additive changes are far easier to roll back from than destructive ones.
  • Can the old code run against the new schema, even temporarily? If yes, a code-only rollback is viable even with the schema change in place. If no, rollback requires a schema reversal too, which is a much bigger operation.
  • Is there a backup or point-in-time recovery option for the data, in case a schema change needs to be reversed after real production data has already been written under the new shape?
  • What's the actual failure mode being planned for — a deploy that fails immediately (easy, don't proceed), versus one that succeeds but causes problems hours later, once real traffic and real data have flowed through the new code path (much harder to cleanly undo)?

Don't ship until there's an explicit answer to "if this goes wrong, what do we do" — even if the honest answer for a specific irreversible step is "we can't roll back this part, only forward-fix it," that's a decision worth someone consciously approving, not an unplanned discovery mid-incident. Where possible, design the schema change to be backward-compatible (the expand/contract pattern: add new structure first, migrate data, only remove the old structure in a later, separate deploy once the new path is proven) — this keeps rollback genuinely possible at every step instead of only in theory.

At minimum, before shipping: confirm backups are current and tested, confirm whether old code can tolerate the new schema, and get explicit sign-off from whoever owns the risk decision if a true rollback isn't actually available for any part of this change.

Step 5 — Engineering lesson

A deployment plan that only covers the forward path is half a plan. The moment something goes wrong in production — and eventually, something will — the question isn't "did we plan how to ship this," it's "did we plan how to recover if shipping it goes badly." For a change that includes something as hard to reverse as a schema migration, that second question deserves as much attention as the first, not an afterthought squeezed in if there's time.

🧠 Today’s Takeaway

A production deployment isn't complete until rollback is understood.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.