Skip to content

Day 32 of 100

The Broken CI Pipeline

DevOpsCI/CDIntermediate15–20 min

🔥 Problem Statement

A pipeline that's been passing reliably for months suddenly starts failing — same repository, same branch protection rules, and critically, no changes to the application code between the last green run and the first red one. Whatever changed, it wasn't anything a developer wrote.

🏗️ Environment

  • CI: GitHub Actions, previously reliable pipeline
  • Application code: unchanged since the last successful run
  • Pipeline: uses a third-party dependency and a specific runner image
  • Symptom: pipeline started failing without any code change triggering it

🤔 Your Challenge

If the application code didn't change, what else could plausibly cause a previously reliable pipeline to start failing?

  • What parts of a CI pipeline exist outside the application repository itself?
  • Could a dependency, a base image, or a runner environment change without any commit to this repo reflecting it?
  • What's the fastest way to see exactly what step failed and why, rather than guessing?
  • Why would 'it worked yesterday, nothing changed' still be consistent with an actual external change?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

"Nothing changed in the code" is true and also not the whole picture — a CI pipeline depends on far more than the application repository: the runner image it executes on, the versions of tools and dependencies it pulls at run time, any third-party service or package registry it talks to, and the CI platform's own runner environment, all of which can change independently of anything committed to this repo. "It worked yesterday" and "nothing changed" can both be true while something outside the repository's control changed anyway.

Step 2 — Identify the likely bottleneck

The most common versions of this: a dependency (a package, an action, a base image) that isn't pinned to an exact version quietly received an update that introduced a breaking change; the CI platform's default runner image got updated with a new tool version that behaves differently; a secret or credential the pipeline relies on expired or was rotated; or an external service the pipeline calls (a registry, an API) had its own outage or breaking change. None of these require a single line of application code to change.

Step 3 — Investigation

  • Read the actual failure output carefully, not just "the pipeline failed" — the specific step, command, and error message almost always point directly at what actually broke, whether that's a version mismatch, an authentication failure, or a missing dependency.
  • Check whether any dependencies used by the pipeline (not the application) are unpinned — actions referenced by a moving tag like @main or @latest, or packages without an exact version pin, are the single most common cause of "nothing changed but it broke anyway."
  • Compare the runner image or environment version between the last successful run and the first failed one — many CI platforms show this, and a base image update landing between runs is a frequent, silent cause.
  • Check whether any credentials or tokens used by the pipeline are near or past expiry — a rotated or expired secret produces a very specific, usually clear authentication error in the logs.
  • Check the status of any third-party service the pipeline depends on (package registry, external API) for a concurrent outage or reported incident.

Once the log output and version comparison point at a specific cause, the fix usually follows directly: pin the offending dependency, action, or base image to an exact, known-good version instead of a moving reference; rotate or update an expired credential; or wait out (and add resilience against) a third-party outage if that's the cause.

Beyond this specific fix, pin every external dependency this pipeline relies on to exact versions — this is the single most effective way to prevent "nothing in our code changed but CI broke anyway" from recurring, since it removes the pipeline's exposure to unannounced upstream changes entirely.

Step 5 — Engineering lesson

A CI pipeline is itself a system with its own dependency graph — runner images, third-party actions, package versions, external services — and every unpinned reference in that graph is a place where something can change without a commit to explain it. "The code didn't change" rules out one category of cause and should immediately redirect attention to everything else the pipeline depends on, not to re-reading application code that was never the problem.

🧠 Today’s Takeaway

A pipeline is a system with dependencies too.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.