Day 60 of 100
The Production Change Without an Owner
🔥 Problem Statement
A production configuration value is different than it was last week — confirmed, not disputed. What's missing is an answer to who changed it, when, or why. It's not in the recent deployment history through the normal pipeline. Nobody on the team remembers making it. There's a documented change-management process, but it's inconsistently followed, and this looks like exactly the kind of change that fell outside it.
🏗️ Environment
- Production configuration: recently changed
- Change management: exists on paper, inconsistently followed
- IaC and CI/CD: primary deployment path, but console access still exists
- Symptom: nobody can immediately identify who or what made the change
🤔 Your Challenge
How would you trace this change back to its actual source?
- What sources of evidence exist for a change that didn't go through the normal deployment pipeline?
- Would cloud provider audit logs show a direct console change even if it never touched CI/CD?
- Could this have been an automated process rather than a person — and how would you check?
- What would make a future change like this immediately traceable instead of a mystery?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
A change not appearing in the normal deployment pipeline's history doesn't mean it has no record anywhere — it means the record isn't in the first place anyone would think to look. Cloud providers typically log console-level and API-level changes independently of whatever deployment pipeline exists, specifically for cases like this; Git history covers code-driven changes, but a direct console change or an out-of-band script would bypass that entirely while still leaving a trail elsewhere.
Step 2 — Identify the likely bottleneck
Given change management is documented but inconsistently followed, and console access to production still exists alongside the IaC/CI/CD path, this is a strong candidate for exactly the failure mode that process gap allows: someone made a direct, well-intentioned change outside the normal pipeline — an urgent fix, a quick adjustment — without going through the process that would have left a clear, expected record. It's also worth not assuming a human is necessarily responsible; an automation, a scheduled job, or a misconfigured tool with write access could just as easily be the source.
Step 3 — Investigation
- Check cloud provider audit logs (CloudTrail, Cloud Audit Logs, or equivalent) for the specific resource and time window — this is usually the most direct, complete source for exactly this question, since it records changes regardless of how they were made.
- Check for any automation or scheduled process with write access to this configuration — a cron job, a reconciliation script, an integration — that could have made the change without a human directly involved.
- Check IAM activity logs for who authenticated and took action around the relevant timestamp, cross-referenced against the audit log entry for the specific change.
- Ask directly, without assuming blame — sometimes the fastest path is simply asking the team who was working on anything related to this configuration recently; a quick, well-intentioned fix is often exactly what happened, made by someone who didn't think of it as a "change" requiring the full process.
Step 4 — Recommended action
Once identified via audit logs, document what happened and why — not primarily to assign blame, but to understand whether the underlying need (whatever prompted the change) is legitimate and should be reflected properly in code/pipeline, or whether it should be reverted. Use this as the concrete case for tightening what direct production access allows: restrict console-level write access to production resources where the pipeline is meant to be the sole path, or at minimum ensure every path (pipeline and direct) is equally well-logged and equally reviewed after the fact.
Step 5 — Engineering lesson
"Nobody knows who changed this" is a process and auditability gap, not an unsolvable mystery — the change happened somewhere, through some identity or automation, and cloud-provider-level audit logging almost always captures it even when a deployment pipeline's own history doesn't. The deeper fix isn't just answering this one question; it's making sure every path capable of changing production — pipeline or otherwise — is equally traceable, so the next unexplained change doesn't require a full investigation just to find out who made it.
🧠 Today’s Takeaway
Every production change should be traceable to a person, process or automation.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

