Day 34 of 100
The Vulnerability in Production
🔥 Problem Statement
A newly enabled image scanner flags something serious on its first real run: a critical CVE, already sitting in the base image of a service that's currently running in production and facing the internet. This isn't a finding in a pre-deploy check that caught something before it shipped — it's a live production workload that's already vulnerable, discovered after the fact.
🏗️ Environment
- Image scanning: recently enabled, first real finding
- Finding: critical CVE in a base image dependency, already in production
- Service: internet-facing, moderate traffic
- No emergency patch process currently defined
🤔 Your Challenge
What should actually happen first — and what should the team resist doing first?
- Is 'critical' severity by itself enough to determine urgency, or does exposure and exploitability matter too?
- What's the difference between patching immediately and understanding the vulnerability first?
- What would you want to know about whether this CVE is actively being exploited in the wild?
- How would you verify a patch actually resolved this specific vulnerability, not just that a new image built successfully?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
A "critical" severity rating on its own, straight from the scanner, is a starting point for triage, not a complete risk assessment by itself. Two findings both rated critical can have very different real-world urgency depending on whether the vulnerable code path is actually reachable from this service's exposed surface, whether exploitation requires conditions this deployment doesn't meet, and whether it's being actively exploited in the wild right now versus theoretical. None of that comes for free from the severity label alone.
Step 2 — Identify the likely bottleneck
The pressure in a moment like this is to patch immediately, as fast as possible, given the word "critical" and the fact that it's internet-facing. That instinct isn't wrong exactly, but acting before understanding creates its own risk: a rushed patch, applied under panic with no clear rollback plan, to a service currently serving real traffic, can turn a vulnerability (which hasn't (yet) caused an incident) into an actual outage (which definitely has) if the patch introduces a regression or breaks compatibility.
Step 3 — Investigation
- Confirm exploitability for this specific deployment — is the vulnerable code path actually reachable given how this service uses the affected dependency? A CVE in an unused feature of a library is a very different urgency than one in a code path this service exercises directly.
- Check exposure — internet-facing and moderate traffic already establishes real exposure, but confirm what specifically is reachable: is the vulnerable component directly network-accessible, or several layers removed?
- Check whether this CVE is known to be actively exploited — public vulnerability databases and security advisories often note active exploitation, which should significantly raise urgency versus a theoretical, unexploited CVE.
- Check what patch or mitigation is actually available — a direct version bump, a configuration workaround, or (if no fix exists yet) a compensating control like a WAF rule or network restriction.
- Check whether a patched version has been tested against this service's actual usage, not just confirmed to build successfully — a successful build doesn't confirm the vulnerability is actually resolved in the deployed configuration, or that nothing broke in the process.
Step 4 — Recommended action
Assess severity and exploitability first — quickly, but deliberately — to determine real urgency rather than acting purely on the word "critical." If the vulnerability is confirmed actively exploited and reachable, treat this as an emergency patch with appropriately fast turnaround, but still through a real (even if compressed) process: build the patched image, verify the fix actually addresses the specific CVE, test before rolling out, and have a rollback plan ready in case the patch itself causes problems.
If exploitability turns out to be low (unreachable code path, no known active exploitation), patch on a fast but not panicked timeline, prioritized appropriately against other work — still urgent, not necessarily a 2 a.m. emergency deploy.
Either way, this incident is also the moment to establish an actual emergency patch process if one doesn't exist — a defined, rehearsed path for exactly this scenario, so the next critical finding doesn't require inventing a process under pressure.
Step 5 — Engineering lesson
A vulnerability scanner produces a finding; it doesn't produce a risk assessment. Severity, exploitability, and exposure are three different questions, and conflating "critical" with "must patch in the next ten minutes, however it's done" is how a legitimate security issue turns into a self-inflicted outage. Move fast on real risk — but understand the risk first.
🧠 Today’s Takeaway
A vulnerability is a risk to assess and act on—not just a scanner finding.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

