Day 44 of 100
The High-Cardinality Metric Problem
π₯ Problem Statement
Monitoring costs jump sharply, and dashboards that used to load quickly start feeling sluggish. Nothing about traffic volume changed. The one recent change on the observability side: someone added a new label to an existing, frequently-emitted metric, intending to make it easier to filter and investigate by a specific dimension.
That new label happens to carry a unique identifier β something close to one distinct value per user or per request.
ποΈ Environment
- Metrics platform: time-series database, usage-based pricing
- Recent change: a new label added to an existing, high-volume metric
- Symptom: monitoring cost sharply increased, dashboard queries slower
- The new label: a unique user or request identifier
π€ Your Challenge
What is metric cardinality, and why would adding one seemingly small label cause this scale of impact?
- If a metric has labels, how does the number of unique label-value combinations affect how many actual time series get created?
- What's the difference between a label with a handful of possible values and one with effectively unlimited unique values?
- Why would a near-unique-per-request label multiply the number of time series dramatically?
- What's a reasonable way to keep this kind of dimension available for debugging without putting it in a metric label?
Solution Hidden
Think through the problem yourself before looking at the answer.
π‘Solution
Step 1 β Understand the symptoms
A metrics time-series database creates one distinct time series for every unique combination of label values on a metric β not one time series per metric name. A metric with two labels, each having ten possible values, produces up to 100 distinct time series. A metric with a label carrying a unique value per user or per request effectively multiplies the number of time series by the number of distinct users or requests observed β which, for any meaningfully-trafficked service, is a number in the thousands or millions, not the handful of dimensions a metrics system is designed to handle efficiently.
Step 2 β Identify the likely bottleneck
This is a textbook high-cardinality metric problem: the new label wasn't chosen for its cardinality, it was chosen for its usefulness in filtering and debugging β a completely reasonable instinct that happens to interact very badly with how metrics storage actually works under the hood. Every unique user or request ID that appears creates a brand new time series, most of which will only ever have one or a few data points before that identifier never appears again, and the metrics platform still has to allocate storage and indexing overhead for each of them.
Step 3 β Investigation
- Confirm the label's actual cardinality β count the number of distinct values it's taken on since being added. If it's in the thousands or more, that's the direct cause, not a coincidence.
- Check the metrics platform's time-series count before and after this label was added β most platforms expose this directly, and the jump should correlate tightly with when the label was introduced.
- Check whether other labels on high-volume metrics across the system carry similarly high cardinality β if this pattern exists elsewhere, it's worth a broader review, not just fixing this one instance.
- Understand what the label was actually meant to help with β filtering by a specific user during debugging is a legitimate need; the question is whether a metric label is the right tool for that need, or whether something else (logs, traces) serves it better without the cardinality cost.
Step 4 β Recommended action
Remove the high-cardinality label from the metric. For the underlying need β being able to filter or investigate by a specific user or request β logs and traces are generally the right tool, not metrics: they're designed to carry high-cardinality, per-event detail, while metrics are designed for aggregation across many events. A trace or a structured log line can carry the exact user or request ID without multiplying the number of stored time series, because that's not how those systems represent data.
If a coarser version of the same dimension is genuinely useful in a metric β a user tier rather than a user ID, for instance β that's usually a reasonable, low-cardinality alternative that preserves most of the debugging value without the cost.
Step 5 β Engineering lesson
Metrics, logs, and traces exist as separate observability signals partly because they're suited to different cardinality profiles β metrics for low-cardinality aggregation, logs and traces for high-cardinality, per-event detail. A label that would be perfectly reasonable in a log line or a trace attribute can be genuinely damaging as a metric label, not because the dimension itself is a bad idea, but because it's the wrong tool for that specific dimension's cardinality.
π§ Todayβs Takeaway
Not every useful dimension belongs in a metric label.
π Donβt Miss Tomorrowβs Challenge
A new real-world engineering challenge is released every day.
100 Days β 100 Challenges β 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things Iβm learning along the way.

