Skip to content

Day 44 of 100

The High-Cardinality Metric Problem

ObservabilityAdvanced20–25 min

πŸ”₯ Problem Statement

Monitoring costs jump sharply, and dashboards that used to load quickly start feeling sluggish. Nothing about traffic volume changed. The one recent change on the observability side: someone added a new label to an existing, frequently-emitted metric, intending to make it easier to filter and investigate by a specific dimension.

That new label happens to carry a unique identifier β€” something close to one distinct value per user or per request.

πŸ—οΈ Environment

  • Metrics platform: time-series database, usage-based pricing
  • Recent change: a new label added to an existing, high-volume metric
  • Symptom: monitoring cost sharply increased, dashboard queries slower
  • The new label: a unique user or request identifier

πŸ€” Your Challenge

What is metric cardinality, and why would adding one seemingly small label cause this scale of impact?

  • If a metric has labels, how does the number of unique label-value combinations affect how many actual time series get created?
  • What's the difference between a label with a handful of possible values and one with effectively unlimited unique values?
  • Why would a near-unique-per-request label multiply the number of time series dramatically?
  • What's a reasonable way to keep this kind of dimension available for debugging without putting it in a metric label?

Solution Hidden

Think through the problem yourself before looking at the answer.

πŸ’‘Solution

Step 1 β€” Understand the symptoms

A metrics time-series database creates one distinct time series for every unique combination of label values on a metric β€” not one time series per metric name. A metric with two labels, each having ten possible values, produces up to 100 distinct time series. A metric with a label carrying a unique value per user or per request effectively multiplies the number of time series by the number of distinct users or requests observed β€” which, for any meaningfully-trafficked service, is a number in the thousands or millions, not the handful of dimensions a metrics system is designed to handle efficiently.

Step 2 β€” Identify the likely bottleneck

This is a textbook high-cardinality metric problem: the new label wasn't chosen for its cardinality, it was chosen for its usefulness in filtering and debugging β€” a completely reasonable instinct that happens to interact very badly with how metrics storage actually works under the hood. Every unique user or request ID that appears creates a brand new time series, most of which will only ever have one or a few data points before that identifier never appears again, and the metrics platform still has to allocate storage and indexing overhead for each of them.

Step 3 β€” Investigation

  • Confirm the label's actual cardinality β€” count the number of distinct values it's taken on since being added. If it's in the thousands or more, that's the direct cause, not a coincidence.
  • Check the metrics platform's time-series count before and after this label was added β€” most platforms expose this directly, and the jump should correlate tightly with when the label was introduced.
  • Check whether other labels on high-volume metrics across the system carry similarly high cardinality β€” if this pattern exists elsewhere, it's worth a broader review, not just fixing this one instance.
  • Understand what the label was actually meant to help with β€” filtering by a specific user during debugging is a legitimate need; the question is whether a metric label is the right tool for that need, or whether something else (logs, traces) serves it better without the cardinality cost.

Remove the high-cardinality label from the metric. For the underlying need β€” being able to filter or investigate by a specific user or request β€” logs and traces are generally the right tool, not metrics: they're designed to carry high-cardinality, per-event detail, while metrics are designed for aggregation across many events. A trace or a structured log line can carry the exact user or request ID without multiplying the number of stored time series, because that's not how those systems represent data.

If a coarser version of the same dimension is genuinely useful in a metric β€” a user tier rather than a user ID, for instance β€” that's usually a reasonable, low-cardinality alternative that preserves most of the debugging value without the cost.

Step 5 β€” Engineering lesson

Metrics, logs, and traces exist as separate observability signals partly because they're suited to different cardinality profiles β€” metrics for low-cardinality aggregation, logs and traces for high-cardinality, per-event detail. A label that would be perfectly reasonable in a log line or a trace attribute can be genuinely damaging as a metric label, not because the dimension itself is a bad idea, but because it's the wrong tool for that specific dimension's cardinality.

🧠 Today’s Takeaway

Not every useful dimension belongs in a metric label.

πŸ”” Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days β†’ 100 Challenges β†’ 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.