Skip to content

Day 42 of 100

The Missing Trace

OpenTelemetryAdvanced20–25 min

🔥 Problem Statement

Every service in the architecture has metrics. Every service has logs. Tracing is instrumented broadly using OpenTelemetry — except one service, where traces simply never show up in the tracing backend. Not partial traces, not broken traces — nothing at all, for this one specific service, while metrics and logs from that same service look completely normal.

🏗️ Environment

  • Observability: metrics and logs fully available for all services
  • Tracing: OpenTelemetry SDK instrumented across most services
  • One specific service: no traces appearing in the tracing backend
  • No errors reported by the instrumentation itself

🤔 Your Challenge

Given that tracing is a multi-stage pipeline, where specifically would you look for a service producing zero traces?

  • What are the distinct stages a trace has to pass through between being generated in code and appearing in a backend?
  • If metrics and logs work fine for this service, does that rule out the SDK entirely, or just parts of it?
  • Could sampling configuration cause a service to appear to produce zero traces even if instrumentation is working?
  • What would you check first: the application's instrumentation, or the collector's configuration?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

A trace's path from creation to visibility passes through several independent stages: the application's SDK instrumentation generates spans, an exporter sends them (often to a local collector), the collector processes and forwards them, and a backend ingests and stores them for querying. Metrics and logs working fine for this service confirms the application itself and its general observability setup are functioning — but tracing is a genuinely separate pipeline with its own configuration at every one of those stages, and a failure anywhere along it produces exactly this symptom: total silence, not partial or degraded data.

Step 2 — Identify the likely bottleneck

Because metrics and logs are fine, this narrows meaningfully toward tracing-specific configuration rather than a broad instrumentation or infrastructure problem: the OpenTelemetry SDK might not actually be initialized for tracing in this service (even if it's correctly initialized for metrics), the exporter endpoint might be misconfigured or unreachable specifically for traces, sampling configuration could be set to effectively zero for this service, or the collector might be dropping or misrouting traces from this specific source while correctly handling its metrics and logs.

Step 3 — Investigation

  • Check the SDK initialization code for this service specifically — is a tracer provider actually configured and registered, separate from whatever's configured for metrics and logs? It's common for a service to have metrics fully wired up while tracing was never actually completed.
  • Check the exporter configuration — is it pointed at the correct collector endpoint, and is that endpoint actually reachable from this service? A silent export failure (network issue, wrong endpoint) often produces zero visible errors from the application's own point of view.
  • Check sampling configuration — a sampling rate effectively at zero for this service (intentional or misconfigured) would produce genuinely zero traces while the SDK itself functions correctly.
  • Check the collector's logs and configuration for anything related to this service — a routing rule, a processor, or a filter that's specifically excluding or dropping data from this source.
  • Check for propagation issues at the entry point — if this service is never receiving an initial trace context to continue, and isn't configured to start new traces on its own, it may simply never be generating spans in the first place.

Once the specific stage is identified, the fix is usually narrow and direct: complete tracing initialization if it was simply never finished for this service, correct the exporter endpoint, adjust sampling configuration, or fix the collector-side routing or filtering rule. Confirm the fix by checking for a test trace actually arriving in the backend, not just by confirming configuration looks correct on paper.

Step 5 — Engineering lesson

Observability isn't a single switch that's either on or off — metrics, logs, and traces are three separate pipelines, each with their own instrumentation, transport, and backend configuration, even when they share an underlying SDK. A service can be fully observable on two of the three and completely dark on the third, and confirming that requires checking each pipeline's actual data flow end to end, not just confirming the SDK is present in the codebase.

🧠 Today’s Takeaway

Observability is a pipeline, not a checkbox.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.