Logs, metrics and traces: three views of the same failing request Basic

The problem

A customer reports “the checkout page sometimes spins forever”. A request failed log tells you there is an error but not how many. The p95 dashboard rises but does not point to the slow hop.

A trace shows payment taking most of the time but does not say whether all traffic has the same symptom. Collect only one kind of data and you have to guess the rest.

Signal	Answers	Limit	Concrete example
Metric	How many, what trend, which route?	Does not point to which request broke	5xx error rate of /orders/:id/pay
Trace	Which hops did the request pass through, how long each?	Only sees what is instrumented and sampled	Span payment.charge takes 850 ms of 900 ms
Log	What happened at this event?	May be sampled, redacted or missing	payment.timeout, deadline_ms: 330

What each signal looks like

Structured logs: named fields instead of one long string. This is actual output from the demo in the repo (two parallel requests, traceId preserved across await):

{"event":"request.start","traceId":"796e70b7-...","orderId":42}
{"event":"request.error","traceId":"1884c8d2-...","reason":"invalid order"}

Metrics: the “error rate” question is a PromQL query, not a log line:

sum(rate(http_requests_total{route="/orders/:id/pay", status=~"5.."}[5m]))
  /
sum(rate(http_requests_total{route="/orders/:id/pay"}[5m]))

Traces: a tree of spans with start, end and parent-child relationships; trace_id links them to logs. A trace is only a map of the segments that were measured, not an explanation of the cause: an 850 ms payment span may include waiting for a connection pool or retries.

The default drill-down: alert, exemplar, trace, log

Loading diagram…

An exemplar is the link between a metric and a trace: a histogram data point that carries the trace_id of a representative request.

Prometheus stores exemplars only when --enable-feature=exemplar-storage is enabled; they live in a fixed-size circular buffer in RAM, configured with the storage.exemplars block. Without exemplars, fall back to the manual approach: filter traces by route and time window.

Stop at a hypothesis, not a conclusion: one trace does not prove a cause. Cross-check with the dependency’s metrics, for example pool utilization.

Sampling: head vs tail

	Head sampling	Tail sampling
Decides at	The start of the trace, before the outcome is known	After seeing most of the spans
Criteria	Probability based on trace_id (for example 5%)	Errors, total latency, attributes, per-service rates
Pros	Simple, cheap, runs anywhere	Keeps error traces and slow traces
Cons	No guarantee of keeping error traces	Hard to operate: a stateful component, resource-hungry

It runs in the OpenTelemetry Collector (Tail Sampling Processor). Very large systems often combine both: head sampling to protect the pipeline, then tail sampling to select. If you use only head sampling, logs must carry enough information to investigate without a trace.

Percentiles: compute from histograms, never average them

• Per-instance p95 values cannot be summed or averaged; Prometheus states plainly that averaging quantiles yields statistically meaningless values.
• Use histograms: sum the buckets of every instance, then compute the quantile.

histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Bucket boundaries must be identical across instances. Native histograms make buckets compatible automatically and are recommended by Prometheus where available.

When to use which

Question	Start with
Is there an incident, how big?	Metrics (RED: Rate, Errors, Duration, for request-driven services)
Is a resource exhausted?	Metrics (USE: Utilization, Saturation, Errors, for resources)
Which hop is slow?	Traces
What exactly is the error?	Logs with trace_id

Real-world example

5xx rising on a payment route

Metrics point to the /orders/:id/pay route and when the rise started. From an exemplar, the team opens an error trace: the API waits a long time on the payment client. Logs with the same trace_id record timeout and the deadline value, with no card data or secrets.

The team checks connection pool metrics and finds the pool exhausted when traffic rises, fixes concurrency and the timeout, then uses metrics to confirm errors and latency are back to normal.

Run the demo (two parallel requests, ID 42 succeeds, -1 fails):

npx --prefix examples ts-node examples/src/observability-signals/signals-demo.ts

The last line is metrics { total: 2, errors: 1 }; requests is just an in-memory object, not a Prometheus counter, and spanId is just an illustrative string, not a standard span.

Remove context.run(...) to see logs lose the traceId while the counter still increments. For AsyncLocalStorage and propagation across the network, see correlation ID; for metric labels, see metric cardinality.

One request produces observability data at several layers

Enlarge diagram

Request → APItrace context

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The request enters the API, which creates or receives a trace context and records the start time.

🎮 Try it

• Log, metric or trace?: pick the right signal for each question.

Production checklist

1. Structured logs, every line with trace_id, route, status, duration_ms.
2. Metrics count exactly once per definition (do not count in both middleware and controller).
3. Metric labels have a bounded number of values; individual IDs go in logs or traces.
4. Latency is a histogram; percentiles are computed with histogram_quantile over summed buckets.
5. Standard propagation over HTTP and messages; verified by integration tests.
6. An explicit sampling strategy: whether error traces are kept (tail), or logs are enough to compensate.
7. Redact secrets and PII at the source; access control and retention for logs.

Trade-offs

Combining the three signals

• Metrics detect trends, traces narrow down, logs keep details.
• trace_id in logs shortens the search for one request’s data.
• Conventions for route, service and status make it easy to correlate layers.

Costs to control

• Logs and traces cost ingest; sampling and retention lose part of them.
• High-cardinality labels raise metric costs.
• Wrong instrumentation or broken context leads to wrong conclusions.

Common mistakes

Common pitfall

Treating a traceId in logs as having distributed tracing. Without spans, propagation and an exporter, it is only a correlation ID. The demo deliberately stops at this level.

Common pitfall

Averaging p95 across instances. The result has no statistical meaning. Sum the histogram buckets, then compute the quantile.

Common pitfall

Logging the whole payload “just to be sure there is debug data”. Secrets, tokens and PII live a long time in the log store. Pick the fields you need for investigation and redact at the source.

Summary

Key takeaways

• Metrics give scope and trend, traces give the path, logs give the event. Default drill-down: alert, exemplar, trace, log.
• Exemplars link metrics to traces; Prometheus needs --enable-feature=exemplar-storage.
• Head sampling is cheap but misses error traces; tail sampling keeps them but is hard to operate.
• Percentiles are computed from summed histogram buckets, never by averaging quantiles.

Self-check: (1) Why is a rising p95 not enough to conclude that payment is slow? (2) What does head sampling miss that tail sampling keeps? (3) Why not average the p95 across Pods?

References

• Signals — OpenTelemetry
• Sampling — OpenTelemetry
• Histograms and summaries — Prometheus
• AsyncLocalStorage — Node.js