High metric cardinality: one label can create thousands of time series Intermediate

Symptoms

After a deploy, Prometheus memory grows, scrapes slow down and dashboard queries take longer, while traffic is unchanged. The diff contains a seemingly harmless change: a user_id label added to the http_requests_total counter “to make filtering by customer easier”.

Cardinality is the number of distinct label value combinations of a metric; each combination is its own time series. A label with unbounded values (user ID, email, request ID, a URL containing an ID) turns the metric into an event lookup store, a job metrics were not designed for.

Reproduction: label dimensions multiply

Three route templates, two status groups, 100 fake users: 3 × 2 = 6 series; adding user_id gives 3 × 2 × 100 = 600.

Loading diagram…

The number is the ceiling of combinations that appear, not bytes of RAM. The table below shows where it starts to matter:

Series	Example	Risk level
6	route × status	Negligible
600	add 100 users	Still small, but the model is wrong; real users number in the millions
400,000	50 routes × 5 methods × 10 statuses × 8 regions × 20 instances	Needs a series budget and monitoring
10 million	add an unbounded label	An incident; do not rely on a “so many KB per series” rule, since it depends on version and workload

Histograms multiply further by the number of buckets (plus _count, _sum); an unbounded label on a histogram is even more expensive. instance, attached by Prometheus at scrape time, is also a multiplier by the number of replicas.

Run the combination simulation (no Prometheus needed):

npx --prefix examples ts-node examples/src/metric-cardinality/cardinality-demo.ts

Change 100 to 10_000 to see the number of combinations grow. Only combinations actually recorded create series; constantly changing dynamic values create churn (old series stop, new series appear), so the cost also depends on the rate of change.

How to detect it

Questions and the matching tools:

Want to know	Use
Total active series and trend	prometheus_tsdb_head_series
Rate of new series creation (churn)	rate(prometheus_tsdb_head_series_created_total[5m])
Which metrics have the most series	topk(10, count by (__name__) ({__name__=~".+"})) (heavy query, run with care)
Which labels have the most values	GET /api/v1/status/tsdb, read seriesCountByMetricName, labelValueCountByLabelName, memoryInBytesByLabelName, seriesCountByLabelValuePair
Deep analysis of one block	promtool tsdb analyze <db path>

The /api/v1/status/tsdb endpoint accepts a limit parameter (default 10, max 10000) and is an operational endpoint that should not be public.

Compare before and after a deploy to find new metrics or labels; total memory alone is not enough to conclude anything, because retention, ingest and query load all affect it.

In code, review every call to .labels(...): what is the maximum set of values, and can the client control them? Also check middleware that exports metrics on its own; it often uses the raw URL if route normalization is not configured.

Immediate mitigation

When Prometheus is already overloaded

Stop the source of new series first: roll back the instrumentation or drop the label at the source.

If you need to relieve pressure quickly, use metric_relabel_configs to drop the metric or label at scrape time, and check which alerts/SLOs depend on that metric before dropping it. Do not delete TSDB data by hand as the first step: it does not fix the source of new series.

Guardrails: let the scrape fail instead of letting Prometheus OOM

By default these limits are 0 (unlimited). When sample_limit, label_limit, label_name_length_limit or label_value_length_limit is exceeded, the whole scrape is treated as failed, so you see up == 0 instead of losing memory.

scrape_configs:
  - job_name: orders-api
    sample_limit: 20000             # max samples per scrape (after metric relabeling)
    label_limit: 30                 # max labels per sample
    label_value_length_limit: 200   # bytes
    target_limit: 50                # max targets for the job (experimental feature)

Pick the numbers from real observations of each job, plus a margin. The limits only block at the scrape boundary; labels derived from input still have to be normalized in code.

Long-term prevention

Series budget: estimate product of label values × number of replicas for each metric family before merging, and alert on prometheus_tsdb_head_series together with churn.

Review labels with a table:

Label	Max number of values	Bounded?	Owner
route	number of route templates	Yes (templates; unknown routes become unmatched)	API team
status_class	5	Yes	API team
tier	3 (free, paid, enterprise)	Yes	Billing team
user_id	millions	No, forbidden

Alternatives when you need to look up individuals:

• Exemplars (a metric pointing to a trace_id) plus logs or traces with retention and access control; see logs, metrics, traces.
• Normalize labels derived from input into a finite set; test that unknown input falls into a fallback group.
• Native histograms reduce the series caused by buckets compared to classic histograms.
• For high-dimensional analysis, use an event store or analytics system, not metrics.

🎮 Try it

• Log, metric or trace?: the trap of adding userId as a label.

Checklist

1. Every new label has a maximum number of values, is bounded and has an owner (the label review table above).
2. Do not use user_id, request_id, email or a raw path containing IDs as a label; use route templates, with unknown routes as unmatched.
3. Estimate the series budget (product of label values × number of replicas, times buckets for histograms) before merging.
4. Set sample_limit and label_limit per job based on real observations plus a margin.
5. Alert on prometheus_tsdb_head_series and churn rate(prometheus_tsdb_head_series_created_total[5m]).
6. After a deploy, compare /api/v1/status/tsdb before and after to find new metrics or labels.
7. When Prometheus is overloaded: stop the source first (roll back or drop the label), use metric_relabel_configs to relieve pressure, never delete TSDB data by hand.

Summary

Key takeaways

• Each combination of label values is a time series; dimensions multiply, and are multiplied again by replicas and buckets.
• user_id, request_id, email and raw paths containing IDs are unbounded labels.
• Detect with prometheus_tsdb_head_series, churn rate and /api/v1/status/tsdb; block with sample_limit and label_limit.
• Metrics use route templates and finite groups; individual details belong in logs or traces.

Self-check: (1) Why does adding 100 user IDs turn 6 combinations into up to 600? (2) What happens to the scrape when sample_limit is exceeded? (3) To look up one specific request, do you use metrics, logs or traces?

References

• Metric and label naming — Prometheus
• Instrumentation — Prometheus
• scrape_config — Prometheus
• TSDB stats API — Prometheus