Cardinality

Cardinality is the concept behind more “why did our observability fall over” incidents than any other. It’s also invisible until it hurts, which is exactly why it belongs in this group. If you internalise one metrics concept, make it this one.

What it is#

Cardinality = the number of unique time series you are storing.

A time series is a metric name plus every label value on it. Change any single label value and it becomes a different series, stored, indexed, and tracked separately.

http.server.duration{service="nest-prod", route="/users/:id", status="200", le="250ms"}   ← series #1
http.server.duration{service="nest-prod", route="/users/:id", status="200", le="500ms"}   ← series #2  (different le!)
http.server.duration{service="nest-prod", route="/reviews",   status="200", le="250ms"}   ← series #3

Each of those is a separate bucket of numbers-over-time the database maintains forever (until retention drops it).

Why it’s so impactful: it multiplies#

Cardinality isn’t additive, it’s combinatorial. A metric’s series count is roughly the product of its label cardinalities. A real example from our stack, signoz_latency:

Upper bound = 774 × 18 × 5 × 2 × 3 ≈ 418,000 possible series. Add one more label with 100 values and you don’t add 100, you multiply the whole thing by up to 100. That’s the trap: labels feel free when you add them, and cost is hiding in a product you never multiplied out.

The three costs (why one number hurts everywhere)#

1. Query cost, the one you feel. To answer “p99 for service X over 7 days,” the database reads every datapoint of every matching series and aggregates. More series = more rows. We measured a single 7-day latency query reading 15.5 million rows. Cardinality is the direct input to how much a query has to chew through. (This is the “why is it doing so much work?” smell from how to debug.)

2. Memory & storage. The system keeps per-series metadata, a fingerprint, an index entry, cache slots. The collector holds a live map of every active series in RAM. 12k series is fine; 400k blows the cache, bloats the series table, and can OOM the collector or the database.

3. Ingestion & merges. Each series is written and continuously compacted. High cardinality means more parts and more background merge CPU, the quiet load that starves everything else on the box.

The rule that prevents 90% of disasters#

Before adding a label, ask one question: can its value space grow without bound?

One unbounded label = unbounded cardinality = eventual meltdown. This is why “template your routes” is the classic advice: it converts an unbounded dimension (/users/12345, /users/98, …) into one bounded value (/users/:id).

The counter-intuitive part: cardinality ≠ traffic#

This is what trips everyone up. It is not how many requests you get, it’s how many distinct label combinations:

• 1,000,000 requests to /health → 1 series. Cheap, at any volume.
• 1,000 requests, each to a unique /users/{id} → 1,000 series. Expensive.

High traffic on low cardinality is easy. Low traffic on high cardinality is what kills observability systems. You optimise for diversity, not volume.

Histograms are cardinality amplifiers#

A histogram (like latency) is not one series, it’s one series per bucket. Those 18 le buckets mean everything else is ×18. That’s why latency metrics are almost always your most expensive, and why percentile panels are the first to fall over.

How to audit it yourself#

The one query that shows your worst offenders (run in ClickHouse, see finding slow queries):

SELECT metric_name, uniqExact(fingerprint) AS series
FROM signoz_metrics.time_series_v4
GROUP BY metric_name
ORDER BY series DESC
LIMIT 20;

And to find which label is exploding a given metric, count distinct values per label key. A label with thousands of values on a metric you didn’t expect is your culprit.

The mental model to keep#

A metric costs roughly (product of its label cardinalities) × (data points over time). Before adding a label, ask “how many distinct values can this ever have?” If the answer is “it grows with users / requests / time,” it does not belong in a metric label, it belongs in a trace or a log, which are built for high-cardinality detail.

That last sentence is the whole reason we say metric is metric, trace is trace (see metrics vs traces vs logs): metrics are for bounded, aggregatable dimensions; traces and logs are where unbounded, per-request detail lives.