Cardinality

Cardinality is the concept behind more “why did our observability fall over” incidents than any other. It’s also invisible until it hurts — which is exactly why it belongs in this group. If you internalise one metrics concept, make it this one.

What it is#

Cardinality = the number of unique time series you are storing.

A time series is a metric name plus every label value on it. Change any single label value and it becomes a different series — stored, indexed, and tracked separately.

http.server.duration{service="nest-prod", route="/users/:id", status="200", le="250ms"}   ← series #1
http.server.duration{service="nest-prod", route="/users/:id", status="200", le="500ms"}   ← series #2  (different le!)
http.server.duration{service="nest-prod", route="/reviews",   status="200", le="250ms"}   ← series #3

Each of those is a separate bucket of numbers-over-time the database maintains forever (until retention drops it).

Why it’s so impactful: it multiplies#

Cardinality isn’t additive, it’s combinatorial. A metric’s series count is roughly the product of its label cardinalities. A real example from our stack — signoz_latency:

Upper bound = 774 × 18 × 5 × 2 × 3 ≈ 418,000 possible series. Add one more label with 100 values and you don’t add 100 — you multiply the whole thing by up to 100. That’s the trap: labels feel free when you add them, and cost is hiding in a product you never multiplied out.

The three costs (why one number hurts everywhere)#

1. Query cost — the one you feel. To answer “p99 for service X over 7 days,” the database reads every datapoint of every matching series and aggregates. More series = more rows. We measured a single 7-day latency query reading 15.5 million rows. Cardinality is the direct input to how much a query has to chew through. (This is the “why is it doing so much work?” smell from how to debug.)

2. Memory & storage. The system keeps per-series metadata — a fingerprint, an index entry, cache slots. The collector holds a live map of every active series in RAM. 12k series is fine; 400k blows the cache, bloats the series table, and can OOM the collector or the database.

3. Ingestion & merges. Each series is written and continuously compacted. High cardinality means more parts and more background merge CPU — the quiet load that starves everything else on the box.

The rule that prevents 90% of disasters#

Before adding a label, ask one question: can its value space grow without bound?

One unbounded label = unbounded cardinality = eventual meltdown. This is why “template your routes” is the classic advice: it converts an unbounded dimension (/users/12345, /users/98, …) into one bounded value (/users/:id).

The counter-intuitive part: cardinality ≠ traffic#

This is what trips everyone up. It is not how many requests you get, it’s how many distinct label combinations:

• 1,000,000 requests to /health → 1 series. Cheap, at any volume.
• 1,000 requests, each to a unique /users/{id} → 1,000 series. Expensive.

High traffic on low cardinality is easy. Low traffic on high cardinality is what kills observability systems. You optimise for diversity, not volume.

Histograms are cardinality amplifiers#

A histogram (like latency) is not one series — it’s one series per bucket. Those 18 le buckets mean everything else is ×18. That’s why latency metrics are almost always your most expensive, and why percentile panels are the first to fall over.

How to audit it yourself#

The one query that shows your worst offenders (run in ClickHouse — see finding slow queries):

SELECT metric_name, uniqExact(fingerprint) AS series
FROM signoz_metrics.time_series_v4
GROUP BY metric_name
ORDER BY series DESC
LIMIT 20;

And to find which label is exploding a given metric — count distinct values per label key. A label with thousands of values on a metric you didn’t expect is your culprit.

The mental model to keep#

A metric costs roughly (product of its label cardinalities) × (data points over time). Before adding a label, ask “how many distinct values can this ever have?” If the answer is “it grows with users / requests / time,” it does not belong in a metric label — it belongs in a trace or a log, which are built for high-cardinality detail.

That last sentence is the whole reason we say metric is metric, trace is trace (see metrics vs traces vs logs): metrics are for bounded, aggregatable dimensions; traces and logs are where unbounded, per-request detail lives.