Prometheus + LLM observability — the cardinality problem

The Prometheus model is one of the cleanest in monitoring: pull-based scrape, dimensional labels, a query language that does the right thing for nearly every traditional observability question. The trade-off, which has not changed since Prometheus 1.0, is that the data model has a hard ceiling on cardinality. Time series with hundreds of distinct label values do not perform well. Time series with thousands kill the server.

LLM observability wants the opposite shape. Every call has a model name, a model version, a prompt template id, a tenant id, a feature flag bucket, a latency, a token count, a cost, and (if you are being thorough) a structured outcome label. You want to slice by every one of those dimensions. The cross product is the cardinality bomb.

This piece is about the patterns that let you keep Prometheus for the metrics you should and route the rest somewhere appropriate, without giving up on observability for your LLM stack.

What blows up

A naive instrumentation looks like this in pseudocode:

llm_request_count.labels(
    model=model_name,
    version=model_version,
    prompt_id=prompt_template_id,
    tenant_id=tenant_id,
    outcome=outcome_label,
).inc()

Five labels. Each one has, say, twenty distinct values. Cross product: 3.2 million time series. Add a sixth label with twenty values: 64 million. Prometheus is unhappy at that scale; your Prometheus host is unhappy first.

The cardinality problem is not unique to LLM observability — it shows up wherever the work has natural per-call uniqueness. But LLM observability is the most concentrated example I have seen. Every team I have advised on this in the last year has either melted their Prometheus or paid for managed Prometheus at a scale that justified the conversation.

What belongs in Prometheus

Prometheus is the right tool for low-cardinality, high-frequency aggregates. The LLM observability questions that fit:

Request rate per model.
Error rate per model.
p50, p95, p99 latency per model.
Token throughput per model.
Cost per model per hour.

“Per model” is two or three label values in a healthy production system. “Per model per version” is six or eight. “Per model per version per tenant” is wherever the cardinality cliff starts.

The rule of thumb: if a label can take more than ~50 values, it does not belong on a Prometheus metric. Promote it to a different observability layer.

What does not belong in Prometheus

Per-request detail, structured outcomes, prompt-template id, tenant id, user id, feature flag bucket. These are trace and log concerns, not metric concerns. The Prometheus model is not the right place for them, and the cost of forcing them in is real.

Three layers handle the per-request detail correctly:

Structured logs (Loki, Elasticsearch, BigQuery, whatever your team is on). Every LLM call writes one log line with all the dimensions you might want to query later. You aggregate at query time. The cost model is storage + query, not per-time-series.
Distributed tracing (Jaeger, Tempo, Honeycomb, the OTLP-compliant trace vendors). Every LLM call is a span. The span carries the dimensions you care about. You aggregate or filter by them when investigating. We covered the LLM-tracing patterns in a separate piece.
A purpose-built LLM observability vendor (LangSmith, Helicone, Phoenix, Arize, others). The category is real and the tooling is improving fast. The vendors handle the high-cardinality side natively and the integration cost is small.

The architecturally correct setup is Prometheus for the aggregates, structured logs or traces for the per-call detail, and (if you have the budget) a purpose-built LLM observability vendor for the work the first two cannot do natively.

The split-the-metric pattern

When you must keep something in Prometheus that has higher cardinality than is comfortable, the split-the-metric pattern is useful:

# instead of one metric with a tenant_id label:
llm_request_count.labels(model=model_name, tier="enterprise").inc()
llm_request_count.labels(model=model_name, tier="standard").inc()

# the tenant_id goes in the log line, not the metric.
log({
    "event": "llm_request",
    "model": model_name,
    "tenant_id": tenant_id,
    "prompt_id": prompt_template_id,
    "latency_ms": latency,
})

The metric carries the aggregate. The log line carries the specifics. The two correlate through timestamp and (if you set it up) trace id.

The histogram trap

Histograms are the metric primitive most likely to blow up. A histogram with default buckets has 10-12 buckets per label combination. Add a model label with 20 values, a version label with 5 values, and a tenant_id label with 100 values, and you are now at 10,000 distinct time series for one histogram metric.

The fix is to be deliberate about buckets and to never put a high-cardinality label on a histogram. A latency histogram should have model and version labels only. Tenant-side latency belongs in traces or logs.

When to leave Prometheus

There is a point in growth at which Prometheus is not the right tool for the metric layer either. You hit it when the aggregate metrics you legitimately want to keep are at scale that exceeds what a healthy Prometheus host can hold (or a managed Prometheus tier you are willing to pay for).

The alternatives:

VictoriaMetrics or Mimir for Prometheus-compatible storage at much higher scale. The migration cost is small if you have not built around Prometheus-specific tooling.
A purpose-built observability vendor (Datadog, New Relic, Honeycomb metrics). The cost model is per-metric-volume, which can be much friendlier than per-time-series at scale.
A pre-aggregation pipeline in front of Prometheus. You compute the aggregates in your application code (or in a streaming layer) and push the result. The trade-off is you cannot query at higher resolution later.

The choice depends on what the bottleneck is. If it is storage cost, VictoriaMetrics or Mimir. If it is operational complexity, a vendor. If it is the metric volume itself, pre-aggregation.

What we settled on

For our own stack, the structure is:

Prometheus for the four-or-five aggregate metrics per model (rate, error rate, latency histogram, token throughput, cost).
Structured logs in Loki for the per-call detail. Every LLM call writes one log line with the full dimension set.
OpenTelemetry traces, with the LLM call as a span on the request trace, carrying the prompt-template id and tenant id as span attributes.
A small slice of work in a purpose-built LLM observability vendor — the eval results, the per-prompt regression tracking, the things the general-purpose layer does not do well.

The total cost is lower than the alternative, which was running everything through Prometheus with a tenant label that we eventually had to remove after the third outage. The observability quality is higher because every layer is doing the work it is best at.

What I would tell a team starting today

Do not put a high-cardinality label on a Prometheus metric. The metric you put it on will be the metric you can no longer query when you most need to.

Decide early which observability layer owns which dimensions. The decision is much harder to walk back later than to make at the start.

And spend a little time auditing your Prometheus’s actual cardinality. The prometheus_tsdb_head_series counter is the one. If it is climbing, find the labels that are climbing with it and move them somewhere appropriate.

— Reza Mokhtari