OpenTelemetry for LLM applications — instrumentation patterns

The OpenTelemetry GenAI semantic conventions stabilized late last year. They are not perfect — the spec is still missing some patterns the production teams I have talked to have wanted for two years — but they are good enough that “we are waiting for the standard” is no longer a defensible reason to not instrument.

This piece is the practitioner’s read on the spec: what to instrument, what the conventions name (and what they do not), what to do for the patterns the spec misses, and what we have learned by running the resulting traces under real production load.

What the spec covers

The GenAI semantic conventions name a small set of attributes that every LLM call should carry on its span:

• gen_ai.system — the provider (anthropic, openai, google, bedrock).
• gen_ai.request.model — the model name as requested.
• gen_ai.response.model — the model name as returned (these can differ on aliased calls).
• gen_ai.request.max_tokens, temperature, top_p — the request parameters.
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — the post-hoc token usage.
• gen_ai.response.finish_reason — stop, length, content_filter, etc.
• gen_ai.operation.name — chat, text_completion, embeddings.

The spec also names events for the prompt content and the completion content, with an opt-in flag for whether to record them. The conservative position is to not record them by default and to record them when investigating a specific issue.

That is the official surface. It is enough for the basic observability work — request rate, error rate, latency, token throughput per model — and it lets you slice by provider and by model. It does not give you, by itself, prompt-template-level visibility or tenant-level visibility. Those are work the spec leaves to the application.

The standard instrumentation pattern

A standard pattern, in pseudocode:

from opentelemetry import trace
tracer = trace.get_tracer("my-llm-app")

def call_llm(prompt, model):
    with tracer.start_as_current_span(
        "chat",
        attributes={
            "gen_ai.system": "anthropic",
            "gen_ai.operation.name": "chat",
            "gen_ai.request.model": model,
            "gen_ai.request.max_tokens": 2048,
            "gen_ai.request.temperature": 0.7,
        },
    ) as span:
        try:
            response = client.messages.create(model=model, messages=prompt)
            span.set_attributes({
                "gen_ai.response.model": response.model,
                "gen_ai.usage.input_tokens": response.usage.input_tokens,
                "gen_ai.usage.output_tokens": response.usage.output_tokens,
                "gen_ai.response.finish_reason": response.stop_reason,
            })
            return response
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

This is enough to give you the basic observability picture in any OTLP-compatible backend (Tempo, Jaeger, Honeycomb, Datadog APM, the OTel-native LLM observability vendors). You can query by model, by provider, by finish reason, and by latency. You can correlate with the rest of your application traces because the span is part of the same trace.

What the spec does not cover

Three patterns I have wanted in production and the spec does not provide directly:

Prompt-template identification

Every LLM call in a serious application is rendered from a versioned prompt template. The version matters because prompt changes are a primary cause of behavior change. The spec does not have a gen_ai.request.prompt_template attribute.

The workable workaround is a custom attribute. We use app.prompt_template_id and app.prompt_template_version. The spec allows custom attributes; the cost is that vendor dashboards do not know to surface them. If you are running your own queries this is fine.

Tenant identification

Most production agentic systems are multi-tenant. Tenant-level slicing of LLM cost and latency is a basic requirement. The spec does not include a tenant id attribute (correctly — tenant id is application-specific).

We use app.tenant_id. Same caveat: vendor dashboards do not know to surface it.

Tool call structure

The spec does name tool calls but the structure is thin. For an agentic call where the LLM decided to call three tools, the span captures the parent call but does not natively capture the per-tool spans as children. The pattern we use is one parent span for the LLM call and child spans for each tool execution, manually correlated.

In code:

with tracer.start_as_current_span("agent.step") as parent:
    parent.set_attribute("gen_ai.system", "anthropic")
    response = client.messages.create(...)
    parent.set_attribute("gen_ai.usage.input_tokens", ...)

    for tool_call in response.content:
        if tool_call.type == "tool_use":
            with tracer.start_as_current_span(f"tool.{tool_call.name}") as ts:
                ts.set_attribute("app.tool.name", tool_call.name)
                ts.set_attribute("app.tool.input", json.dumps(tool_call.input))
                result = execute_tool(tool_call)
                ts.set_attribute("app.tool.output_chars", len(result))

This composes correctly with the spec’s call structure and gives you the tool-level latency picture, which is the thing you need when an agentic call is slow and you want to know which tool is the culprit.

Sampling

The default OTel sampler keeps every span. For LLM calls this is fine in low-volume contexts and disastrous in high-volume contexts. At 1,000 calls per second, recording every span is meaningful storage and meaningful cost.

The pattern that works:

• Sample LLM spans at 100% during incidents. Use a runtime-toggleable sampler that flips to 100% when a feature flag is set.
• Sample at 1-10% during steady state. Use head-based sampling to keep the rate manageable.
• Always record error spans. Tail-based sampling, where the collector decides post-hoc to keep an error trace at 100%, is the standard pattern.

The OTel collector’s tailsampling processor handles this. We have used it in production for over a year; it works.

Cardinality on spans

The cardinality problem we wrote about for Prometheus applies to traces too, but the trade-off is different. Span attributes do not pay the same per-time-series cost; they pay a per-span cost. You can put high-cardinality attributes on spans freely. The trade-off is dashboard usability and query latency.

Practical rules:

• Put app.tenant_id and app.prompt_template_id on the span. They are query-time slices you will want.
• Do not put the prompt content on the span unless you have opted in deliberately. It is expensive in storage and a privacy risk by default.
• Put the response token counts on the span. They are cheap and useful.

What we are watching

Three threads:

• The LLM-evals semantic conventions are not yet stable. Once they land, the OTel-native vendor dashboards will likely converge on a more useful default surface. We will revisit instrumentation.
• Streaming responses are still a sharp edge in the spec. The span is opened when the call starts and closed when streaming ends; the response_model and usage attributes are set at the end. If your application processes streaming chunks before the call closes, your span data is incomplete during the stream. The spec is being worked on; for now, we record an app.stream.first_chunk_ms attribute manually.
• OTel for multi-step agentic flows is the gap that needs the most work. A single trace for an agent that runs ten steps over thirty seconds is hard to navigate in current dashboards. Several vendors are building purpose-built agent-trace UIs on top of OTel data. We will write a survey when the category matures.

The basics — instrument every LLM call as a span, use the GenAI semantic conventions, add custom attributes for tenant and template, sample with tail-based logic, do not record prompt content by default — work. Most production teams I talk to are doing some subset of this. The teams that are doing none of it are flying blind on the most important layer of their stack.

Start with the spec. Add the custom attributes you need. Sample. Watch.

— Reza Mokhtari