On-call runbooks for AI products — the new incident class

The 3 AM page used to be predictable. The database is melting. The cache is cold. The deploy broke the auth path. The runbook said: roll back, restart, redeploy, escalate.

The new pages are not predictable. The model provider rolled out a silent change. The retrieval quality degraded by some amount that is hard to quantify in the moment. The agent loop is now infinite on a class of inputs it handled fine yesterday. The runbook does not exist for these, because the runbooks were written for systems that fail in measurable, mechanical ways.

This piece is the field guide we have built over the last year of running AI-product on-call. It is the list of new incident classes, the detection patterns, and the runbook templates that have held up when the page lands.

The new incident classes

Class 1: Quality degradation without an outage

The system is up. The latency is fine. The error rate is the same. The answers got worse. This is the incident class that wakes people up most often in mature AI products, and it is the one for which traditional runbooks are most useless.

Causes we have seen:

• Provider-side model change (silent or announced but missed).
• Retrieval quality drop from corpus drift, embedding drift, or index rebalancing.
• Prompt change in an upstream system that altered the input distribution.
• Tool-side change (an API a tool calls is returning different data).

The detection signal is almost always downstream — user feedback rate, support ticket volume, a slow drop in a quality metric you track over a long window. The first hour of an incident in this class is usually “is this actually a regression or is it noise.”

Class 2: Cost spike without a traffic spike

The bill is rising. The traffic is flat. Somewhere in the system, a loop is generating more tokens than it used to, or a routing decision is sending more traffic to an expensive model, or a tool is being called more often per request.

Causes we have seen:

• A retry loop that triggers on a class of inputs that used to be rare.
• A prompt change that made the model verbose.
• A routing rule that now sends more traffic to a frontier model than to the cheap model.
• A tool that is being called repeatedly because the agent does not believe its first response.

Detection: per-route token-cost metric, alerting on sudden derivative. The alert should not fire on a traffic spike; it should fire on cost-per-request rising.

Class 3: The infinite (or near-infinite) agent loop

An agent step decides to call itself, or two agents start calling each other, or a tool’s output triggers the same tool again. The agent runs until the wall-clock timeout, the cost budget, or the safety net.

Causes we have seen:

• A prompt change that made the agent more willing to retry.
• A tool that started returning a response the agent interpreted as “try again.”
• A change in the agent’s stopping criterion.
• A new agent step that the orchestrator does not have a budget for.

Detection: per-agent-call step-count metric. Alert on p99 step count rising.

Class 4: The silent prompt or model drift

The prompt was updated. The model was migrated. The eval set is six months old. Quality dropped on a slice of inputs the eval set does not cover. The dashboards look fine because the dashboards were built against the eval set.

Causes:

• Prompt rotation without a corresponding eval-set update.
• Model migration without a regression test on the production traffic distribution.
• A new feature flag that changed prompt selection for a tenant cohort.

Detection: shadow eval against a sample of production traffic, run continuously. The eval set is not the ground truth; it is the regression baseline. The production sample is the ground truth.

Class 5: The tool dependency cascade

A downstream tool is degraded. The agent that uses it produces lower-quality outputs. The agent that uses that agent (in a multi-agent system) compounds the lower quality. The user-facing impact is much larger than the underlying tool incident.

Causes:

• A vendor outage on a single tool.
• A rate limit being hit on a tool the agent had been freely using.
• A schema change in a tool’s output that the agent does not handle gracefully.

Detection: per-tool error rate AND per-tool downstream-quality metric. Either one is not enough on its own.

What the runbooks need

The runbooks for these classes need to look different from web-tier runbooks. Three structural differences:

Less rollback, more freeze

The instinct in a traditional outage is to roll back the deploy. In an AI-product incident, the deploy is rarely the cause. The cause is usually upstream of your deploy (provider-side change, vendor-side change, corpus drift). Rolling back is not the right first step. Freezing the configuration is.

The first runbook step for most of these incidents is: pin the model version, pin the prompt template version, pin the tool API versions, freeze the routing rules. Stop the ground from moving while you investigate.

More observability, less restart

A web-tier runbook ends with “if all else fails, restart the service.” An AI-product runbook ends with “if all else fails, lower the budget and route traffic to a known-good path.” Restarting a stateless inference service does not fix a model-quality regression. Reducing the surface area until the regression is contained does.

Different escalation paths

The on-call rotation is usually owned by infra or platform. AI-product incidents often need ML or applied-AI input — someone who knows what a healthy embedding distribution looks like, or what a prompt change means semantically. The runbook should name the second-tier contact explicitly and the escalation should not wait for the first-tier to fully diagnose.

A template runbook for “quality dropped, no outage”

1. Confirm the regression.
   - Check the production-eval shadow score over the last 24 hours.
   - Check the user-feedback rate over the last 24 hours.
   - If neither shows a clear drop, downgrade to investigation.

2. Identify the change window.
   - Last 24 hours: any prompt deploys? any model migrations? any provider announcements?
   - Last 7 days: any corpus changes? any tool updates?
   - If none, suspect provider-side change.

3. Freeze.
   - Pin the current model version (set the request to a versioned alias).
   - Pin the prompt-template version.
   - Disable any feature flags that touch model routing.

4. Run the regression test.
   - Re-run the eval set against the current production config.
   - Compare against the last known-good baseline.
   - If the eval set shows the regression, you have a reproducible failure.

5. Bisect.
   - Roll back changes one at a time and re-run.
   - The first change that restores the metric is the cause.

6. Mitigation.
   - Roll forward to a fixed prompt or routed-around config.
   - Notify users if the impact was meaningful.

7. Post-incident.
   - Add a regression test that would have caught this.
   - Update the eval set if the failure mode was novel.

This is condensed and incomplete; the production version of this runbook is about 4 pages with screenshots. The structure holds.

What we are still learning

Three things on the open-questions list:

• Severity classification. Web-tier severity is well-understood (SEV-1 is “site is down,” SEV-3 is “one user has a workaround”). AI-product severity is harder. A 5-point quality drop on a slice of traffic that affects 2% of users is what severity? We have not settled on the answer.
• Cross-team handoff. AI-product incidents often span infra, ML, applied AI, and the product team. The handoff protocols are still being figured out. We have run incidents where the first-responder was on infra and the actual cause was a prompt change owned by a product engineer who was off-shift.
• The post-incident review template. Traditional Five Whys does not work well on AI-product incidents because the cause is often distributed across multiple subtle factors. We are experimenting with a “contributing factors” template that lists 3-5 factors and the interaction between them. Not yet sure it is better.

The thing that is settled is that the runbooks need to be written. The default of “we will figure it out at the moment” produces 4 AM incident chats where everyone is reverse-engineering the system from the symptoms. The teams that have invested in runbooks for these classes resolve incidents in roughly one-third the time of teams that have not.

If you are running AI products and your on-call runbook is the runbook you wrote for the web-tier service, the runbook is incomplete. Write the AI-product chapters.

— Ginger Wolfe-Suarez