Embedding drift in production — what actually moves and how to detect it

The retrieval system was working fine. Until it was not. We did not push a new model. We did not change the chunker. We did not migrate the vector store. The quality of retrieved context just degraded over four weeks, and we caught it because the eval harness started failing on a slice we usually trusted.

This piece is about the class of failures that ate that quarter. It is about embedding drift in production — what moves, what does not, and how to detect the kind of drift that does not announce itself the way model regressions do.

What “drift” actually means for embeddings

The word “drift” is overloaded. In the classifier world, drift usually means one of three things: input distribution shift (your traffic looks different than it used to), concept drift (the relationship between inputs and labels changed), or model decay (the model is the same but it is now wrong about the world more often than it was). All three apply to embedding systems, but they show up differently.

Embedding drift comes in five flavors worth distinguishing:

Corpus drift. The thing you are embedding has changed. Your customer base is asking different questions, your docs got rewritten, the source-of-truth feed pushed an update. The embeddings themselves did not move; the work they have to do moved.
Distribution drift in the query side. Your retrieval queries are coming from different upstream sources, or your agent’s prompting changed, or seasonality kicked in. The query embeddings now sit in a different region of the space than the index assumes.
Tokenizer drift. You upgraded the embedding library and the tokenizer changed subtly. The same input string now produces different tokens, and therefore different vectors. The drift is silent because the API surface is the same.
Quantization drift. Your vector store applied product quantization or scalar quantization at index time, and the centroids it picked at indexing are no longer representative of new inserts. Recall degrades on the new inserts even though their nominal embeddings are fine.
Model drift. The provider updated the underlying embedding model. The model name is the same. The embeddings are different. This one is rare with proper versioning but still happens.

Three of those five are not visible at the model layer at all. If you only watch “is the embedding model healthy,” you will miss them.

What to instrument

The minimum-viable observability surface for an embedding system has four parts.

1. Distribution checks on the embedding space itself

Sample a fixed evaluation set of queries — five hundred to two thousand prompts that cover your real traffic — and run them through your embedding pipeline daily. Store the resulting vectors. Compute, against a frozen baseline:

Mean vector cosine. The cosine similarity between today’s mean embedding and the baseline mean. A drop of 0.02 is loud; a drop of 0.05 is screaming.
Per-dimension variance ratios. Some dimensions of your embedding space carry most of the signal. If their variance collapses, you have lost expressive range.
Cluster centroid stability. Run k-means on the eval set, save the centroids. Re-run weekly. Centroids that wander by more than a few cosine units are the most reliable early-warning signal we have found.

These do not require a real label set. They are cheap. They run in a nightly job.

2. Recall on a frozen ground-truth slice

The eval set should have a small subset (50–200 items) with known good retrievals — a query and the chunk(s) that the agent should retrieve to answer it. Run those queries against production every day. Record top-k recall. Alert on a 5-point drop.

This is the slowest signal but the most actionable. If recall stays flat while other metrics move, the drift is probably tolerable. If recall moves with the distribution metrics, something concrete broke.

3. Reindex-cost telemetry

Track the cost of re-embedding your entire corpus at current rates. Track when you last did it. The carrying cost of a stale index is real and often hidden. Teams routinely run six-month-old indexes against three-month-old query patterns without noticing.

4. Provider-side version stamps

Every embedding call should record the provider, model name, and (where available) model version. If your provider does not expose a version, complain and pin a checksum of a probe vector at deploy time. When the checksum changes, you have been silently upgraded.

Detection in practice

A working monitor looks something like this in pseudocode:

# nightly_drift_check.py
EVAL_SET = load_eval_set()                # 1000 frozen prompts
BASELINE = load_baseline_embeddings()     # locked at deploy
TODAY = embed_all(EVAL_SET)

mean_cosine = cosine(mean(TODAY), mean(BASELINE))
centroid_walk = max_centroid_distance(TODAY, BASELINE)

assert mean_cosine > 0.97, f"mean drift: {mean_cosine}"
assert centroid_walk < 0.08, f"centroid walk: {centroid_walk}"

recall_at_5 = ground_truth_recall(TODAY, GROUND_TRUTH, k=5)
assert recall_at_5 > 0.85, f"recall degraded: {recall_at_5}"

The thresholds are the part that takes work. Pick them on two weeks of healthy baseline data, then bake in 20% margin. The first time the check fails, you will be tempted to widen the threshold. Resist. The threshold is doing exactly what it should.

What we have seen

In the last four quarters we have seen each of the five drift flavors in production:

Corpus drift caught by a 7-point recall drop after a customer’s knowledge base was restructured. The embeddings were fine; the chunker had been tuned to the old structure and now produced misaligned chunks. Fix: re-chunk, re-index, re-run.
Query distribution drift caught by mean-cosine moving by 0.04 over two weeks. Cause: a new upstream agent had started rewriting user questions through a different system prompt, narrowing the query distribution. Fix: align the rewriter’s prompt with the original distribution; the alternative would have been to retrain the chunking strategy.
Tokenizer drift caught by a probe-vector checksum changing on dependency upgrade. The provider had pushed a backward-incompatible tokenizer fix. Fix: pin the library version; only upgrade with a full re-index.
Quantization drift caught by a recall metric that started rising and falling on a weekly cycle that matched our weekly bulk-insert job. Cause: the new inserts were not represented by the original PQ centroids. Fix: rebuild centroids on the unioned dataset.
Model drift caught once, by accident, when a colleague noticed the dimensionality of returned vectors had changed. Provider had migrated us silently to a new version that produced different dimensions. We caught it because the downstream code threw a shape error. We would not have caught it otherwise.

The last one is the most embarrassing and the most instructive. If your monitoring would not catch a silent dimensional change at the provider, your monitoring is not enough.

When to re-embed

Re-embedding the entire corpus is the nuclear option. It costs real money and the index is unavailable during the rebuild unless you double-write. The rule of thumb we use:

Re-embed when distribution metrics move by more than 2x your noise floor and stay there for three consecutive samples.
Re-embed when a tokenizer or model version pin changes.
Re-embed on a calendar cadence at least quarterly, even when metrics look fine. The drift you can measure is not the only drift there is.

We have stopped trusting “the index has not moved, so the index is fine.” It is fine until it is not. The cost of catching drift two weeks late is much larger than the cost of running the monitor.

What this does not cover

Three things are out of scope for this piece. Reranker drift is a different problem and deserves its own piece. Cross-encoder drift between retrieval and ranking is real and we have not figured out a clean monitor for it. And vector-store-side drift caused by index-side rebalances is something only the larger managed services would even know to look for; we do not have a public methodology for that.

For now: instrument the four signals above, set the thresholds on healthy baseline data, and stop treating embedding pipelines as “set it and forget it.” They drift. The cost of finding out which kind, after the fact, is much higher than the cost of watching.

— Reza Mokhtari