Fine-tuning vs prompt engineering — a cost/quality decision tree

The fine-tuning conversation is loud again, mostly because two of the larger model providers shipped easier fine-tuning APIs last quarter. Teams that had ruled out fine-tuning on operational grounds two years ago are revisiting the decision. Most of them should not actually fine-tune. A few of them should. The decision is more conditional than the marketing makes it sound.

This piece is the decision tree we have helped teams walk through over the last quarter. It is not a benchmark; it is a framework for picking between three things — prompt engineering, retrieval-augmented generation, and fine-tuning — that are often presented as alternatives when they are actually compositional.

The three options are not really three

Most production deployments use all three in combination. The question is not “fine-tune or not” but “how much of the work should each layer do.”

• Prompt engineering does the work that is task-specific but context-light. System prompts, few-shot examples, output formatting, role framing. Cheap to iterate. Easy to roll back.
• RAG does the work that requires the model to know specific facts the base model does not know. Customer data, internal docs, recent events. The “knowledge” layer.
• Fine-tuning does the work that requires the model to behave differently — different output structure, different domain vocabulary, different reasoning patterns. The “behavior” layer.

Most teams come asking “should we fine-tune?” when the actual question is “is our prompting work hitting a ceiling that fine-tuning would lift?” Usually it is not. Sometimes it is. The decision tree below is the heuristic.

Step 1: Is your output structure stable?

If the model needs to produce JSON, XML, or some fixed format on every call, and you are seeing format errors more often than you can tolerate, you have a structure problem. Prompt engineering plus a good library (BAML, Outlines, Instructor) usually solves it. Fine-tuning solves it more reliably but is overkill for the problem.

If your structure-error rate is below 0.5% on a high-quality model, the structure layer is fine. Move to step 2.

Step 2: Is the model missing specific facts?

If the failures look like “the model said something true in general but wrong for our domain,” you have a knowledge problem, not a behavior problem. RAG is the answer. Fine-tuning to inject facts is one of the most common ways to spend $50,000 on something that does not work; the model conflates facts, forgets them, or produces them in the wrong contexts. Use retrieval.

If the failures persist after a healthy RAG setup with a good reranker, move to step 3.

Step 3: Is the failure mode a stylistic or structural mismatch?

This is the slot where fine-tuning is the right answer.

Examples of fine-tuning-worthy failure modes:

• The model produces verbose output when your application needs terse. Prompting can help; fine-tuning is more reliable.
• The model uses standard medical or legal vocabulary when your audience expects domain-specific terminology. Prompting can teach it the terms; fine-tuning teaches it the register.
• The model reasons in a step-by-step style when your output channel needs single-paragraph conclusions. Prompting can constrain this; fine-tuning bakes it in.
• The model follows the safety-trained refusal patterns of the base model when your domain (medical research, security research) requires it not to. Fine-tuning can shift the default; prompting is unreliable at this.

If your failure mode is one of these, and you have ruled out prompting and RAG, fine-tuning is the right call. Be honest with yourself about which step you are at.

The cost numbers

Approximate at provider list rates, for a typical production fine-tune workload:

The per-call delta for fine-tuning is sometimes negative. A well-fine-tuned smaller model can replace a larger base model with prompt engineering, and the inference cost drops. This is where the fine-tuning case actually makes economic sense for high-volume calls: you trade a one-time setup cost for a permanent per-call savings.

The threshold to justify a $20,000 fine-tune on inference savings alone, at a $0.0003-per-call savings, is about 70M calls. If your volume is in that range, run the math. If it is not, do not fine-tune for cost alone.

What goes wrong

Three failure modes we have seen repeatedly:

Fine-tuning to compensate for a small training set

The team has 200 examples of the behavior they want. They fine-tune. The result is a model that memorizes the 200 examples and generalizes badly. The fix: fine-tuning wants 5,000–50,000 examples for behavior shift; if you have 200, use them as few-shot examples in the prompt.

Fine-tuning a model you cannot version

The team fine-tunes a model. Six months later, the base model is deprecated. The fine-tuned weights are not portable to the successor. The team has to redo the fine-tune. The fix: budget for re-fine-tuning. Pick base models with longer deprecation timelines if you can.

Fine-tuning instead of RAG for fact injection

This is the most common mistake. The team has facts they want the model to know. They fine-tune them in. The model conflates the facts with the base training data. RAG would have done the job better with less work. The fix: fine-tune for behavior, retrieve for facts.

The decision tree, condensed

Are output structures failing?
  → Yes: Try prompting + a structured-output library first.
         If still failing: fine-tune for structure.
  → No: Continue.

Is the model wrong about specific facts in your domain?
  → Yes: RAG. Add a reranker. Tune retrieval before considering fine-tuning.
  → No: Continue.

Is the failure mode stylistic, structural, or about register?
  → Yes: Fine-tune. Budget 5,000+ examples.
  → No: Improve prompts and instrumentation; the problem may be evaluation, not modeling.

Is high-volume inference cost a binding constraint?
  → Yes: Fine-tune a smaller base model. Run the math on the call volume threshold.
  → No: Stay on the bigger base model with prompt engineering.

What we are watching

Two threads we will write about again over the next quarters.

The first is parameter-efficient fine-tuning (LoRA, QLoRA, prefix-tuning). The cost numbers above assume full fine-tuning. PEFT methods are 10–100x cheaper to set up, harder to do right, and are increasingly the right answer for the middle-ground use cases. We will publish a methodology piece on PEFT against the decision tree above.

The second is fine-tuning observability. The fine-tuning workflow is bad at telling you whether the fine-tune actually did what you wanted. Eval sets need to be rebuilt for each fine-tune, drift needs to be tracked, and the comparison against the base model is rarely apples-to-apples. The tooling is improving but is still ahead of its time. A future piece will cover what to instrument when you do fine-tune.

For most teams, the right answer to “should we fine-tune?” is “not yet.” The decision tree above is the version of that answer with specifics.

— Ginger Wolfe-Suarez