A product manager walks over after a demo and asks a familiar question: should we fine-tune the model? The assistant answered in the wrong tone, missed one policy detail, and invented a field that does not exist in the CRM. Around the table, people start naming solutions before naming the problem. Someone says prompt. Someone says RAG. Someone says fine-tuning. The model, quietly, is being blamed for three different failures at once.
Prompting, RAG, and fine-tuning are often discussed as if they are competing options on the same shelf. They are related, but they change different things. Prompting changes the instruction and context you give the model at request time. RAG changes the information the model can look at before answering. Fine-tuning changes the model behavior by training it on examples. Choosing well starts by asking what is actually missing: clearer instructions, fresher knowledge, or a repeated behavior the base model does not reliably perform.
Prompting is usually the first lever because it is fast and reversible. A better prompt can define the task, audience, tone, format, boundaries, examples, and refusal behavior. If the assistant writes too casually, ask for a calmer tone and show two examples. If it returns inconsistent JSON, define the schema and include valid output. If it answers outside the policy, tell it to use only provided context and say when evidence is missing. Prompting is not a toy step. It is often the cheapest way to discover what the product actually needs.
The limit of prompting is that it cannot create reliable knowledge from nowhere. A prompt can remind the model to follow the refund policy, but it cannot know the newest refund policy unless you provide it. A prompt can ask for a database field name, but if the schema changes every week, the field list needs to come from somewhere current. When the problem is changing knowledge, internal documents, product rules, support history, or citations, RAG is usually the more honest tool.
RAG, or Retrieval-Augmented Generation, means the system searches trusted sources first and gives the relevant pieces to the model before it answers. It is useful when the answer should be grounded in material your team owns: docs, tickets, knowledge base articles, runbooks, release notes, or code snippets. The model still writes the response, but the evidence comes from retrieval. This is why RAG pairs naturally with citations, freshness checks, and source visibility.
RAG fails in different ways from prompting. It may retrieve the wrong chunk, miss the right document, pull stale content, or bring back conflicting sources. The model may also ignore part of the retrieved context. Improving a RAG system often means improving the boring pipeline around it: document ownership, chunking, metadata, permissions, evaluation sets, and observability. If the knowledge base is messy, RAG will not make it wise. It will often make the mess easier to find.
Fine-tuning enters the conversation when the team needs the model to learn a repeated pattern from examples. Maybe the product requires a very specific classification style, a consistent extraction format, a domain-specific rewriting pattern, or a support response voice that ordinary prompting cannot hold reliably at scale. Fine-tuning can reduce prompt length, improve consistency on narrow tasks, and make behavior feel more native to the model.
Fine-tuning is not the right answer to missing facts. If the pricing table changes every month, training it into the model is usually the wrong place to put that knowledge. The model will become stale, and the team will need another training cycle for what should have been a data update. Fine-tuning also needs good examples, evaluation, versioning, and rollback discipline. A weak dataset can teach the model the team's old mistakes with impressive confidence.
A practical decision path is simple. If the model is confused about the task, improve the prompt. If it lacks current or private knowledge, use RAG. If it repeatedly performs the task in the wrong style or structure even with good prompts and context, consider fine-tuning. Many good systems combine all three: a clear prompt, retrieved evidence, and a tuned model for a narrow behavior. The point is not purity. The point is using the smallest intervention that makes the product reliable.
Evaluation keeps the choice grounded. Before changing the system, collect a small set of real user inputs and expected outcomes. Mark which failures are instruction failures, retrieval failures, or behavior failures. After each change, test the same set again. This habit prevents a team from fine-tuning to solve a documentation problem, or building a vector database to solve a vague prompt.
There is also a cost conversation. Prompting costs time and tokens. RAG costs indexing, permissions, retrieval quality, and knowledge maintenance. Fine-tuning costs dataset preparation, training, monitoring, and model lifecycle management. None of these are free, even when the API call looks simple. The right choice is the one whose operational cost matches the value and risk of the feature.
The calm version of the question is not whether prompting, RAG, or fine-tuning is best. It is what kind of failure you are seeing. Is the model unclear about the job? Is it missing evidence? Is it unable to repeat a behavior your product needs? Once the failure has a name, the solution becomes less fashionable and more useful. If your team has tried all three, the most interesting stories are often not about which one won, but about the moment you finally understood what problem you were actually solving.