Skip to content
  • AI
  • RAG
  • LLMs

RAG, fine-tuning or a better prompt? How to choose for your AI feature

Should your AI feature use a better prompt, retrieval (RAG) or fine-tuning? A plain guide for founders and CTOs, with the questions that decide it.

Portrait of Deep Soni

Deep Soni · Co-founder & Director, Trinevia

· 4 min read

Almost every AI project we scope starts with the same question from the founder: "Do we need to train our own model?" The honest answer is nearly always no. But "no" isn't a plan. There are three ways to make a language model good at your specific job, and they solve different problems. Pick the wrong one and you spend months and money fixing something that wasn't broken.

This guide explains the three options in plain terms, what each one is actually for, and the questions that tell you which one your feature needs.

The three options, in one sentence each

  • A better prompt: you give the model clearer instructions, examples and the right context in every request. Nothing is trained.
  • Retrieval-augmented generation (RAG): before the model answers, your system searches your own documents or database and passes the most relevant pieces into the prompt.
  • Fine-tuning: you train an existing model further on many examples of the input and output you want, so it learns a style, format or narrow task.

The key difference: RAG changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves. Prompting does a little of both, cheaply, and should always come first.

Always start with the prompt

Modern models are very capable out of the box. A large share of "the AI isn't good enough" problems are really prompt problems: vague instructions, no examples, no definition of a good answer, or asking for free text when the app needs structured data.

Before you consider anything heavier, try these:

  1. Write down what a perfect answer looks like, then put two or three real examples of one in the prompt.
  2. Ask for structured output (JSON with a fixed schema) when your code needs to use the result.
  3. Tell the model what to do when it isn't sure, such as "say you don't know" or "ask a clarifying question".
  4. Split one big request into smaller steps when the task has several parts.
  5. Try a stronger model. Sometimes the cheapest fix is a better model, not more engineering.

Use RAG when the model needs your knowledge

If the answer depends on information the model was never trained on, such as your help centre, contracts, product catalogue, policies or customer records, no prompt wording will fix it. The model has to be given that information at the time of the question. That is what RAG does.

RAG is the right choice when:

  • Answers must come from your documents or data, not general knowledge.
  • That information changes often. Update the documents and the answers change the same day, with no retraining.
  • Users need to see sources, so they can check an answer against the original document.
  • Different users may only see different data. Retrieval can respect permissions; a fine-tuned model cannot forget what it learned.

Most of the work in RAG is not the model. It's preparing the documents (splitting them sensibly, keeping titles and dates), choosing how to search (keyword, vector or both), and checking that the right pieces are being found. When a RAG system gives a bad answer, the cause is usually that it retrieved the wrong text, not that the model reasoned badly.

If the knowledge lives in a database rather than documents, the same idea applies with a different tool: the model writes a query instead of searching text. That brings its own safety questions, which we covered in what to check before connecting an AI agent to your production database.

Use fine-tuning when you need a consistent behaviour

Fine-tuning is useful for a narrower set of problems than most people expect. It teaches patterns, not facts. Consider it when:

  • You need a very consistent format, tone or style that prompting can't hold reliably across thousands of requests.
  • The task is narrow and repetitive, such as classifying support tickets into your own categories or extracting fields from one kind of document.
  • You want a smaller, faster or cheaper model to do one job as well as a larger one does it with a long prompt.
  • You already have hundreds or thousands of good input and output examples, or can create them.

A quick way to decide

  1. Does the answer depend on private or frequently changing information? Use RAG.
  2. Is the problem the format, tone or consistency of answers, and have prompt fixes run out? Consider fine-tuning.
  3. Is it neither, or are you not sure yet? Improve the prompt and measure.

These are not exclusive. A mature product might use a fine-tuned small model to classify each incoming question, RAG to fetch the right documents, and a carefully written prompt to produce the final answer. But almost every good AI feature we have seen got there in that order: prompt first, retrieval when knowledge is missing, fine-tuning last and only for a clear reason.

What each option costs in time

  • Prompting: days. Changes are instant and cheap to try.
  • RAG: typically weeks for a solid first version, most of it spent on preparing data and testing retrieval quality.
  • Fine-tuning: depends mostly on your training data. If good examples don't exist yet, creating them is the biggest job, and you'll repeat some of that work whenever the base model changes.

If you're planning an AI feature and aren't sure which route fits, book a call and bring a few real examples of the questions your users ask. That is usually enough to tell which option you need.

Let's talk about what you're building.

Book a free 30-minute call. You'll leave with honest feedback on scope, a rough timeline and a budget range, whether or not we work together.

Pick a time that suits you

Times are shown in your own time zone.

Or open the booking page in a new tab

Loads Calendly, which may set its own cookies.