Fine-tuning: when it pays to train a model on your data

What actually happens

A model learns in two big phases (I describe them in the lesson on how an LLM works): pre-training on huge amounts of text, then training to follow instructions and behave like an assistant. Fine-tuning is a third phase, small and yours: you show it many “request → ideal answer” pairs and the model slightly adjusts its parameters to behave that way.

The result is a model that:

• always produces the same format without long instructions;
• adopts the style and vocabulary of your examples;
• performs a specific task (classify, extract, rewrite) more precisely;
• often lets you use a smaller model, faster and cheaper, instead of a large one.

What it doesn’t do well: reliably learn new facts. Train it on this year’s price lists and it will “remember” them approximately and won’t be able to tell you where a figure comes from; and when the prices change, you have to retrain it.

Prompt, RAG or fine-tuning?

You need the model to…	Right tool	Why
Follow rules and tone	Prompt with instructions and examples	Immediate, free to change
Answer from up-to-date documents and data	RAG	Cites sources, updates when you upload files
Always do the same narrow task, in a precise format, at high volume	Fine-tuning	More precise and often cheaper per request
Speak a very particular industry language	Fine-tuning (possibly with RAG)	Learns vocabulary and style from examples

The three combine: a model specialised by fine-tuning can still receive the right documents from RAG and instructions in the prompt.

Five questions before you start

1. Have I tried a serious prompt? With clear instructions and 5-10 complete examples in the prompt. Often the problem is solved here.
2. Is the problem about knowledge or behaviour? If it’s knowledge (documents, prices, procedures), you need RAG, not fine-tuning.
3. Do I have enough good examples? You need at least a few hundred correct, consistent pairs, ideally thousands. Wrong examples teach mistakes.
4. Does the volume justify it? With a few dozen requests a day, the savings won’t repay the work.
5. Can I measure whether it improves? Without a test set (a set of questions with the right answer, to compare models on) you won’t know if the specialised model is really better.

If you answer “yes” to all of them, fine-tuning makes sense.

The dataset is everything

Fine-tuning quality depends almost entirely on the examples. Each example is a conversation: the instructions, the request and the answer you’d always want.

• Variety: cover normal cases, hard cases and exceptions, in realistic proportions.
• Consistency: the same rules in every example. If half the answers use one format and half another, the model will learn the confusion.
• Cleanliness: remove personal and confidential data. What goes into training can resurface in answers.
• Separation: set aside 10-20% of examples the model never sees in training, for testing.

A common trick: have a large model draft the answers, then correct them by hand and use them to specialise a small model. It’s called distillation: the small model learns to imitate the large one on your task, at a much lower cost.

Two routes: provider or open model

1. Fine-tuning with the provider. Some closed-model providers let you specialise certain models by uploading a dataset: you pay for the tokens used in training and then a price to use the specialised model. It’s the simplest route, no hardware. Limits: not every model is available, your data goes to the provider (check the contract) and you stay tied to them.

2. Fine-tuning an open model. You take an open-weights model (see the lesson on local AI) and specialise it yourself, on your hardware or on rented GPUs. This is where LoRA and QLoRA come in: instead of modifying all the billions of parameters, you train a small “add-on layer” of a few million parameters, and with QLoRA the base model is also compressed to 4 bits. Result: specialising a 7-30 billion parameter model becomes possible on a single GPU, often in a few hours and for tens of euros of rented compute. Libraries like Hugging Face’s PEFT or Unsloth make the process far more accessible.

How to tell if it worked

Compare on the test set you set aside:

• the base model with your best prompt;
• the specialised model with a short prompt.

Look at quality, cost per request and speed. If the specialised model doesn’t clearly beat the well-crafted prompt, it’s not worth keeping in production. Also check that it hasn’t lost general abilities you need: an over-specialised model can get worse at everything else.

Examples where it makes sense

• A firm that classifies tens of thousands of documents a month into internal categories: a small specialised model is precise and very cheap.
• A company that must generate product descriptions always in the same format and in its own style, for a huge catalogue.
• A service handling data that can’t leave the company: a specialised open model installed on-site (with the hardware cost limits I discuss in the local AI lesson).

And one where it doesn’t: a chatbot that must answer about company procedures that change every month. That calls for RAG.

In practice

• First try a prompt with examples, then RAG: they solve most cases.
• Use fine-tuning to teach a behaviour (format, style, task), not to teach facts.
• Collect at least a few hundred correct, consistent examples, and keep some aside for testing.
• Consider the provider route first; an open model with LoRA pays off when you need control and confidentiality.
• Always measure against the base model with a good prompt, and repeat the comparison when new models come out.

Sources

• Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021)
• Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (2023)
• OpenAI — Model optimization (fine-tuning)
• Hugging Face — PEFT, parameter-efficient fine-tuning
• Unsloth — documentation for fine-tuning open models

Informational content, current as of the date shown. Prices, models and rules change often: always check official sources. Written with the support of AI tools and reviewed by the author.