Fine-tuning & Training LLMs

Engineers who are new to LLMs often think: "the model doesn't quite do what I want, so I should fine-tune it." This is almost always the wrong first step. In 2026, the order of operations is prompt engineering → RAG → fine-tuning, and most teams never reach the third step. Fine-tuning is more expensive, less flexible and harder to debug than the alternatives. But for a small set of problems, it is the right tool.

This lesson helps you tell the difference. It covers when to fine-tune and when not to, how the modern parameter-efficient methods (LoRA, QLoRA) work, what a real training pipeline looks like, and why "I want the model to learn these new facts" means you should use RAG instead.

Run locally on a GPU machine with pip install transformers peft datasets bitsandbytes accelerate. Training requires a CUDA GPU (24 GB+ VRAM for the QLoRA example with a 7B model). Inference of fine-tuned adapters works on smaller hardware. The code samples are examples, and full runs take 30 minutes to several hours.

1. The 2026 Decision Tree

Need the model to behave differently?
  │
  ├─ Does it lack information?         → RAG (retrieve facts, don't train them in)
  ├─ Does it use the wrong format?     → Prompt engineering, few-shot examples
  ├─ Is the style/voice off?           → Fine-tune (style is learnable)
  ├─ Same task at lower latency/cost?  → Fine-tune a smaller model
  ├─ Proprietary patterns it's never   → Fine-tune
  │  seen in training?
  └─ Adversarial robustness?           → Often fine-tune + careful data curation

The two things fine-tuning is good for in 2026:

1. Consistent style/format/voice: your brand tone, a specific output schema, refusal patterns. Style is what fine-tuning teaches best.
2. Cheap models for one narrow task: fine-tuning a 7B open-weights model on your task may match a frontier API at 1/50th of the cost. This is worth it once your volume is high enough.

The thing fine-tuning is bad for: teaching new facts. Models do not reliably absorb factual knowledge during fine-tuning the way they do during pretraining. You will spend days training, and the model will still sometimes get the facts wrong. You also can't update those facts without training again. For knowledge, RAG is better in every way.

2. The Spectrum of "Training"

"Fine-tuning" is one point on a wider spectrum:

In practice, you spend most of your time on the first two and very little on LoRA. The bottom rows of the table are there so you can see the full range.

3. LoRA: The Key Idea

LoRA (Low-Rank Adaptation, Hu et al. 2021) is the method that made fine-tuning practical for individual developers. The idea, in one sentence: instead of updating a giant weight matrix W, learn a small low-rank update BA and add it.

Original layer:           y = W x                  (for example, W is 4096 × 4096)
LoRA-adapted layer:       y = W x + (B A) x        (A is r×4096, B is 4096×r, r is small)

Where:
  W is FROZEN: the original pretrained weights stay unchanged.
  A and B are TRAINABLE, but they are tiny.
  r is the "rank". It is usually 8, 16, or 64.

Why it works: experiments show that the change a fine-tune needs to make is itself low-rank. You don't need to change every weight in W. You only need changes in a small subspace. BA is exactly that low-rank subspace, learned during training.

The difference is large. A 7B-parameter model has ~7B weights. A LoRA adapter on it (rank 16, applied to the attention projections) has ~10M trainable parameters, roughly 700× fewer. Training needs much less GPU memory, you save small adapter files (~40 MB) instead of a full copy of the model (~14 GB), and you can serve dozens of fine-tunes on top of one shared base model.

QLoRA (Dettmers et al. 2023) goes one step further: load the base model in 4-bit quantisation (saving 4× memory), then train LoRA adapters in higher precision on top. The 4-bit quantisation is good enough that quality barely suffers, and you can now fine-tune a 70B model on a single 48 GB GPU instead of needing a small cluster.

4. Hugging Face PEFT: The Library

The PEFT (Parameter-Efficient Fine-Tuning) library implements LoRA, QLoRA and similar methods on top of transformers. The shape of a real LoRA training script:

# For illustration only. It needs a CUDA GPU and the libraries below.
# pip install transformers peft datasets bitsandbytes accelerate trl
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

BASE = "mistralai/Mistral-7B-Instruct-v0.3"

# 1. Load the base model in 4-bit (QLoRA)
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)
tokenizer = AutoTokenizer.from_pretrained(BASE)

# 2. Wrap with LoRA adapters
lora = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 7,254,790,144 || trainable%: 0.188

# 3. Load your dataset (JSONL with chat-format messages)
ds = load_dataset("json", data_files="my_training_data.jsonl")

# 4. Train
trainer = SFTTrainer(
    model=model,
    train_dataset=ds["train"],
    args=SFTConfig(
        output_dir="./lora-out",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        save_steps=200,
    ),
)
trainer.train()

# 5. Save just the adapter (tiny file)
model.save_pretrained("./my-adapter")

The whole script is about 30 lines of code that connects the libraries. The libraries do the hard work. Most of your real work happens before this step: getting the dataset right.

5. The Dataset: Format and Quality

Modern fine-tuning datasets are almost always in chat format, one example per line of a JSONL file:

{"messages": [
  {"role": "system", "content": "You convert SQL to plain English."},
  {"role": "user", "content": "SELECT name FROM users WHERE active = 1;"},
  {"role": "assistant", "content": "Names of all active users."}
]}
{"messages": [
  {"role": "system", "content": "You convert SQL to plain English."},
  {"role": "user", "content": "SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days';"},
  {"role": "assistant", "content": "Number of orders placed in the last seven days."}
]}

The trainer reads the messages, applies the model's chat template to render them as a single string, and trains the model to produce the assistant message given the system + user messages.

Quality matters much more than quantity. A well-known Meta paper ("LIMA", 2023) showed that 1,000 carefully chosen examples beat 50,000 noisy ones for instruction tuning. Teams keep learning this again: spend your time checking 500 examples by hand instead of scraping 50K average ones.

Practical dataset rules:

• Diversity: cover the full range of inputs you expect. Don't have all examples about one sub-topic.
• Format consistency: every assistant response in the dataset must follow the format you want at inference time. Mixed formats teach the model nothing useful.
• No bad examples: one example with a wrong answer can take 10 correct examples to undo. Review the data by hand before training.
• Match your production inputs: if production inputs are short and direct, your training examples shouldn't all be long and chatty.

For RLHF-style preference data (used in DPO/IPO, the modern alternative to PPO-based RLHF), the format adds a "chosen" and "rejected" pair per prompt. Out of scope for this lesson. See the trl library's docs.

6. Cost: Fine-tuned 7B vs Frontier API

Here is a concrete trade-off. You want a classifier for support tickets, 1M requests/month, ~500 input tokens, ~50 output tokens each.

Option A: Frontier API (Sonnet-tier):

• 1_000_000 × (500 × 2 + 50 × 10) / 1_000_000 = $1,500/month


• No infrastructure. No training. Quality is excellent. You can't control the latency, because it depends on the API.

Option B: Fine-tuned Mistral-7B on your own GPU:

• One-off training cost: ~$50-200 (a few hours on a rented H100 or A100)


• Inference: a single A10G or L40S handles ~5 req/sec → 13M reqs/month at one GPU, ~$300-500/month


• You own the deployment, the security model, the latency


• Quality may match or be lower, so you need to evaluate it

The numbers cross over somewhere around 100K-500K requests/month depending on model tier and task complexity. Below that volume, fine-tuning is rarely worth the effort. Above 10M requests/month, it almost always is.

The other axis: privacy. If you cannot send the data to a third-party API at all (HIPAA, financial, defence), open-weights models with fine-tuning are the only option. Then cost is less important.

7. API-Based Fine-tuning: The Easier Path

You don't always need to manage the training yourself. Several providers offer fine-tuning as a service:

• OpenAI fine-tuning API: upload a JSONL file, and you get a fine-tuned model deployed automatically. It mostly uses LoRA internally. It supports gpt-4o-mini and similar models, but access to frontier models is restricted.
• Anthropic: fine-tuning available for select models, via Bedrock/Vertex AI partners (often behind enterprise access).
• Google Vertex AI: fine-tune Gemini variants through their console.
• Together / Fireworks / Anyscale / Replicate: hosted fine-tuning and inference for open-weights models. The easiest way to fine-tune Llama or Mistral.
• Hugging Face AutoTrain: point-and-click fine-tuning, including QLoRA, with deployment.

The trade-off: you have less control over hyperparameters and the vendor keeps your dataset, but you do much less infrastructure work. For a first fine-tune, the API path is usually right. Once you know the data is good and the approach works, you can move to open-weights models.

8. Evaluation: Did Fine-tuning Help?

Before you deploy any fine-tune, you must answer one question: does it beat the base model + your best prompt? Often it does not. Either the dataset wasn't good enough, or the base model was already good enough with prompting.

Build an eval set (50-500 held-out test cases) and run both the base model with your best prompt and the fine-tuned model against it. Compare:

Track:

• Quality on each metric


• Latency (a 7B fine-tune is often faster than a frontier API)


• Cost per request


• Robustness on adversarial/out-of-distribution inputs

A fine-tune that improves quality but handles adversarial inputs worse can cause serious problems in production. Test it on many kinds of input.

9. Catastrophic Forgetting

Fine-tune a model heavily on one narrow task and it can lose general capability. A model that is excellent at "SQL → English" may no longer hold a normal conversation, write code, or follow a basic instruction outside its training data. This is catastrophic forgetting.

Three mitigations:

1. Don't over-train. LoRA on 1-3 epochs usually keeps the base capabilities intact. Watch validation loss, and stop when it plateaus.
2. Mix in general data. Add some general instruction-following examples (10-30%) to your dataset, so the model keeps practising general assistant behaviour while it learns your specific task.
3. Lower learning rate, lower rank. Both reduce how much the adapter can drift from base behaviour.

Keep adapters small and weight changes small. If you need very heavy training to get the task right, the task is probably not suited to fine-tuning. Go back to prompting or RAG.

10. RLHF: A Short Summary

RLHF (Reinforcement Learning from Human Feedback) turns an instruction-tuned model into a helpful, harmless and honest assistant that people want to talk to. The original RLHF pipeline: train a reward model on human preference pairs (response_A vs response_B, which did the human prefer?), then optimise the LLM with PPO to maximise that reward. It is expensive, hard to get right, and not something individual developers do. DPO (Direct Preference Optimisation, Rafailov et al. 2023) is a much simpler alternative: it skips the reward model and trains directly on preference pairs. It has largely replaced PPO-based RLHF for open-source fine-tuning. The trl library implements both. It is worth knowing that they exist, but you will rarely need them.

Common Mistakes

1. Fine-tuning to add facts
This is the most common mistake. "I want the model to know our product catalogue" → fine-tune it. Three weeks later, the model sometimes invents products that don't exist, and the catalogue has changed, so you would need to train again. RAG would have taken a day, and it updates when you re-index the catalogue.

2. Training on tiny noisy datasets
50 examples collected quickly over a weekend produce an over-fit model that memorises your training set and fails on anything slightly different. Either build a small, high-quality set (200-1000 examples) carefully, or use a larger dataset that you have filtered carefully (10K+).

3. Not having a held-out test set
Training loss going down means the model is learning something. It does not mean it generalises. Always hold out 10-20% of your data (or a separately collected eval set) and never train on it.

4. Deploying without comparing to the base model
The mandatory baseline is: same base model, your best zero/few-shot prompt. If the fine-tune doesn't clearly beat that on your eval set, you have spent days and money for nothing.

5. Skipping the chat template
Each model has its own chat template (special tokens like [INST], <|im_start|>, <|user|>). The trainer applies it automatically when you use SFTTrainer with chat-format messages. But if you build prompts by hand and the template doesn't match, the model won't respond properly at inference time. Let the library apply the template.

6. Saving the full model instead of the adapter
A LoRA adapter is ~40 MB. Saving the full merged model is ~14 GB. Save the adapter and load it on top of the base model at inference time. This saves disk space and bandwidth.

7. Ignoring catastrophic forgetting
A fine-tuned model that no longer handles general conversation is unusable in production unless your only call to it is the narrow trained task. Test the fine-tune on out-of-domain inputs as part of your eval suite.

🎯 Your Turn: Build a SQL-to-English Training Set

Write a small JSONL training file (10 examples) for a "convert SQL query to plain English" fine-tune, and describe the LoRA training command you would run.

Requirements:

• 10 examples, one per line of sql_to_english.jsonl
• Each example is a {"messages": [...]} object with a system, user, and assistant message
• System prompt: clear role + instruction
• User content: a SQL query
• Assistant content: a plain-English paraphrase, 1-2 sentences
• Cover a variety of queries: SELECT with WHERE, JOIN, COUNT/aggregate, GROUP BY, INSERT, UPDATE. Don't make all 10 look the same

Then write a train.sh (or describe in comments) that would launch QLoRA fine-tuning on Mistral-7B using the trl library.

# Skeleton: write the data to sql_to_english.jsonl and document the training command.
import json

SYSTEM = "You convert SQL queries into a one-sentence plain-English description. Be precise. Do not add explanations."

# TODO 1: list 10 (sql, english) tuples covering varied query types
# TODO 2: write each to a JSONL line as {"messages": [system, user, assistant]}
# TODO 3: write the conceptual training command

EXAMPLES = [
    # ("SELECT name FROM users WHERE active = 1;", "Names of all active users."),
    # ... fill in
]

# Build a tiny SQL-to-English fine-tuning dataset and document the LoRA command.
import json

SYSTEM = (
    "You convert SQL queries into a one-sentence plain-English description. "
    "Be precise. Do not add explanations."
)

EXAMPLES = [
    # Simple SELECT
    ("SELECT name FROM users;",
     "Names of all users."),
    # SELECT with WHERE
    ("SELECT name FROM users WHERE active = 1;",
     "Names of all active users."),
    # SELECT with multiple conditions
    ("SELECT email FROM users WHERE country = 'UK' AND signup_date > '2025-01-01';",
     "Emails of UK users who signed up after 1 January 2025."),
    # JOIN
    ("SELECT u.name, o.total FROM users u JOIN orders o ON o.user_id = u.id;",
     "Each user's name alongside their order totals."),
    # Aggregate
    ("SELECT COUNT(*) FROM orders WHERE status = 'pending';",
     "The number of pending orders."),
    # GROUP BY
    ("SELECT country, COUNT(*) FROM users GROUP BY country;",
     "The number of users in each country."),
    # ORDER BY + LIMIT
    ("SELECT product, sales FROM stats ORDER BY sales DESC LIMIT 5;",
     "The top five products by sales."),
    # HAVING (slightly trickier)
    ("SELECT customer_id, SUM(total) FROM orders GROUP BY customer_id HAVING SUM(total) > 1000;",
     "Customer IDs whose total order value exceeds 1000."),
    # INSERT
    ("INSERT INTO users (name, email) VALUES ('Bob', 's@example.com');",
     "Add a new user named Bob with email s@example.com."),
    # UPDATE
    ("UPDATE products SET price = price * 1.1 WHERE category = 'electronics';",
     "Increase the price of all electronics products by 10%."),
]

with open("sql_to_english.jsonl", "w", encoding="utf-8") as f:
    for sql, english in EXAMPLES:
        record = {
            "messages": [
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": sql},
                {"role": "assistant", "content": english},
            ]
        }
        f.write(json.dumps(record) + "\n")

print(f"wrote {len(EXAMPLES)} examples to sql_to_english.jsonl")

# Sanity-check by reading it back
with open("sql_to_english.jsonl", encoding="utf-8") as f:
    for line in f:
        obj = json.loads(line)
        assert len(obj["messages"]) == 3
print("validation: all examples parse and have 3 messages each")


# ───────── Training command (conceptual) ─────────
# Requires a CUDA GPU (24 GB+ VRAM for QLoRA on Mistral-7B). Run from the shell:
#
#   pip install transformers peft datasets bitsandbytes accelerate trl
#
#   python -c "
#   from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
#   from peft import LoraConfig, prepare_model_for_kbit_training
#   from trl import SFTTrainer, SFTConfig
#   from datasets import load_dataset
#
#   BASE = 'mistralai/Mistral-7B-Instruct-v0.3'
#   bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4',
#                            bnb_4bit_compute_dtype='bfloat16')
#   model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map='auto')
#   model = prepare_model_for_kbit_training(model)
#   tokenizer = AutoTokenizer.from_pretrained(BASE)
#
#   lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias='none',
#                     task_type='CAUSAL_LM',
#                     target_modules=['q_proj','k_proj','v_proj','o_proj'])
#
#   ds = load_dataset('json', data_files='sql_to_english.jsonl', split='train')
#
#   trainer = SFTTrainer(
#       model=model, train_dataset=ds, peft_config=lora,
#       args=SFTConfig(output_dir='./sql-en-adapter', num_train_epochs=3,
#                      per_device_train_batch_size=2, gradient_accumulation_steps=4,
#                      learning_rate=2e-4, bf16=True, logging_steps=1, save_steps=10),
#   )
#   trainer.train()
#   trainer.save_model('./sql-en-adapter')
#   "
#
# Result: a ~40 MB LoRA adapter in ./sql-en-adapter that you load on top of
# the base Mistral-7B-Instruct model at inference time.

What You Learned

• Default to prompts and RAG. Fine-tune only when style consistency, latency/cost on a narrow task, or proprietary patterns demand it.
• Don't fine-tune to add facts. Use RAG. Models absorb facts unreliably during fine-tuning, and you lose the ability to update.
• LoRA adds small low-rank adapter matrices (BA) next to the frozen base weights: 100-1000× fewer trainable parameters than full fine-tuning.
• QLoRA quantises the frozen base to 4-bit and trains LoRA on top. This makes it possible to fine-tune a 70B-parameter model on a single 48 GB GPU.
• Dataset format: JSONL of {"messages": [system, user, assistant]}. Quality and diversity beat quantity.
• Hugging Face PEFT + TRL is the standard open-source stack: LoraConfig, SFTTrainer, save the adapter.
• API-based fine-tuning (OpenAI, Vertex AI, Together) is the easiest way to start: less control, much less infrastructure.
• Always evaluate against the base model + best prompt before deploying. Often, that baseline wins.
• Catastrophic forgetting is a real risk: keep training light, mix in general data, and evaluate on out-of-domain inputs.
• RLHF / DPO are the next layer for shaping preferences. trl implements both, but most teams never need them.

Next: Building Production AI Applications: turning a working prototype into something you can run at scale, with latency, caching, evaluation pipelines, observability and basic security.