Retrieval Augmented Generation (RAG)

An LLM does not know your company's documentation, your customer's order history, the contents of the PDF a user just uploaded, or anything published after its training cutoff. RAG (Retrieval Augmented Generation) gives the model this information without retraining it. You search a knowledge base for the chunks relevant to the user's question, paste them into the prompt, and ask the LLM to answer using those chunks.

The pattern is simple. Doing it well, with production-level quality, latency and cost, is one of the hardest parts of applied LLM work. This lesson covers a working architecture, the ways RAG fails, and the smallest RAG system you can build, in about fifty lines of Python.

Run locally with pip install anthropic sentence-transformers numpy and ANTHROPIC_API_KEY set. Expected outputs are shown in comments. sentence-transformers downloads a model on first run (~130 MB, and the reranker later in the lesson is ~1.1 GB).

1. The Problem RAG Solves

An LLM is a frozen statistical model. It cannot:

• Know facts more recent than its training cutoff
• Know your facts: internal docs, user data, private content
• Reliably cite sources for what it says

Three workarounds exist:

RAG is the best choice for almost every real "the LLM needs to know X" problem. It is easy to update (re-index when the docs change), cheap (no training) and citable (you know exactly which chunks were used).

2. The Pipeline: Five Steps

RAG, in five stages. Every production system follows this shape, and the differences are in how each stage is implemented.

flowchart LR
    A[Documents] --> B[Chunk]
    B --> C[Embed]
    C --> D[(Vector DB)]
    Q[User query] --> E[Embed query]
    E --> F[Retrieve top-k]
    D --> F
    F --> G[Build prompt]
    G --> H[LLM]
    H --> R[Answer + citations]

1. Chunk documents into passages small enough to embed meaningfully and big enough to carry context.
2. Embed each chunk: turn the text into a fixed-size vector that represents its meaning.
3. Index the vectors in a database that supports fast nearest-neighbour search.
4. At query time: embed the question, retrieve the top-k nearest chunks.
5. Build the prompt with the retrieved chunks as context, send to the LLM.

Each stage has its own failure modes. Bad chunks → no relevant text can be retrieved. Bad embeddings → the wrong chunks are retrieved. Bad prompt → the model ignores the context. This lesson covers all three.

3. Embeddings: Text to Vectors

An embedding model maps a piece of text to a fixed-length vector (typically 384, 768, 1024, or 1536 dimensions). The positions of the vectors have meaning: texts about similar topics end up near each other, and unrelated texts end up far apart.

Three families of embeddings, roughly in order of cost:

The most widely used open-source library is sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")        # 384-dimensional vectors, about 130 MB
vec = model.encode("Python is a programming language.")
print(vec.shape)        # (384,)

Run a sentence through model.encode() and you get back a numpy array. Run a thousand sentences in a list and you get a (1000, 384) matrix. That is the basic operation.

Similarity is measured with cosine similarity: the dot product of the two vectors after scaling them to unit length. Higher means more similar.

import numpy as np
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Most embedding models output normalised vectors (unit length), so cosine similarity is just a dot product. It is faster and gives the same answer.

4. Chunking: The First Real Problem

You cannot embed an entire 50-page PDF as one vector: too much information is squeezed into 384 numbers, and similarity stops being meaningful. So you split into chunks.

Overlap is especially important: if a relevant sentence falls exactly at a chunk boundary, neither chunk has the full context. A 10-20% overlap between adjacent chunks fixes this. Cost: slightly more chunks to embed and store.

For structured documents (markdown, code), split on natural boundaries (headings, function definitions) rather than character counts. The structure carries meaning, and fixed-size chunking loses it.

5. Vector Databases: A Quick Overview

Once you have thousands or millions of embedding vectors, you need fast nearest-neighbour search. Options range from "a numpy array" to "managed cloud service":

For most new production apps, Chroma or Qdrant (run locally in Docker) is enough. You typically don't need managed cloud until you're at hundreds of millions of vectors or need cross-region replication.

All of them implement Approximate Nearest Neighbour (ANN) algorithms (HNSW, IVF, ScaNN). These give up a very small amount of recall in exchange for being much faster than exact search. Exact cosine over 10M vectors is ~seconds, and ANN is ~milliseconds.

6. A Minimal RAG in 40 Lines

To see how the parts fit together, here is the smallest useful RAG, with no vector DB, only numpy:

# Run locally with `pip install sentence-transformers anthropic numpy`.
import numpy as np
from sentence_transformers import SentenceTransformer
from anthropic import Anthropic

# 1. Tiny knowledge base (5 chunks)
CHUNKS = [
    "Python is a high-level, interpreted programming language created by Guido van Rossum in 1991.",
    "Python's official package manager is pip. The Python Package Index (PyPI) hosts over 500,000 packages.",
    "Decorators in Python are functions that modify other functions, denoted with the @ symbol.",
    "Python uses indentation, not braces, to define blocks of code. Standard is 4 spaces.",
    "The Global Interpreter Lock (GIL) means CPython only runs one thread of Python bytecode at a time.",
]

# 2. Embed the chunks once
embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
chunk_vecs = embedder.encode(CHUNKS, normalize_embeddings=True)    # (5, 384)

def retrieve(query: str, k: int = 3) -> list[str]:
    """Return the top-k most similar chunks to the query."""
    q = embedder.encode(query, normalize_embeddings=True)          # (384,)
    scores = chunk_vecs @ q                                        # dot product = cosine (the vectors are normalised)
    top_idx = np.argsort(scores)[::-1][:k]
    return [CHUNKS[i] for i in top_idx]

def answer(query: str) -> str:
    context = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(retrieve(query)))
    prompt = (
        f"Answer the question using ONLY the context below. If the answer "
        f"is not in the context, say 'I don't know'. Cite chunks by their [N].\n\n"
        f"<context>\n{context}\n</context>\n\n"
        f"<question>{query}</question>"
    )
    client = Anthropic()
    resp = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.content[0].text

print(answer("Who created Python and when?"))
# Python was created by Guido van Rossum in 1991. [1]

That is forty lines, with real retrieval, a real LLM and real citations. Replace CHUNKS with the chunks of a PDF, and you have something you could release. The rest of this lesson is about making it good.

7. Retrieval Quality Decides Everything

This is the most important point: if you retrieve the wrong chunks, the LLM cannot give a good answer. No prompt can fix the fact that the relevant passage was never in the context. RAG quality depends mostly on retrieval quality, which means embedding quality, chunking strategy and ranking.

When a RAG system works but gives average results, the biggest improvement usually comes from better retrieval, not better generation.

7.1 Hybrid Retrieval: Vector + Lexical

Dense embeddings are great at semantic similarity but bad at proper nouns, error codes, and exact-string matches. "Error E_INVALID_TOKEN" might not embed near a chunk that contains that exact string. Add BM25 (a word-matching relevance score from the 1990s), and you get the strengths of both methods:

from rank_bm25 import BM25Okapi

# At index time
tokenised = [c.lower().split() for c in CHUNKS]
bm25 = BM25Okapi(tokenised)

# At query time
def hybrid_retrieve(query: str, k: int = 5, alpha: float = 0.5):
    q_vec = embedder.encode(query, normalize_embeddings=True)
    vec_scores = chunk_vecs @ q_vec                           # in [-1, 1]
    bm25_scores = bm25.get_scores(query.lower().split())
    bm25_norm = bm25_scores / (bm25_scores.max() + 1e-9)      # a rough normalisation to [0, 1]
    combined = alpha * vec_scores + (1 - alpha) * bm25_norm
    top_idx = np.argsort(combined)[::-1][:k]
    return [CHUNKS[i] for i in top_idx]

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

CHUNKS = _AutoMock('CHUNKS')
chunk_vecs = _AutoMock('chunk_vecs')
embedder = _AutoMock('embedder')
np = _AutoMock('np')

alpha=0.5 is a sensible starting point. Adjust it using your eval set.

7.2 Re-Ranking: Retrieve Many, Keep Few

A second-stage re-ranker improves quality a lot, with the same number of retrieved chunks. The pattern:

1. First stage (cheap, fast): retrieve top 20-50 candidates with vector / BM25.
2. Second stage (expensive, accurate): score each candidate with a cross-encoder that takes (query, chunk) as a pair and outputs a relevance score.
3. Keep the top 3-5 after re-ranking, send to the LLM.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")             # ~1.1 GB

def rerank(query: str, candidates: list[str], k: int = 5):
    pairs = [[query, c] for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:k]]

Why it works: a bi-encoder (embedding model) has to encode query and chunk separately, so neither sees the other. A cross-encoder runs them together through one transformer, so it can judge relevance much more precisely. It is slower for each pair, but you only run it on 50 candidates, not on the whole corpus.

This pattern (a fast first stage, then a slow and accurate second stage) is standard in search systems. RAG uses it for the same reasons.

8. The RAG Prompt Template

A workable template that handles the common cases:

Use ONLY the context below to answer the question. If the answer is not
in the context, say "I don't have that information in the provided sources".
Cite the source chunks by their [N] identifier wherever you make a factual claim.

<context>
[1] {chunk_1}

[2] {chunk_2}

[3] {chunk_3}
</context>

<question>
{user_question}
</question>

The key elements:

• "ONLY": discourages the model from using its training data, which is where hallucinations come from.
• "If not in the context, say...": gives the model a clear alternative. Without this, it often makes something up.
• Cite by [N]: produces output you can check. You can render the citations as links to the source documents.
• XML tags: keep context and question clearly separate, so the model does not confuse them.

Always include chunk IDs. Always require citations. Always show the citations to the user (or at least show them when debugging). A citation that links to the source document turns "AI said X" into "AI said X, and here is exactly where it came from in your docs".

9. Evaluating RAG: Three Metrics

Generic LLM eval (does the answer look good?) is not enough for RAG. You need to separate retrieval failures from generation failures:

A retrieval-recall@5 of, say, 0.7 means the LLM can't answer 30% of queries, however good the prompt is. Fix retrieval first.

Frameworks like Ragas and TruLens automate these metrics. Set them up before you have many users.

10. The Lost-in-the-Middle Problem

Long context windows don't mean the model uses every part of the context equally well. Liu et al. (2023) showed that information in the middle of a long context is recalled significantly less reliably than information at the beginning or end. A relevant chunk in the middle of a 50-chunk context may be almost ignored.

Implications for RAG:

• Keep retrieved context tight. 5 great chunks beat 50 mediocre ones.
• Order matters. Put the most relevant chunks at the start or end of the context block, not in the middle.
• Re-ranking helps in two ways: a better top-k, and the best chunk ends up in a strong position.

Modern frontier models have improved on this, but the effect has not disappeared. Don't put everything you have into the context and expect the model to find what matters.

Common Mistakes

1. Chunks too large or too small
Chunks of a few hundred tokens usually work best. The ~500-token default from the chunking table above is about 2,000 characters of English text. Chunks of 50 characters lose context, and chunks of 10,000 characters make embedding similarity less precise (one vector cannot represent ten different ideas well).

2. Not deduplicating retrieved chunks
If your corpus has near-duplicates (mirrored docs, identical FAQs across products), the top-5 might all be variants of the same passage. The LLM sees one piece of information five times and nothing else. Run a similarity check on retrieved chunks and drop near-duplicates before building the prompt.

3. Skipping chunk overlap
Without overlap, key sentences at chunk boundaries are split across two chunks, and neither has the full context. A 10-20% overlap is cheap and noticeably improves recall.

4. Not testing retrieval separately from generation
"My RAG is bad" almost always means one of: retrieval is bad, prompting is bad, or both. Build a retrieval eval set first (questions + ground-truth chunks). Fix recall@k before tuning prompts.

5. Thinking RAG solves hallucination
RAG reduces hallucination by giving the model facts to use. It does not remove it. The model can still:

• Misread a chunk and produce a wrong answer


• Combine information from multiple chunks incorrectly


• Add plausible-sounding details not in any chunk


• Cite the wrong chunk for a correct fact

Verification (LLM-as-judge, human review on samples, citation-checking automation) is still needed.

6. Embedding the wrong text
If you embed source code and search it with English questions, results may be poor: the embedding model was trained on natural language, not code. Use an embedding model trained for code search (for example voyage-code-3) or embed natural-language descriptions alongside the code.

🎯 Your Turn: Build a Five-Chunk RAG

Write rag_answer(query, chunks) that:

• takes a list of text chunks and a query,
• embeds the chunks (use sentence-transformers model "BAAI/bge-small-en-v1.5"),
• embeds the query,
• returns the top-3 chunks ranked by cosine similarity (don't call the LLM, because this exercise is only about retrieval),
• returns them as a list of (score, chunk) tuples, highest score first.

Use the provided five-chunk knowledge base. Verify on a few queries.

# Run locally with `pip install sentence-transformers numpy`.
import numpy as np
from sentence_transformers import SentenceTransformer

CHUNKS = [
    "The capital of France is Paris, known for the Eiffel Tower.",
    "Python is a programming language created by Guido van Rossum.",
    "Mount Everest is the tallest mountain in the world, at 8,849 metres.",
    "The Pacific Ocean is the largest ocean, covering about 30% of Earth's surface.",
    "JavaScript runs in every browser and is one of the most popular languages.",
]

def rag_answer(query: str, chunks: list[str], k: int = 3) -> list[tuple[float, str]]:
    # TODO 1: load the embedding model "BAAI/bge-small-en-v1.5"
    # TODO 2: embed chunks with normalize_embeddings=True to get a matrix of shape (N, 384)
    # TODO 3: embed query with normalize_embeddings=True to get a vector of shape (384,)
    # TODO 4: scores = matrix @ vector (since normalised, this is cosine similarity)
    # TODO 5: get top-k indices (descending)
    # TODO 6: return [(float(scores[i]), chunks[i]) for i in top_idx]
    ...

if __name__ == "__main__":
    for score, chunk in rag_answer("Tell me about programming languages", CHUNKS):
        print(f"{score:.3f}  {chunk}")

# Run locally with `pip install sentence-transformers numpy`.
import numpy as np
from sentence_transformers import SentenceTransformer

CHUNKS = [
    "The capital of France is Paris, known for the Eiffel Tower.",
    "Python is a programming language created by Guido van Rossum.",
    "Mount Everest is the tallest mountain in the world, at 8,849 metres.",
    "The Pacific Ocean is the largest ocean, covering about 30% of Earth's surface.",
    "JavaScript runs in every browser and is one of the most popular languages.",
]

# Load once at module level, because the model takes about 1 second to load
_embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")


def rag_answer(query: str, chunks: list[str], k: int = 3) -> list[tuple[float, str]]:
    """Return the top-k chunks ranked by cosine similarity to the query."""
    chunk_vecs = _embedder.encode(chunks, normalize_embeddings=True)       # (N, 384)
    q_vec = _embedder.encode(query, normalize_embeddings=True)             # (384,)
    scores = chunk_vecs @ q_vec                                            # (N,)
    top_idx = np.argsort(scores)[::-1][:k]
    return [(float(scores[i]), chunks[i]) for i in top_idx]


if __name__ == "__main__":
    print("Query: 'Tell me about programming languages'")
    for score, chunk in rag_answer("Tell me about programming languages", CHUNKS):
        print(f"  {score:.3f}  {chunk}")

    print("\nQuery: 'How high is Everest?'")
    for score, chunk in rag_answer("How high is Everest?", CHUNKS):
        print(f"  {score:.3f}  {chunk}")

# Example output:
#   Query: 'Tell me about programming languages'
#     0.789  JavaScript runs in every browser and is one of the most popular languages.
#     0.762  Python is a programming language created by Guido van Rossum.
#     0.421  The Pacific Ocean is the largest ocean, covering about 30% of Earth's surface.
#
#   Query: 'How high is Everest?'
#     0.881  Mount Everest is the tallest mountain in the world, at 8,849 metres.
#     0.376  The Pacific Ocean is the largest ocean, covering about 30% of Earth's surface.
#     0.339  The capital of France is Paris, known for the Eiffel Tower.

What You Learned

• RAG connects an LLM's fixed knowledge to your data: retrieve relevant chunks, paste them into the prompt, generate.
• The five-stage pipeline: chunk → embed → index → retrieve → prompt.
• Embeddings map text to vectors, and cosine similarity (or dot product on normalised vectors) measures relevance.
• Chunking is a separate decision: fixed-size, paragraph, semantic, hierarchical, with overlap. If you get this wrong, nothing else helps.
• Vector databases range from in-memory numpy to managed cloud services. Pick the simplest one that fits your scale.
• Retrieval quality decides most of RAG quality. Hybrid retrieval (vector + BM25), re-ranking with a cross-encoder, and tight top-k all matter.
• A working RAG prompt uses XML tags, says "ONLY use this context", offers a fallback, and demands citations.
• Evaluate retrieval separately from generation: recall@k, faithfulness, relevance.
• Lost-in-the-middle is real: keep context tight, put the best chunks at the edges.
• RAG reduces hallucination, but it does not remove it. Verification still matters.

Next: Fine-tuning & Training LLMs: what to do when prompts and RAG aren't enough, when fine-tuning is the right tool, and when (usually) it is not.