The Transformer Architecture

Every frontier LLM in 2026 (Claude, GPT, Gemini, Llama, Mistral) is a decoder-only transformer. The architecture was published in 2017, in a paper called "Attention Is All You Need". After nine years of scaling experiments, research into training methods and efficiency work, no one has produced anything clearly better. The same architecture that translated English to German in 2017 now writes production code, reasons over million-token contexts, and runs in small versions on phones.

This lesson explains the transformer from an engineer's point of view. It doesn't show how to implement one from scratch (that needs a separate lesson). It explains enough of the mechanism for you to reason about context limits, sampling parameters and the efficiency techniques that make modern inference fast.

The exercise runs right here in the browser. It samples next tokens at different temperatures with NumPy, so you need no API key.

1. Why Transformers Replaced RNNs

Before 2017, sequence models were RNNs (LSTMs, GRUs). They process one token at a time and pass a hidden state forward to the next step. They had two serious problems:

1. Cannot parallelise training. Token t needs token t-1's hidden state, which needs t-2's, and so on. Training is inherently sequential, which wastes the parallelism that GPUs give you.
2. Information decays over distance. By the time the hidden state has compressed 500 tokens of history into one fixed-size vector, very little information about the first 100 tokens is left.

The transformer fixes both at once. Self-attention lets every token look directly at every other token in the sequence, so there is no step-by-step bottleneck and no loss of information over distance. Training can run in parallel across the whole sequence, and this is what made the scaling of the last nine years possible.

The trade-off: attention is O(n²) in sequence length. We will come back to this.

2. Three Types: Encoder, Decoder, Encoder-Decoder

The original 2017 paper introduced an encoder-decoder model for translation. Since then, the architecture has split into three families:

Almost all generative AI today is decoder-only. The other two types are still in use (BERT-style encoders are still the right tool for embeddings, see the RAG lesson), but when someone says "LLM", they almost certainly mean a decoder-only model.

The rest of this lesson focuses on decoder-only.

3. The Attention Mechanism: The Basic Idea

For each token in the sequence, the model asks: which of the earlier tokens are relevant to me right now, and how much should I attend to each?

In detail: each token produces three vectors, using three learned linear projections of its embedding:

• Query (Q): "what am I looking for?"
• Key (K): "what do I offer?"
• Value (V): "what information will I pass on if another token attends to me?"

To compute the new representation of a token, you take its Q, dot-product it with every other token's K to get a score, softmax those scores into attention weights, then take a weighted sum of all the V vectors. The token now has a new representation, built from everything it found relevant in the rest of the sequence.

In a decoder, "every other token" is limited to earlier tokens only. This limit is the causal mask, and it is what makes the model generative. A token at position 5 can attend to positions 1-5, but it cannot see position 6.

4. Scaled Dot-Product Attention: The Formula

The exact operation, from the 2017 paper:

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

Every piece, in plain English:

• Q K^T: dot products between every query and every key. Shape: [seq_len, seq_len]. Each entry is a raw similarity score.
• / sqrt(d_k): divide by the square root of the key dimension. Without this, dot products grow with d_k, and softmax puts almost all the weight on one token, where gradients become close to zero. The scaling keeps the variance roughly constant.
• softmax(...): turn the scores into a probability distribution over earlier tokens. Each row sums to 1.
• ... V: a weighted sum of the value vectors. Each token gets a mixture of the other tokens' V vectors, weighted by attention.

That is one attention head. Modern models stack many of them in parallel.

5. Multi-Head Attention

One head computes one set of attention weights, for one kind of relationship. The model needs to track several kinds at once: syntactic dependencies, coreference, topical similarity, positional patterns. So you run many attention heads in parallel, each with its own Q, K, V projections, and concatenate the results.

A frontier model might have 64-128 attention heads per layer. Studies that examine individual heads have found that many of them learn patterns that people can understand: one head attends to the previous token, another to the syntactic head of a noun, another to recent verbs. The model picks up these "circuits" without being told to.

MultiHead(Q, K, V) = concat(head_1, ..., head_h) W_O

where head_i = Attention(Q W_Q_i, K W_K_i, V W_V_i)

The W_O at the end is a learned projection back to the model dimension.

6. The Full Transformer Block

A single transformer block combines attention and a feed-forward network, with residual connections and layer normalisation:

x = LayerNorm(x + MultiHeadAttention(x))
x = LayerNorm(x + FeedForward(x))

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def LayerNorm(*_a, **_kw):
    print('-> LayerNorm() called')
    return _AutoMock('LayerNorm()')
def MultiHeadAttention(*_a, **_kw):
    print('-> MultiHeadAttention() called')
    return _AutoMock('MultiHeadAttention()')
def FeedForward(*_a, **_kw):
    print('-> FeedForward() called')
    return _AutoMock('FeedForward()')

• Residual connections (x + ...) let gradients flow through deep stacks without vanishing. Without them, training a 100-layer transformer is practically impossible.
• LayerNorm keeps activations in a stable distribution across the network. Modern variants (RMSNorm, used in Llama and most 2024+ models) are slightly faster and work equally well.
• FeedForward is two linear layers with a non-linearity (GELU or SwiGLU). Most of the model's parameters are here: the hidden layer is typically 4× wider than the model dimension.

A frontier model stacks 80-120 of these blocks. Activations pass through each block and become more abstract at each step: early layers track surface patterns (tokens, syntax), middle layers track meaning, and late layers track features specific to the task.

7. Positional Encoding: How the Model Knows Order

Attention is permutation-invariant by default. If you shuffle the input tokens, the attention math produces the same outputs in a different order. That is a problem: "the dog bit the man" and "the man bit the dog" should not look identical.

The fix is to add position information to each token's embedding. There are four approaches, listed roughly in the order they appeared:

RoPE is the usual choice in 2026 because it extends to longer sequences better than the others. A model trained on 8K context can be made to handle 32K+ at inference time by interpolating the rotation frequencies. This is the main reason context windows grew so large so quickly.

8. Why Long Context Is Hard: O(n²)

Attention is quadratic in sequence length. Doubling context quadruples the compute and memory for the attention operation. For a 1M-token context, the raw attention matrix would be 1 trillion entries.

Three families of fixes make modern long context possible:

FlashAttention (Dao 2022): the attention math is the same, but the computation is reorganised so the full n×n matrix is never stored in memory. It is processed tile by tile, on the chip. ~2-4x speed-up, much lower memory.

Sliding-window attention: each token only attends to the last W tokens (e.g. 4K). O(n·W) instead of O(n²). Pairs well with periodic "global" tokens for long-range information. Mistral uses this.

Mixture-of-Experts (MoE): this is not about context length, but it is part of how modern models are made efficient. The feed-forward layer is replaced with N "expert" sub-networks, and a router picks k of them per token. Total parameters scale, but per-token compute does not. DeepSeek, Mixtral, and most 2025+ frontier models are MoE.

Together, these techniques make a million-token context window possible on a single inference machine. They don't guarantee good quality of attention over 1M tokens: researchers are still studying whether models can use that much context well.

9. The KV Cache: Why Inference Is Cheap

During generation, you produce one new token at a time. A simple implementation would recompute attention from scratch over the whole prefix for every new token. That wastes work, because for tokens 1 to t-1, the K and V vectors are exactly the same as in the previous step.

The KV cache stores K and V for every past token. To generate token t+1, you only compute the new K_t, V_t, append them to the cache, and run attention against the whole cache. Generation becomes O(n) per token instead of O(n²).

Practical consequences:

• The KV cache is stored in GPU memory, and at long contexts it is the largest memory cost during inference.
• Prompt caching (Anthropic, OpenAI and Google all offer it) is essentially "save the KV cache between requests so the same system prompt does not get re-encoded". In production it can reduce costs a lot. See production.
• Running a model locally without a KV cache (for example, in a custom inference loop with a bug) is much slower. Always check that the cache is used.

10. The Generation Loop

Putting it together, every modern LLM runs this loop at inference time:

# Decoder-only inference, pseudocode
def generate(prompt: str, max_tokens: int = 256) -> str:
    tokens = tokenise(prompt)
    kv_cache = []                                  # one (K, V) entry per layer

    # Prefill: process the whole prompt in parallel, fill the KV cache
    logits, kv_cache = model.forward(tokens, kv_cache)

    output = []
    for _ in range(max_tokens):
        # Sample next token from the last position's logits
        next_tok = sample(logits[-1], temperature=0.7, top_p=0.9)
        if next_tok == END_OF_TEXT:
            break
        output.append(next_tok)

        # Decode: process just the new token, append to KV cache
        logits, kv_cache = model.forward([next_tok], kv_cache)

    return detokenise(output)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def tokenise(*_a, **_kw):
    print('-> tokenise() called')
    return _AutoMock('tokenise()')
def detokenise(*_a, **_kw):
    print('-> detokenise() called')
    return _AutoMock('detokenise()')
model = _AutoMock('model')
def sample(*_a, **_kw):
    print('-> sample() called')
    return _AutoMock('sample()')
END_OF_TEXT = _AutoMock('END_OF_TEXT')

Two phases, two different cost profiles:

• Prefill (the whole prompt at once): limited by compute, runs in parallel, fast per token, but processes every input token once.
• Decode (one token at a time): limited by memory bandwidth and sequential. This is why output tokens cost 4-5x more than input.

Streaming responses (stream=True in the SDK) start emitting tokens as soon as the prefill is done. That is why a 30-token reply feels instant but a 3,000-token essay still takes seconds: most of the wait is decode.

11. Sampling: Temperature, top-p, top-k

sample(logits, ...) is where you control how random (or "creative") the output is. The raw logits become a distribution, and you choose how to draw from it.

Temperature divides the logits before softmax:

probs = softmax(logits / temperature)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def softmax(*_a, **_kw):
    print('-> softmax() called')
    return _AutoMock('softmax()')
logits = _AutoMock('logits')
temperature = _AutoMock('temperature')

• T = 0 → take the argmax (deterministic, "greedy"). Best for facts, code, classification.
• T = 0.7 → moderate randomness. Sensible default for assistants.
• T = 1.0 → use the model's raw distribution.
• T > 1 → a flatter distribution: more varied, less coherent.
• T = 2+ → mostly random text. Rarely useful.

Top-p (nucleus) sampling: keep only the smallest set of tokens whose cumulative probability is ≥ p, renormalise, sample from that. p = 0.9 is standard. This removes the long tail of very unlikely tokens, which can occasionally ruin an answer.

Top-k sampling: keep only the top k tokens by probability. Less precise than top-p, and rarely used alone in modern APIs.

Where these knobs are exposed (local models through Hugging Face generate(), and many hosted APIs), you set temperature and top_p, and leave top_k alone. Low temperature for repeatable output (testing, structured generation), higher for creative writing. temperature=0 makes output more repeatable, not guaranteed identical: hosted APIs don't promise the same text twice.

Current Claude models don't take them at all. Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 reject temperature, top_p and top_k with a 400 error. You steer them with the prompt instead (and with the effort setting for how much they think). The sampling mechanism above still runs inside them, but you just can't change its settings.

Stop tokens (or stop sequences): extra strings that stop the loop when the model generates them. They are useful when a model tends to keep writing after the real answer.

12. Multi-Modal Models: A Short Note

Modern frontier models handle more than text. Vision transformers (ViT) cut an image into a grid of patches, treat each patch as a token, and run the same attention mechanism. CLIP trained a vision encoder and a text encoder together, so that an image's embedding and its caption's embedding end up close to each other in vector space. Many text-to-image and image search systems are built on this idea. Frontier multi-modal models (Claude, GPT-4o, Gemini) combine vision-encoder outputs with the text tokens, so the LLM can "read" images directly. The architecture is the same transformer, but some of the tokens come from pixels.

Common Mistakes

1. Thinking attention "understands"
Attention weights tell you what the model is looking at, not why. A head that consistently attends to the previous token has not learned grammar. It has learned a useful statistic. Interpretability research has many examples of attention patterns that looked meaningful but turned out to work differently from the obvious human explanation. Use attention visualisations to investigate problems, not as an explanation of the model's reasoning.

2. Assuming a bigger context window helps if your prompt is poorly structured
A 1M-token context does not fix a poorly organised prompt. Models show a "lost in the middle" effect: they use information in the middle of a long context less reliably than information at the beginning or end. Put critical instructions at the start and end. Structure long context with clear sections (XML tags help, see prompt engineering). Don't add 500K tokens of unimportant text and expect the model to find the one fact that matters.

3. Not using the KV cache when running models locally
If you write your own inference loop with transformers and forget past_key_values, you re-encode the entire prefix on every generated token. A 200-token response then takes much longer than it should. Always pass and update the KV cache, or use model.generate(...) which handles it for you.

4. Misreading temperature
temperature=0 is not "more accurate". It is more deterministic. For some tasks (generating ideas, brainstorming) it makes results worse, because the model keeps repeating the same choices. temperature=1 is not "broken". It is the model's own distribution. Choose the value based on the task.

5. Ignoring tokenisation when counting "characters"
"My prompt is 5,000 words, so it should fit in an 8K context!" But code, rare words and many non-English languages need many more tokens per word than ordinary English text. Always measure with the actual tokeniser (the anthropic SDK has client.messages.count_tokens(...)).

🎯 Your Turn: Temperature Comparison

Current Claude models don't let you set temperature, but you can see the mechanism on its own. Here is a toy model's next-word scores after "The detective lit a":

tokens = ["cigarette", "match", "candle", "fuse", "torch"]
logits = [3.0, 2.2, 1.0, 0.3, -0.5]

Write sample_with_temperature(logits, tokens, temps=(0.0, 0.7, 1.0, 2.0), n=1000, seed=0) that:

• for each temperature, draws n next tokens from softmax(logits / T) (at T = 0, always take the top token),
• counts how often each token was picked,
• returns a dict mapping each temperature to {token: count},
• prints the counts as a small table, one row per temperature.

import numpy as np


def sample_with_temperature(logits, tokens, temps=(0.0, 0.7, 1.0, 2.0), n=1000, seed=0):
    # TODO 1: turn logits into a NumPy array and make a seeded generator: np.random.default_rng(seed)
    # TODO 2: for each t in temps: at t == 0 pick logits.argmax() every time,
    #         otherwise softmax(logits / t) and draw n indices with rng.choice(..., p=probs)
    # TODO 3: count the picks per token (np.bincount with minlength=len(tokens))
    # TODO 4: print one row per temperature, then return {t: {token: count}}
    ...


tokens = ["cigarette", "match", "candle", "fuse", "torch"]
logits = [3.0, 2.2, 1.0, 0.3, -0.5]
results = sample_with_temperature(logits, tokens)

import numpy as np


def sample_with_temperature(logits, tokens, temps=(0.0, 0.7, 1.0, 2.0), n=1000, seed=0):
    """Draw n next tokens at each temperature; return {temperature: {token: count}}."""
    logits = np.asarray(logits, dtype=float)
    rng = np.random.default_rng(seed)
    results = {}

    for t in temps:
        if t == 0:
            picks = np.full(n, logits.argmax())          # greedy: always the top token
        else:
            z = logits / t
            probs = np.exp(z - z.max())                  # subtract the max so exp() can't overflow
            probs /= probs.sum()
            picks = rng.choice(len(tokens), size=n, p=probs)
        counts = np.bincount(picks, minlength=len(tokens))
        results[t] = {tok: int(c) for tok, c in zip(tokens, counts)}

    print("T      " + "".join(f"{tok:>11}" for tok in tokens))
    for t, counts in results.items():
        print(f"{t:<7}" + "".join(f"{counts[tok]:>11}" for tok in tokens))
    return results


tokens = ["cigarette", "match", "candle", "fuse", "torch"]
logits = [3.0, 2.2, 1.0, 0.3, -0.5]       # the model's raw scores for the next word
results = sample_with_temperature(logits, tokens)

T        cigarette      match     candle       fuse      torch
0.0           1000          0          0          0          0
0.7            693        249         36         18          4
1.0            624        244         72         42         18
2.0            409        278        137        117         59

What You Learned

• Decoder-only transformers (Claude, GPT, Llama) are the architecture of every modern LLM.
• Self-attention lets every token directly attend to every earlier token: softmax(Q K^T / sqrt(d_k)) V.
• Multi-head attention runs many attention heads in parallel, and each learns different relationships.
• A transformer block: attention → add & norm → feed-forward → add & norm. Stack 80-120 of them.
• Positional encoding (sinusoidal, learned, RoPE) tells the model token order. RoPE is the 2026 default.
• Attention is O(n²) in sequence length, so long contexts are hard. FlashAttention, sliding-window, MoE, and KV caching make modern inference fast.
• The KV cache turns generation from O(n²) per token into O(n) per token. It is the dominant memory cost at long contexts and the basis of prompt caching.
• Prefill vs decode: prefill is parallel and fast per token, and decode is sequential and slow. This is why output tokens cost ~4-5x more.
• Temperature controls determinism vs creativity, and top-p trims the long tail. Choose them carefully where you can set them. Current Claude models don't accept them, so with Claude you steer the output with the prompt.
• Multi-modal models reuse the same architecture: images become patch tokens, combined with text tokens.

Next: Prompt Engineering, the techniques that turn a basic API call into a useful, structured and reliable feature.