How Large Language Models Work

Without the marketing, a large language model is a single mathematical object: a function that takes a sequence of tokens and returns a probability distribution over the next token. Everything else (conversation, code generation, summarisation, tool use) is that one operation, run in a loop, using a very large set of learned statistics.

Once you understand this, the rest of the LLM stack becomes an engineering problem you can reason about. You can explain why models hallucinate, why prompts matter so much, why context windows have a limit, and why a 70B-parameter model costs what it does to run. This lesson covers this mental model, the three training stages that produce a modern assistant, what models can and can't do in 2026, and your first API call from code.

Run locally with pip install anthropic and ANTHROPIC_API_KEY set. LLM APIs require keys and network access, so they do not run in the in-browser sandbox. Expected output is shown in comments.

1. An LLM Is P(next_token | previous_tokens)

Forget "AI" for a moment. The core object is a probability distribution:

P(next_token | token_1, token_2, ..., token_n)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def P(*_a, **_kw):
    print('-> P() called')
    return _AutoMock('P()')
token_2 = _AutoMock('token_2')
token_n = _AutoMock('token_n')
next_token = _AutoMock('next_token')
token_1 = _AutoMock('token_1')

Given everything seen so far, what is the probability of every possible next token? The model outputs one number per token in its vocabulary (typically 50K-200K tokens). The code picks one token (usually by sampling), appends it, and repeats. That is the entire generation loop.

# Pseudocode for the generation loop every modern LLM runs
tokens = tokenise(prompt)
while not done:
    logits = model(tokens)              # shape: [vocab_size]
    probs  = softmax(logits)
    next_tok = sample(probs)            # temperature, top-p and top-k are applied here
    tokens.append(next_tok)
    if next_tok == END_OF_TEXT: break

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def tokenise(*_a, **_kw):
    print('-> tokenise() called')
    return _AutoMock('tokenise()')
prompt = _AutoMock('prompt')
done = 1
def model(*_a, **_kw):
    print('-> model() called')
    return _AutoMock('model()')
def softmax(*_a, **_kw):
    print('-> softmax() called')
    return _AutoMock('softmax()')
def sample(*_a, **_kw):
    print('-> sample() called')
    return _AutoMock('sample()')
END_OF_TEXT = _AutoMock('END_OF_TEXT')

The model itself (matrices and non-linear functions with billions of parameters) is a learned approximation of that conditional distribution. Everything useful it does (reasoning, code, dialogue) depends on how good that approximation is on text it has never seen.

A token is roughly a piece of a word: "hello" is one token, "unimaginable" might be three. English averages around four characters per token. Pricing, context limits, and rate limits all count tokens, not words. See tokenisation from the NLP lesson.

2. The Training Corpus: Where the Statistics Come From

A frontier model in 2026 is trained on something like 15-30 trillion tokens. The blend (no provider publishes exact numbers, but the rough shape is public):

• Web crawl (Common Crawl, filtered): most of the data
• Books, papers, code (GitHub, arXiv, Stack Exchange)
• Curated high-quality text: Wikipedia, textbooks, reference works
• Synthetic data: text generated by models to fill gaps, used more and more
• Licensed data: news, private collections of text, paid partnerships

The model never "looks anything up". Whatever it knows is compressed into its weights during pretraining. After training stops, the model learns nothing new. The date when its training data ends is the training cutoff.

3. The Three Training Stages

A model that has only been pretrained can finish your sentences, but it cannot follow an instruction. Producing a useful assistant takes three stages:

Pretraining is where the bulk of the capability comes from. To predict the next token across a large part of the internet, the model has to learn grammar, facts, programming languages, basic reasoning patterns and how languages relate to each other. This is what makes it useful at all.

Instruction tuning (or supervised fine-tuning, SFT) changes what the model tries to do. After pretraining, asked "what is 2+2?", it might continue with "Solve the following equation: 3x+5=14" because that pattern occurs in textbooks. SFT teaches it that a question deserves an answer.

RLHF (Reinforcement Learning from Human Feedback): humans (or another model, in "RLAIF") choose which of two responses is better, and the model is fine-tuned to produce the preferred kind of response. This stage shapes helpfulness, honesty and when the model refuses. It also produces most of the model's "personality".

4. Scale: The Numbers That Matter

Scale has three axes, each measured in different units:

A parameter is a single floating-point number in the model's weights. More parameters give the model more capacity to store patterns and combine them. Modern Mixture-of-Experts (MoE) models have trillions of total parameters but only activate a fraction per token, so inference cost scales with the active count, not the total.

The context window is the maximum number of tokens the model can attend to at once. Larger windows let you include long documents, codebases and conversation history. The compute cost grows roughly quadratically with the window size. The transformer lesson explains why.

5. Emergent Abilities, and a Correction

The "emergent abilities" claim (Wei et al. 2022) was that certain capabilities, such as multi-step arithmetic, instruction following and chain-of-thought reasoning, appear suddenly once a model crosses a size threshold, instead of improving smoothly. This claim got a lot of attention.

The current view: a follow-up paper (Schaeffer et al. 2023) showed that "emergence" often disappears when you use a smoother metric. A task scored as "completely wrong / completely right" looks discontinuous, but the same task scored on partial credit improves steadily with scale. The capabilities are real, but the suddenness was partly caused by the way they were measured.

In practice: bigger models are reliably better, but there is no specific size at which "general intelligence" appears. Don't pay for Opus when Haiku does the job, and don't expect a 7B local model to match a frontier API.

6. Capabilities in 2026

A frontier LLM today can plausibly:

• Write idiomatic code in any mainstream language, given a clear spec
• Reason through multi-step problems if prompted to think step by step
• Use tools (calling functions you describe) and chain them together
• Read images, charts, and short videos (multi-modal frontier models)
• Maintain coherent conversation across 100K+ tokens of context
• Translate between most language pairs at near-professional quality
• Summarise, restructure, and critique long-form text

What it still cannot reliably do:

• Know recent events beyond its training cutoff (without a tool to look them up)
• Do exact arithmetic beyond a few digits (use a calculator tool)
• Reason about completely new problems that are unlike anything in its training data
• Judge its own uncertainty correctly: it can sound confident when it is wrong
• Remember anything across sessions unless you give it a memory system

7. Hallucinations: Why They Happen

Hallucination is the model generating fluent, plausible text that is factually wrong. It is not a bug that can be fixed with a patch. It is a direct result of how the model works.

The model picks the most probable next token given the prompt. If you ask "What was Hedy's PhD thesis about?", the training data contains no facts about Hedy, but it contains millions of patterns of "X's PhD thesis was about Y". The model fills in something plausible, because nothing in its training signal taught it to say "I don't know" when the answer is missing.

Mitigations:

• RAG (a later lesson): give the model the facts in the prompt


• Tool use: let it call a calculator, a search engine, a database


• Prompt design: tell it directly "say 'I don't know' if uncertain"


• Verification: check the output with code when accuracy matters

You can't remove hallucination completely. You can only reduce it. Designing around this fact is a large part of building production LLM systems.

8. The Two-API World

You will use LLMs from one of two ecosystems:

For most application work, a frontier API is the right default: better quality per dollar at low or medium volume, and no infrastructure to run. For high-volume, privacy-sensitive or heavily fine-tuned workloads, open-weights models are the better choice. Many teams use both: frontier models where quality matters most, and open-weights models for large amounts of cheap work.

This lesson series uses Anthropic's Claude API as its main example. The same patterns work with OpenAI's API, Google's Gemini and any OpenAI-compatible local server (vLLM, llama.cpp, Ollama).

9. Your First API Call

Install the SDK and put your key in an environment variable. Never put it in your code or in git. See envconfig.

pip install anthropic
export ANTHROPIC_API_KEY="sk-ant-..."     # or set in your shell profile

The minimal call:

# Run this locally. It needs ANTHROPIC_API_KEY in your environment.
from anthropic import Anthropic

client = Anthropic()        # reads ANTHROPIC_API_KEY from the environment

resp = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    messages=[{"role": "user", "content": "In one sentence, what is a token?"}],
)
print(resp.content[0].text)
# A token is a sub-word unit (typically a few characters) that a language
# model treats as the smallest piece of text it reads and emits.

resp is a structured object, not a string. The text is in resp.content[0].text, because the response can contain several content blocks (text, tool calls, images). resp.usage.input_tokens and resp.usage.output_tokens tell you exactly what you were charged for, so log them.

10. The Cost Model: A Rough Estimate

Pricing is per token, with output tokens roughly 4-5x more expensive than input across most providers (output is sequential and cannot be batched as cheaply). Anthropic's list prices as of September 2026, per million tokens:

These numbers go out of date quickly, because they have fallen every year so far. Check the provider's pricing page before you plan a budget. The ratios are what to remember.

A rough estimate for a chatbot:

• Average input: 1,000 tokens (system prompt + history + user message)


• Average output: 300 tokens


• 10,000 chats/day on Sonnet: 10_000 * (1000 * 2 + 300 * 10) / 1_000_000 = $50/day = ~$1,500/month


• Same workload on Haiku: ~$750/month. On Opus: ~$3,750/month. On Fable: ~$7,500/month.

Choosing the right model can change your cost by 10x. That is less than it used to be (in 2024 the top tier cost 60x the cheapest), but it is still the biggest single cost factor you control. Two more settings can cut costs further: prompt caching bills repeated input at a tenth of the price, and the batch API halves everything for work that can wait. We will return to cost optimisation in production AI applications.

Common Mistakes

1. Treating LLM output as ground truth
The model writes fluently, but it doesn't know everything. When correctness matters (facts, calculations, code that runs in production), check the output. Run the code, check the citation, parse the JSON. Treat the output as a draft, not as the source of truth.

2. Not handling rate limits and transient errors
API calls fail. The SDK raises anthropic.RateLimitError, anthropic.APITimeoutError, anthropic.APIConnectionError. Wrap calls in retry-with-backoff (see APIs for the general pattern) or use the SDK's built-in max_retries.

3. Ignoring token costs until the bill arrives
Long system prompts multiplied by high volume give unexpectedly large bills. Log resp.usage on every call. Set a daily spend alert with your provider. Test at small volume before you start a large job.

4. Hardcoding API keys
A key committed to git stays in the git history, even if you git rm it five minutes later, so treat it as leaked. Always read keys from environment variables. If you accidentally commit one, rotate it (replace it with a new key) immediately. Deleting the line is not enough.

5. Assuming the model "knows" things it does not
Because of the training cutoff, anything after that date is a hallucination risk. If you ask the model what year it is, the answer may be two years out of date. For current information, use search tools or RAG.

🎯 Your Turn: Build simple_chat

Write a function simple_chat(question, model="claude-opus-4-7") that:

• reads the API key from the ANTHROPIC_API_KEY env var (raise a clear error if missing),
• sends question as a single user message,
• returns the assistant's text response as a plain string,
• catches anthropic.APIError and re-raises as RuntimeError with a useful message,
• prints the input and output token counts (so the caller can see what the call costs).

# Run locally with `pip install anthropic` and ANTHROPIC_API_KEY set.
import os
from anthropic import Anthropic, APIError

def simple_chat(question: str, model: str = "claude-opus-4-7") -> str:
    # TODO 1: check ANTHROPIC_API_KEY is set; raise RuntimeError if not
    # TODO 2: create the Anthropic() client
    # TODO 3: call client.messages.create with model, max_tokens=1024, and the question
    # TODO 4: catch APIError and re-raise as RuntimeError("LLM call failed: ...") from e
    # TODO 5: print "used: <in_tok> in / <out_tok> out"
    # TODO 6: return resp.content[0].text
    ...

if __name__ == "__main__":
    print(simple_chat("Give me one sentence on why transformers replaced RNNs."))

# Run locally with `pip install anthropic` and ANTHROPIC_API_KEY set.
import os
from anthropic import Anthropic, APIError


def simple_chat(question: str, model: str = "claude-opus-4-7") -> str:
    """Send a one-shot question to Claude and return the text response."""
    if not os.environ.get("ANTHROPIC_API_KEY"):
        raise RuntimeError("set ANTHROPIC_API_KEY before calling simple_chat")

    client = Anthropic()
    try:
        resp = client.messages.create(
            model=model,
            max_tokens=1024,
            messages=[{"role": "user", "content": question}],
        )
    except APIError as e:
        raise RuntimeError(f"LLM call failed: {e}") from e

    print(f"used: {resp.usage.input_tokens} in / {resp.usage.output_tokens} out")
    return resp.content[0].text


if __name__ == "__main__":
    print(simple_chat("Give me one sentence on why transformers replaced RNNs."))

# Example output:
#   used: 19 in / 38 out
#   Transformers replaced RNNs because their self-attention mechanism processes
#   all tokens in parallel, which allows much longer effective context and
#   dramatically better scaling on modern GPUs.

import os  # noqa: F401
os.environ.setdefault("ANTHROPIC_API_KEY", "example-anthropic-api-key")

What You Learned

• An LLM is a function P(next_token | previous_tokens). Generation is sampling from that distribution in a loop.
• Tokens are sub-word units. Pricing, context limits, and rate limits all count tokens, not words.
• Three training stages: pretraining (raw text, where capability comes from), instruction tuning (follow prompts), RLHF (match human preferences).
• Scale matters along three axes: parameters, training tokens, context window. More is reliably better, but no specific size produces general intelligence.
• "Emergent abilities" are real, but their suddenness was partly caused by how they were measured. Capability grows smoothly with scale.
• Hallucinations are part of how next-token prediction works. You can reduce them, but you can't remove them.
• Two ecosystems: frontier APIs (Anthropic/OpenAI/Google) vs open-weights (Llama, Mistral, Qwen). Use both where they fit.
• Output tokens cost ~4-5x more than input. The choice of model tier changes cost by up to 10x, and caching and batching reduce it further.
• Always set keys in env vars, log token usage, wrap API errors, and retry transient failures.

Next: The Transformer Architecture, how the function P(next_token | previous_tokens) is implemented, with enough detail to reason about context windows, sampling and efficiency techniques.