Natural Language Processing: The Modern Stack

Runtime note: Hugging Face Transformers, sentence-transformers, FAISS, and spaCy don't run in the Pyodide sandbox. Execute these snippets in a local Python environment with pip install transformers sentence-transformers faiss-cpu spacy. CPU is fine for everything in this lesson except the BERT fine-tune.

Before 2018, most NLP systems were custom pipelines: tokenisers combined with handwritten rules, feeding LSTMs trained from scratch on a small labelled dataset. In 2026, NLP is mostly transfer learning (transfer) plus a handful of well-supported libraries. The skill that matters has shifted from "implement an attention mechanism" to "pick the right pretrained model and wire it into your product".

This lesson is a practical overview: the task taxonomy, the tokenisation choices, the Hugging Face ecosystem, the supporting libraries (spaCy, sentence-transformers), and the failure modes. It builds on dl-nlp, which covered the architecture side.

1. The NLP Task Taxonomy

Almost every NLP product fits one of six core task types. Recognising which one you have is a big part of the work.

The model architecture follows from the task shape. Encoder-only models (BERT, RoBERTa) suit classification, tagging, and retrieval. Decoder-only models (GPT, Llama) suit generation and structured extraction. Encoder-decoder models (T5, BART) suit translation and summarisation. These lines are less clear now that instruction-tuned LLMs are used for most generation, but matching the task shape to the architecture is still a useful guide.

2. Preprocessing: A Brief History

The progression is the same as in the rest of deep learning: hand-crafted rules → learned representations → bigger learned representations.

• Rule-based (pre-2013). Tokenise on whitespace, strip stopwords, stem with Porter, hand-curate feature lists. These systems broke easily and worked for only one language.
• Static embeddings (2013–2017). Word2Vec and GloVe. Every word gets one vector, learned from which words appear near it. "Bank" gets the same vector whether it's a river bank or a financial one.
• Contextual embeddings (2018–2020). ELMo, then BERT. The vector for a word depends on the sentence it is in. "Bank" near "river" looks different from "bank" near "deposit".
• Instruction-tuned LLMs (2021–today). The model produces text instead of a vector. The "representation" is a hidden state inside the model, and you usually don't inspect it.

You still preprocess in 2026 (lowercasing, deduplication, PII removal), but you no longer hand-design features. The tokenizer that ships with your model decides almost everything.

3. Tokenisation: Where Many Bugs Start

Tokenisers split text into integer IDs. Different model families use different schemes, and using the wrong tokeniser produces meaningless outputs without any error.

Subword tokenisers (BPE, WordPiece, SentencePiece) share one useful property: they can represent words the model never saw during training. "Crypto-currency-mania" becomes a sequence of known subwords. That's why most modern models use them. Byte-level BPE (used by GPT-2 and later GPT models) can encode any text. WordPiece can still produce an [UNK] token for a character that is not in its vocabulary, such as a rare emoji.

The practical rule: always load the tokenizer that comes with the model, and use it for both training and inference. AutoTokenizer.from_pretrained(model_name) is what makes this safe. Don't reuse a BERT tokenizer for a T5 model.

4. The Hugging Face Ecosystem

Hugging Face's transformers library is the standard library for applied NLP. It exposes thousands of pretrained models behind a stable API. Two entry points cover most usage:

• pipeline(): three-line inference for a named task. This is the quickest way to start.
• AutoModel + AutoTokenizer: full control for fine-tuning or non-standard workflows.

pipeline(): The Three-Line Path

from transformers import pipeline

# Sentiment: the default model is a small DistilBERT classifier
clf = pipeline("sentiment-analysis")
print(clf("This lesson is shorter than I expected."))
# [{'label': 'POSITIVE', 'score': 0.998}]

# Named entity recognition
ner = pipeline("ner", aggregation_strategy="simple")
print(ner("Grace works on PythonMastery in Bengaluru."))
# [{'entity_group': 'PER', 'word': 'Grace', ...},
#  {'entity_group': 'ORG', 'word': 'PythonMastery', ...},
#  {'entity_group': 'LOC', 'word': 'Bengaluru', ...}]

# Summarisation
summ = pipeline("summarization", model="facebook/bart-large-cnn")
print(summ("...long article...", max_length=60, min_length=20)[0]["summary_text"])

# Translation
en_fr = pipeline("translation_en_to_fr", model="Helsinki-NLP/opus-mt-en-fr")
print(en_fr("Transfer learning saves time.")[0]["translation_text"])

# Zero-shot classification: classify text without any training
zsc = pipeline("zero-shot-classification")
print(zsc(
    "The stock dropped 12% on weak earnings.",
    candidate_labels=["finance", "sports", "weather", "politics"],
))
# Returns ranked scores; "finance" wins.

Zero-shot classification needs a closer look. The model is a generic natural-language-inference classifier (typically BART-MNLI). For each candidate label, it scores "does this text entail the label?" and then picks the label with the highest score. You get a classifier for any label set with zero training data. Its best accuracy is lower than a fine-tuned model's, but for prototypes or rare-label tasks it works surprisingly well without any training loop.

AutoModel and AutoTokenizer: Custom Workflows

from transformers import AutoTokenizer, AutoModel
import torch

name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)

inputs = tokenizer("Hello, transformers.", return_tensors="pt", padding=True)
with torch.no_grad():
    outputs = model(**inputs)

# Last hidden state: one vector per token
print(outputs.last_hidden_state.shape)   # torch.Size([1, 7, 768])
# The vector of the [CLS] token, used to represent the whole sentence
print(outputs.last_hidden_state[:, 0].shape)   # torch.Size([1, 768])

You use this level of the API when fine-tuning, doing custom pooling, or extracting embeddings for downstream tasks.

5. Fine-Tuning BERT for Classification

The pattern from transfer applied to text:

from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    Trainer, TrainingArguments,
)
from datasets import load_dataset

raw = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, max_length=256)

ds = raw.map(tokenize, batched=True)
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2,
)

args = TrainingArguments(
    output_dir="imdb-distilbert",
    learning_rate=2e-5,                  # the usual learning rate for BERT-family models
    per_device_train_batch_size=16,
    num_train_epochs=3,
    eval_strategy="epoch",
    weight_decay=0.01,
    warmup_ratio=0.1,
)

Trainer(
    model=model, args=args,
    train_dataset=ds["train"], eval_dataset=ds["test"],
    tokenizer=tokenizer,
).train()

Three epochs of distilbert-base-uncased on IMDb reach about 92% accuracy on a single GPU in under twenty minutes. An LSTM trained from scratch on the same task usually scores several points lower.

max_length=256 is a deliberate truncation. BERT-family models have a 512-token cap. Truncating to 256 doubles throughput at small accuracy cost. For long documents you need a long-context model (Longformer, BigBird) or a sliding-window approach: split the document into overlapping chunks, score each chunk, and combine the scores.

6. Sequence-to-Sequence: T5 and BART

Any task that maps text to text (translation, summarisation, paraphrasing, style transfer, question generation) fits the encoder-decoder shape. T5 is known for framing every task this way: the input is "translate English to German: …" and the output is the translation.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

name = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSeq2SeqLM.from_pretrained(name)

prompt = "summarize: " + "Transfer learning reuses pretrained weights..."
ids = tokenizer(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, num_beams=4)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Fine-tuning works like the BERT case, but you use AutoModelForSeq2SeqLM and Seq2SeqTrainingArguments. Modern instruction-tuned LLMs (genai-how-llms-work) have replaced T5/BART in most general-purpose generation tasks, but for narrow seq2seq tasks where you have labelled pairs and want a small, fast model, T5/BART fine-tuning is still excellent.

7. Embeddings and Retrieval: sentence-transformers + FAISS

Classification answers "what is this?". Retrieval answers "what is this like?". For search, deduplication, clustering, recommendation, and the retrieval half of RAG (genai-rag), you need embeddings: a dense vector per document such that semantically similar texts have nearby vectors.

from sentence_transformers import SentenceTransformer
import numpy as np
import faiss

model = SentenceTransformer("all-MiniLM-L6-v2")    # 384-dimensional vectors, fast

docs = [
    "Transfer learning reuses pretrained model weights.",
    "FAISS is a library for fast similarity search.",
    "Cats are independent and aloof animals.",
    "Dogs love their owners unconditionally.",
]
embeddings = model.encode(docs, normalize_embeddings=True)

# Build a FAISS index over the embeddings
index = faiss.IndexFlatIP(embeddings.shape[1])     # inner product = cosine similarity for normalised vectors
index.add(np.asarray(embeddings, dtype="float32"))

query = model.encode(["pretrained networks"], normalize_embeddings=True)
scores, ids = index.search(np.asarray(query, dtype="float32"), k=2)
print([docs[i] for i in ids[0]])
# ['Transfer learning reuses pretrained model weights.',
#  'FAISS is a library for fast similarity search.']

all-MiniLM-L6-v2 is the most commonly used sentence embedder: 384 dimensions, fast enough to embed millions of documents on a laptop, accurate enough for most retrieval tasks. For higher accuracy at the cost of speed, all-mpnet-base-v2 (768-dim) is the standard upgrade.

For fewer than a million documents you often don't need FAISS: a NumPy dot product works fine. For larger corpora, FAISS provides the fast search algorithms (IVF, HNSW, PQ), so you don't have to write them. This pattern (embed → index → retrieve top-k) is the whole retrieval half of retrieval-augmented generation.

8. Modern Instruction-Tuned Models

The pipeline-API examples above use task-specific fine-tuned models. The same tasks can be solved by prompting a general-purpose LLM:

Classify the sentiment of this review as POSITIVE or NEGATIVE.
Review: "This lesson is shorter than I expected."
Sentiment:

For most off-the-shelf classification tasks, a competent instruction-tuned LLM (genai-how-llms-work) matches or beats a small fine-tuned encoder, with zero training. The trade-offs are predictable:

• LLM prompting: zero training, slow, expensive per call, opaque, easy to change behaviour
• Fine-tuned encoder: needs labelled data, fast inference, cheap at scale, deterministic, hard to change behaviour

A common approach in 2026: prompt an LLM for low-volume tasks or tasks that change often. Once the task is stable and the volume is high enough to justify the work, distil it into a small fine-tuned encoder.

9. spaCy: When You Want Traditional NLP

spaCy is a fast NLP pipeline library with many components already included. POS tagging, dependency parsing, NER, lemmatisation and sentence segmentation all run in one call, quickly, on a CPU.

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Grace shipped PythonMastery on May 14, 2026.")

for ent in doc.ents:
    print(ent.text, ent.label_)
# Grace PERSON
# PythonMastery ORG
# May 14, 2026 DATE

for token in doc:
    print(token.text, token.pos_, token.dep_, token.head.text)

Use spaCy when you need:

• Speed at scale: millions of documents, CPU only.
• Linguistic structure: dependency trees, lemmas, morphology. Transformers don't expose these as a first-class output.
• Rule-based additions: Matcher and PhraseMatcher for hybrid rule-plus-ML systems.
• Production pipelines: spaCy's Language object is serialisable, versionable, and works the same in training and serving.

You can use spaCy and Hugging Face together. spaCy's spacy-transformers integration lets you use a HF model as the encoder inside a spaCy pipeline.

Common Mistakes

1. Training a transformer from scratch on a small dataset.

To beat a fine-tuned pretrained model by training from scratch, you need millions of labelled examples. If your data is under a million examples, fine-tune. If it's under ten thousand, you may not need full fine-tuning at all: training only the head, or few-shot prompting, is often enough.

2. Ignoring sequence length limits.

BERT-family models cap at 512 tokens. Simple truncation drops everything after the first 512 tokens of every long document. If documents routinely exceed the limit, either chunk-and-aggregate, use a long-context model (Longformer, ModernBERT), or switch to an encoder-decoder that handles longer inputs.

3. Wrong tokenizer at inference.

You trained with bert-base-uncased, then loaded bert-base-cased's tokenizer at inference because of a copy-paste. The integer IDs no longer mean what the model expects, and the outputs are meaningless. Keep the model and the tokenizer together, and always load both from the same from_pretrained name.

4. Treating LLM output as ground truth.

A model that produces fluent English will produce fluent plausible English even when the underlying facts are wrong. For any user-facing system, verify the model's claims: check them against a database, against retrieved sources (genai-rag), or by routing high-stakes decisions to a human. "Hallucination" is not a bug that a patch can remove. A model trained to predict the next token will sometimes produce false text that sounds right.

5. Embedding without normalisation.

Cosine similarity is the inner product of unit vectors. If you forget normalize_embeddings=True and use an inner-product index, you're ranking by vector length as well as direction. That is usually wrong, and you get no error. Normalise both your corpus and your queries. After normalisation, L2 distance gives the same ranking as cosine similarity, so an L2 index also works. L2 distance on un-normalised vectors has the same problem: it also depends on vector length.

🎯 Your Turn: Zero-Shot Topic Classification

Use the Hugging Face pipeline("zero-shot-classification") to classify a list of news headlines into one of five topics, without training a model. Requirements:

• Load the zero-shot pipeline (the default model is fine. It is facebook/bart-large-mnli.)
• For each headline, return the top label and its score.
• Print results as a small table.

Headlines and candidate labels are given. Your task is to write the pipeline call and format the output.

Skeleton:

from transformers import pipeline

headlines = [
    "Federal Reserve raises rates 25 basis points amid sticky inflation",
    "Lakers defeat Celtics in overtime thriller",
    "New strain of avian flu detected in poultry farms",
    "Senate passes bipartisan infrastructure bill",
    "Hurricane Erin strengthens to Category 4 off the Atlantic coast",
]
candidate_labels = ["finance", "sports", "health", "politics", "weather"]

# TODO 1: build the zero-shot pipeline
classifier = ...

# TODO 2: loop over headlines, classify each against candidate_labels,
#         collect (headline, top_label, top_score)

# TODO 3: print a small table with the headline cut to 50 characters,
#         the label, and the score to 3 decimals

from transformers import pipeline

headlines = [
    "Federal Reserve raises rates 25 basis points amid sticky inflation",
    "Lakers defeat Celtics in overtime thriller",
    "New strain of avian flu detected in poultry farms",
    "Senate passes bipartisan infrastructure bill",
    "Hurricane Erin strengthens to Category 4 off the Atlantic coast",
]
candidate_labels = ["finance", "sports", "health", "politics", "weather"]

classifier = pipeline("zero-shot-classification")

rows = []
for h in headlines:
    result = classifier(h, candidate_labels=candidate_labels)
    rows.append((h, result["labels"][0], result["scores"][0]))

print(f"{'headline':<52} {'label':<10} {'score':>6}")
print("-" * 72)
for h, label, score in rows:
    print(f"{h[:50]:<52} {label:<10} {score:>6.3f}")

headline                                             label       score
------------------------------------------------------------------------
Federal Reserve raises rates 25 basis points amid  finance     0.954
Lakers defeat Celtics in overtime thriller         sports      0.991
New strain of avian flu detected in poultry farms  health      0.973
Senate passes bipartisan infrastructure bill       politics    0.985
Hurricane Erin strengthens to Category 4 off the   weather     0.987

What You Learned

• The NLP task taxonomy: classification, tagging, generation, retrieval, QA, structured extraction. The task shape decides the architecture.
• Preprocessing has moved from hand-crafted rules → static embeddings → contextual embeddings → instruction-tuned LLMs. You no longer hand-design features.
• Subword tokenisers (BPE, WordPiece, SentencePiece) almost remove the OOV problem, which is why most modern models use them. Always load the tokenizer that comes with the model.
• Hugging Face pipeline() gives you three-line inference for the standard tasks. AutoModel + AutoTokenizer is the fine-tuning layer.
• Fine-tuning BERT-family for classification: 2e-5 LR, 3 epochs, weight_decay=0.01, warmup. This recipe works well for most tasks.
• Embeddings + FAISS is the retrieval pattern. sentence-transformers/all-MiniLM-L6-v2 for speed, all-mpnet-base-v2 for accuracy. Always normalise for cosine.
• Zero-shot classification gets you a usable classifier with zero training data via an NLI model.
• spaCy works alongside transformers. Use it for fast pipelines, linguistic structure, and production-grade rule + ML hybrids.
• Verify LLM outputs. Fluent text is not always correct text.

Next: GANs & Generative Models: adversarial training, diffusion, and which generative approach is used for which task in 2026.