Working with Text: NLP Basics

A neural network only accepts numbers as input. Text is made of symbols. Every NLP system is therefore a two-step problem: turn text into numbers (preprocessing + vectorisation), then learn from those numbers (model). Once you understand the first step, the second step is almost the same as in any other deep learning task.

This lesson covers the classical pipeline (tokenise → vocab → embed → pool → classify), shows it end-to-end in Keras, and shows how transformers have changed the field.

Run in Colab or with pip install tensorflow. Outputs are shown inline.

1. The Two Halves of NLP

"the movie was great"  →  [4, 18, 7, 92]  →  embeddings  →  pooled vector  →  prediction
        text             integer IDs       dense vectors    fixed-size       such as positive

Half 1, representation: tokenise, build a vocabulary, map each token to an ID, look up a learned dense vector for that ID.

Half 2, modelling: a normal neural network operating on those dense vectors.

Most NLP bugs are in half 1.

2. Preprocessing: The Pipeline

A typical preprocessing pipeline before vectorisation:

In modern deep learning, the trend is to do less preprocessing. An embedding layer can learn that "running" and "run" are related, so you don't need to merge them by hand. The one step you must always do is tokenisation.

3. Tokenisation: Three Types

Classical Keras pipelines use word tokenisation. Transformer models (BERT, GPT, T5) all use subword tokenisation. For your first NN text classifier, word-level via TextVectorization is fine.

4. Vocabulary and Integer IDs

A vocabulary is a dictionary that maps token strings to integer IDs:

vocab = {
    "[PAD]":  0,        # padding to fixed length
    "[UNK]":  1,        # out-of-vocabulary token
    "the":    2,
    "and":    3,
    "movie":  4,
    "great":  5,
    ...
}

text = "the movie was great"
ids  = [2, 4, 1, 5]     # "was" is not in the vocabulary, so it becomes 1

Two special IDs you almost always need: a padding token (so sequences in a batch can be the same length) and an unknown token (for words not in the training vocabulary).

5. Embeddings: Words as Dense Vectors

A single integer ID tells the model nothing about meaning. An embedding layer maps each ID to a dense learned vector (for example, 64 numbers long) that does encode meaning.

"king"   →  [0.12, -0.47,  0.83, ...,  0.05]
"queen"  →  [0.10, -0.45,  0.81, ..., -0.02]
"taco"   →  [0.78,  0.91, -0.12, ...,  0.66]

Two important properties of trained embeddings:

• Similar words are close to each other. "king" and "queen" end up close because they appear in similar contexts during training.
• Direction has meaning. Famous example: "king" − "man" + "woman" ≈ "queen" (in word2vec-style embeddings).

In Keras, the embedding layer is a learnable lookup table:

from tensorflow.keras.layers import Embedding

# input_dim = vocab size, output_dim = embedding dimension
emb = Embedding(input_dim=10000, output_dim=64)

Input shape (batch, seq_len) of integer IDs → output shape (batch, seq_len, 64).

6. The Classical Baselines: Don't Skip Them

Before you use a neural network, try two simpler methods. They are often better than you would expect:

• Bag-of-Words: represent a document as the count of each vocabulary word. Word order is ignored.
• TF-IDF: weight those counts by how rare the word is across the corpus. Common words get a lower weight.

Plug either into a LogisticRegression from scikit-learn. On small text classification datasets (under ~10k documents) this often beats a small neural network. Neural networks start to do better with more data and longer documents.

Always run a TF-IDF + logistic regression baseline before you spend time on deep models. If TF-IDF with logistic regression gets 88% and your neural network gets 89% with ten times the training time, the simpler model is the right choice.

7. The Keras Text Pipeline

Modern Keras gives you TextVectorization, a preprocessing layer that lives inside the model. So the same preprocessing runs at training time and at inference time, and your training scripts and production code can't drift apart.

import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential

# A small toy corpus
texts = ["the movie was great", "boring and slow", "loved every minute"]

vectorize = layers.TextVectorization(max_tokens=10000,
                                     output_mode='int',
                                     output_sequence_length=100)

# adapt() learns the vocabulary from the data
vectorize.adapt(texts)

# Test it
print(vectorize(["the movie was great"]))
# → tf.Tensor([[3 5 2 8 0 0 ... 0]], shape=(1, 100), dtype=int64)

adapt() reads the training texts to build the vocabulary. After that, the layer always turns the same text into the same integers.

8. A Minimal Text Classifier

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (TextVectorization, Embedding,
                                     GlobalAveragePooling1D, Dense, Dropout)

VOCAB_SIZE = 10000
SEQ_LEN    = 100
EMB_DIM    = 64

vectorize = TextVectorization(max_tokens=VOCAB_SIZE,
                              output_mode='int',
                              output_sequence_length=SEQ_LEN)
vectorize.adapt(X_train_text)         # X_train_text: a list/array of strings

model = Sequential([
    vectorize,
    Embedding(VOCAB_SIZE, EMB_DIM),
    GlobalAveragePooling1D(),
    Dropout(0.3),
    Dense(64, activation='relu'),
    Dense(1,  activation='sigmoid'),
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

model.fit(X_train_text, y_train,
          epochs=10,
          batch_size=32,
          validation_split=0.2)
# → Epoch 10/10  loss: 0.21 - accuracy: 0.92 - val_loss: 0.30 - val_accuracy: 0.87

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

X_train_text = _AutoMock('X_train_text')
y_train = _AutoMock('y_train')

Read it:

1. TextVectorization turns each string into a (SEQ_LEN,) vector of integer IDs.
2. Embedding turns each ID into a 64-dim vector. Output shape: (SEQ_LEN, 64).
3. GlobalAveragePooling1D averages the 64-dim vectors across the sequence, collapsing to a single (64,) vector per document.
4. Two Dense layers: a standard binary classifier on top.

Why this works: the embeddings learn to push positive-sentiment words to similar directions. Averaging them gives a "sentiment direction" for the whole document. The Dense layers project that direction onto a yes/no decision.

9. Beyond Average Pooling: RNNs

Averaging throws away word order. "not good" and "good not" produce identical pooled vectors. For tasks where order matters (translation, summarisation), you need a model that processes the sequence in order.

This is what RNNs / LSTMs / GRUs do. These networks keep a hidden state and update it one token at a time. Briefly:

from tensorflow.keras.layers import LSTM

Sequential([
    vectorize,
    Embedding(VOCAB_SIZE, EMB_DIM),
    LSTM(64),                          # processes the sequence left-to-right
    Dense(1, activation='sigmoid'),
])

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')
vectorize = _AutoMock('vectorize')
def Embedding(*_a, **_kw):
    print('-> Embedding() called')
    return _AutoMock('Embedding()')
VOCAB_SIZE = _AutoMock('VOCAB_SIZE')
EMB_DIM = _AutoMock('EMB_DIM')
def Dense(*_a, **_kw):
    print('-> Dense() called')
    return _AutoMock('Dense()')

LSTMs were the dominant NLP architecture from ~2014 to ~2018. They still work, but transformers have mostly replaced them. They are covered fully in rnn.

10. Transformers: One Paragraph

In 2017 the paper "Attention Is All You Need" introduced the transformer, a model that processes a whole sequence in parallel using self-attention (each token decides how much to pay attention to every other token). Transformers scale much better than RNNs, and they handle relationships between distant words directly. Most modern foundation models, including BERT, GPT, T5, LLaMA and Claude, are transformers. We cover them in genai-transformer.

11. Hugging Face: Ready-Made Models

Today you will rarely train a sentiment classifier from scratch. Hugging Face's transformers library gives you access to most important pretrained models:

from transformers import pipeline

sentiment = pipeline("sentiment-analysis")

sentiment("I absolutely loved this film.")
# → [{'label': 'POSITIVE', 'score': 0.9998}]

sentiment("Two hours I will never get back.")
# → [{'label': 'NEGATIVE', 'score': 0.9994}]

Three lines, good accuracy, no training. For most real text tasks, the right starting point is a pretrained transformer fine-tuned on your data, not a new embedding-pool-Dense model. The pipeline above is just the easiest way to start.

Still, learning the pipeline in Section 8 first helps you understand what happens inside those pretrained models.

Common Mistakes

• Stopword removal on tasks where stopwords matter. Sentiment ("not good", "never worked") and negation tasks fail without stopwords. If you remove "not" in a sentiment task, many training examples end up meaning the opposite of their label.
• Inconsistent preprocessing between train and inference. This is a classic cause of "works on my laptop, fails in production". TextVectorization inside the model prevents this, so use it.
• Vocab too large for embedding dim. A vocab of 1,000,000 with 8-dim embeddings has too little room to tell that many words apart. There is no exact formula. Word vectors commonly have 50 to 300 dimensions (the published GloVe vectors come in 50, 100, 200 and 300). Treat the size as a hyperparameter and tune it.
• Sequence length too short. Truncating reviews to 50 tokens when the average review is 300 throws away most of the signal. Plot a histogram of token counts and pick a length that covers most of the distribution.
• Skipping the TF-IDF baseline. Before you commit to a deep model, check whether logistic regression already solves your problem.

🎯 Your Turn: Sentiment Classifier

Assume you have train_texts (list of strings) and train_labels (0 = negative, 1 = positive). Build a binary text classifier: TextVectorization (vocab 5000, seq length 80) → Embedding(5000, 32) → GlobalAveragePooling1D → Dense(32, relu) → Dense(1, sigmoid).

Adapt the vectorizer on the training texts, compile, and fit for 10 epochs.

Skeleton:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (TextVectorization, Embedding,
                                     GlobalAveragePooling1D, Dense)

# TODO 1: create TextVectorization (5000 tokens, sequence length 80)
vectorize = ...

# TODO 2: adapt it on train_texts

model = Sequential([
    # TODO 3: vectorize
    # TODO 4: Embedding(5000, 32)
    # TODO 5: GlobalAveragePooling1D
    # TODO 6: Dense(32, relu)
    # TODO 7: Dense(1, sigmoid)
])

# TODO 8: compile (adam, binary_crossentropy, accuracy)
# TODO 9: fit for 10 epochs with validation_split=0.2

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (TextVectorization, Embedding,
                                     GlobalAveragePooling1D, Dense)

vectorize = TextVectorization(max_tokens=5000,
                              output_mode='int',
                              output_sequence_length=80)
vectorize.adapt(train_texts)

model = Sequential([
    vectorize,
    Embedding(5000, 32),
    GlobalAveragePooling1D(),
    Dense(32, activation='relu'),
    Dense(1,  activation='sigmoid'),
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

history = model.fit(train_texts, train_labels,
                    epochs=10,
                    batch_size=32,
                    validation_split=0.2,
                    verbose=2)
# → Epoch 10/10  loss: 0.24 - accuracy: 0.91 - val_loss: 0.34 - val_accuracy: 0.86

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

train_texts = _AutoMock('train_texts')
train_labels = _AutoMock('train_labels')

What You Learned

• NLP = text → numbers → model. Most bugs happen in the first arrow.
• Use subword tokenisation in production, but word-level with TextVectorization is fine for first models.
• Embeddings are learned dense vectors, one per token. Similar words get vectors that point in similar directions.
• The minimal classifier is TextVectorization → Embedding → GlobalAveragePooling1D → Dense.
• For tasks where word order matters, use LSTM/GRU (rnn). For state-of-the-art, use transformers (genai-transformer) and the Hugging Face ecosystem.
• Always run a TF-IDF + logistic regression baseline first.

Next: Deploy Your Neural Network, where you turn the model on your laptop into a service your users can call over HTTPS.