RNNs & LSTMs: Sequential Architectures

A CNN looks at all of an image at once. A transformer looks at all of a sequence at once. An RNN looks at one token at a time, and carries a hidden state forward from one step to the next. It is the only mainstream architecture that works step by step through a sequence.

Transformers have replaced RNNs for almost all NLP since 2018. So why is this lesson here? Because RNNs are still the right choice for streaming inference (token-by-token generation without re-processing the whole history), time series forecasting (small datasets, structured temporal correlations), and edge devices (tiny models, tiny memory). Also, attention was first added to RNNs, and transformers later kept the attention and removed the RNN. If you understand the LSTM, the transformer's design choices are much easier to follow.

1. The Problem RNNs Solve

A standard MLP has a fixed input shape, for example a 224×224 image. But what's the "input shape" of a sentence? Sentences vary in length. Audio clips vary in length. Time series vary in length. You need an architecture that:

• Accepts variable-length input,
• Maintains state across time steps,
• Shares parameters so the model doesn't grow with the sequence length.

RNNs do all three. The same set of weights is applied at every time step, with a hidden state passing information forward:

$$
\mathbf{h}_t = f(\mathbf{x}_t, \mathbf{h}_{t-1})
$$

where $\mathbf{x}_t$ is the input at time $t$ and $\mathbf{h}_t$ is the hidden state. The function $f$ has learned weights, but those weights are shared across all $t$. A 100-step sequence and a 10-step sequence use the same parameters. The RNN applies $f$ once for each step in the sequence.

2. The Vanilla RNN

The simplest version:

$$
\mathbf{h}_t = \tanh(W_h \mathbf{h}_{t-1} + W_x \mathbf{x}_t + \mathbf{b})
$$

$$
\mathbf{y}_t = W_y \mathbf{h}_t + \mathbf{b}_y
$$

Three weight matrices: $W_h$ (state-to-state), $W_x$ (input-to-state), $W_y$ (state-to-output). Same matrices used at every $t$. The hidden state is the network's memory.

In code, conceptually:

def rnn_step(x_t, h_prev, W_h, W_x, b):
    return np.tanh(W_h @ h_prev + W_x @ x_t + b)

h = np.zeros(hidden_size)
for x_t in sequence:
    h = rnn_step(x_t, h, W_h, W_x, b)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

sequence = ["alpha", "beta", "gamma"]
hidden_size = _AutoMock('hidden_size')
np = _AutoMock('np')

Training works the same as for any other network (backprop), except that the gradient flows backward through time (BPTT, backpropagation through time): from the final output, back through every time step, accumulating gradients on the shared weights at every step.

3. Vanishing & Exploding Gradients

BPTT through $T$ time steps involves multiplying $T$ Jacobians of the recurrence together. If the Jacobian's spectral norm is consistently less than 1, the product shrinks towards zero. This is the vanishing gradient problem. The model can't learn long-range dependencies because the gradient signal can't reach back far enough. If the spectral norm is consistently more than 1, the product grows very large. This is the exploding gradient problem, and it shows up as NaNs in the loss.

For a vanilla RNN with tanh, the Jacobian is bounded by the spectral norm of $W_h$ times the slope of tanh (≤ 1). With typical weights, the gradient vanishes: by step 20 or 30, almost no gradient information from the start of the sequence is left.

Gradient clipping is the simple fix for exploding gradients: $\mathbf{g} \leftarrow \mathbf{g} \cdot \min(1, c / \|\mathbf{g}\|)$. In Keras it is one argument on the optimiser (clipnorm=1.0). Clipping does not fix vanishing gradients. That needs a change to the architecture: the LSTM.

4. LSTM: The Long Short-Term Memory Cell

Hochreiter & Schmidhuber introduced the LSTM in 1997 to solve vanishing gradients. The key idea: alongside the hidden state, maintain a cell state $\mathbf{c}_t$ that flows through time with only minor modifications, controlled by gates that decide what to forget, what to write, and what to read.

For each time step, given input $\mathbf{x}_t$, previous hidden $\mathbf{h}_{t-1}$, previous cell $\mathbf{c}_{t-1}$:

$$
\mathbf{f}_t = \sigma(W_f [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) \qquad \text{forget gate}
$$

$$
\mathbf{i}_t = \sigma(W_i [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) \qquad \text{input gate}
$$

$$
\tilde{\mathbf{c}}_t = \tanh(W_c [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c) \qquad \text{candidate cell update}
$$

$$
\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \qquad \text{new cell state}
$$

$$
\mathbf{o}_t = \sigma(W_o [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o) \qquad \text{output gate}
$$

$$
\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t) \qquad \text{new hidden state}
$$

$\sigma$ is the sigmoid (gate values in $[0, 1]$), and $\odot$ is element-wise multiply. The cell state $\mathbf{c}_t$ is updated by an addition rather than a multiplication. Repeated addition does not shrink to zero or grow to infinity over many steps, so the gradient can flow back along the cell state with little decay. This is the main reason LSTMs work.

Conceptually:

Three sigmoid gates and one tanh candidate, all from the same $[\mathbf{h}_{t-1}, \mathbf{x}_t]$ input. Roughly 4× the parameter count of a vanilla RNN with the same hidden size.

5. GRU: A Simpler LSTM

Cho et al. (2014) introduced the GRU as a stripped-down LSTM. It has two gates instead of three, and no separate cell state, only the hidden state.

$$
\mathbf{z}_t = \sigma(W_z [\mathbf{h}_{t-1}, \mathbf{x}_t]) \qquad \text{update gate}
$$

$$
\mathbf{r}_t = \sigma(W_r [\mathbf{h}_{t-1}, \mathbf{x}_t]) \qquad \text{reset gate}
$$

$$
\tilde{\mathbf{h}}_t = \tanh(W [\mathbf{r}_t \odot \mathbf{h}_{t-1}, \mathbf{x}_t])
$$

$$
\mathbf{h}_t = (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t
$$

The update gate $\mathbf{z}_t$ interpolates between keeping the old hidden state and writing the candidate. A GRU has about 25% fewer parameters than an LSTM, and often performs as well on small and medium tasks. Try both, and pick the one that trains better on your problem. The general finding from experiments: roughly equivalent on most tasks. LSTM is slightly better on very long sequences, and GRU is slightly faster.

6. Bidirectional RNNs

A unidirectional RNN at step $t$ has only seen tokens $1$ through $t$. For sequence tagging (NER, POS), each token's classification benefits from seeing the whole sentence. A bidirectional RNN runs two RNNs, one forward and one backward, and concatenates their hidden states:

$$
\mathbf{h}_t = [\overrightarrow{\mathbf{h}_t} \,;\, \overleftarrow{\mathbf{h}_t}]
$$

In Keras:

from tensorflow.keras import layers

x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(embedded)
# Output channel dim is 64 * 2 = 128

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

embedded = _AutoMock('embedded')

Use bidirectional when:

• The whole sequence is available at inference time (text classification, NER).
• The future context is useful.

Don't use bidirectional RNNs for streaming (you don't have the future yet) or for autoregressive generation (the model would see the tokens it is supposed to predict).

7. Stacked RNNs

A single RNN layer captures one level of abstraction. Stack multiple layers, feeding the lower layer's hidden states as input to the upper:

x = layers.LSTM(128, return_sequences=True)(x)    # MUST return sequences
x = layers.LSTM(64,  return_sequences=True)(x)
x = layers.LSTM(32,  return_sequences=False)(x)   # final layer returns just the last state

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

layers = _AutoMock('layers')

return_sequences=True is required on every layer of a stack except the last one. Without it, the layer returns only the final hidden state, and the next LSTM gets no sequence to read. This is a very common LSTM bug.

Deeper stacks (3-4 layers) help on large datasets with rich structure. On small datasets, one bidirectional layer + a dense head is usually enough.

8. Sequence-to-Sequence

When the input and output are both sequences (translation, summarisation, speech recognition), you have seq2seq. The encoder reads the input and produces a fixed-size context vector. The decoder reads the context vector and generates the output one token at a time.

Input:  "The cat sat" →  [encoder LSTM] → context vector
                                                  ↓
Output: "Le chat est"  ← [decoder LSTM, conditioned on context]

During training, the decoder is fed the correct previous token at each step ("teacher forcing"). This trains much faster than letting the decoder use its own (probably wrong) predictions, especially early in training. At inference, you have no ground truth, so the decoder uses its own previous prediction (greedy decoding, beam search, sampling, etc.).

The problem with a fixed-size context: a single vector has to encode the entire input sequence, regardless of length. For a 50-word sentence, that's an information bottleneck. The fix:

9. Attention: The Pre-Transformer Breakthrough

Bahdanau et al. (2015): instead of compressing the input to one vector, let the decoder attend to all encoder hidden states at each step, weighting them by relevance.

At each decoder step $t$, given the current decoder state $\mathbf{s}_t$ and encoder states $\mathbf{H} = [\mathbf{h}_1, \ldots, \mathbf{h}_T]$:

$$
e_{t,i} = \text{score}(\mathbf{s}_t, \mathbf{h}_i) \qquad \text{e.g. } \mathbf{s}_t^\top W \mathbf{h}_i
$$

$$
\alpha_{t,i} = \text{softmax}_i(e_{t,i}) \qquad \text{alignment weights, summing to 1}
$$

$$
\mathbf{c}_t = \sum_i \alpha_{t,i} \mathbf{h}_i \qquad \text{context vector for step } t
$$

The decoder now uses $\mathbf{c}_t$ (which depends on the decoder's current state) instead of a fixed encoded context. Each decoder step looks at the most relevant parts of the input. Translation models learn alignments that look very similar to the word alignments a human translator would make.

This is the same idea that transformers later generalised: a query (the decoder state), keys (the encoder states), values (the encoder states) and a softmax-weighted sum. Transformers did not invent attention. Their change was to remove the RNN completely and use only attention.

10. Why Transformers Replaced RNNs

For NLP (and increasingly everywhere), RNNs are obsolete because:

Training parallelism is the biggest difference. A transformer processes a 1000-token sequence with a few large matrix multiplies per layer, but an RNN needs 1000 sequential steps. On modern hardware the transformer trains much faster, and that speed is what made pretraining LLMs at scale possible.

See Transformer Architecture for the details of attention, multi-head attention, position embeddings and the modern stack.

11. When RNNs Are Still Right

RNNs are still the better choice for:

Time series forecasting

For univariate or low-dim multivariate time series with strong temporal correlation (sales, sensor data, weather), LSTMs with 50-200 hidden units often beat transformer-based forecasters. Transformers are data-hungry. A 10k-point time series doesn't have enough data to train one well. Specialised time series transformers (Informer, Autoformer, PatchTST, TFT) are improving, but for "I have one signal and want next-month forecasts", an LSTM is still a strong default.

Streaming / token-by-token inference

At inference, an RNN processes one token at a time with O(1) memory: it keeps only the hidden state. A vanilla transformer re-attends to the entire history at every step (the KV cache helps, but it grows with sequence length). For very long generation on memory-constrained devices, RNNs win.

Edge devices

A 50k-parameter LSTM runs on a microcontroller. Most transformers are much larger. Speech keyword detection on phones, gesture recognition on watches and anomaly detection on industrial sensors are common RNN/LSTM tasks.

Practical heuristic

12. Keras LSTM in Practice

The parts of the API you will use most:

from tensorflow.keras import layers

# Many-to-one (sequence classification)
x = layers.LSTM(64, return_sequences=False)(x)        # output: (B, 64)

# Many-to-many (per-step output: NER, tagging)
x = layers.LSTM(64, return_sequences=True)(x)         # output: (B, T, 64)

# Bidirectional
x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x)

# Stacked
x = layers.LSTM(128, return_sequences=True)(x)
x = layers.LSTM(64,  return_sequences=False)(x)

# With dropout (recurrent_dropout also drops the recurrent connections)
x = layers.LSTM(64, dropout=0.2, recurrent_dropout=0.2)(x)

# Return the full state (for seq2seq decoder initialisation)
out, hidden, cell = layers.LSTM(64, return_state=True)(x)

Gradient clipping is close to mandatory for RNNs:

opt = keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

keras = _AutoMock('keras')

clipnorm=1.0 rescales the gradient vector to have at most norm 1. Without it, exploding gradients give NaN losses within a few epochs on hard sequences.

13. A Time Series Example

Predict the next value from the previous 50:

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers, Model, Input

# Toy series: a sine wave with noise
t = np.linspace(0, 100, 5000)
y = np.sin(t) + 0.1 * np.random.randn(len(t))

# Windowed: 50 past values are used to predict the next value
WINDOW = 50
X = np.array([y[i:i+WINDOW]   for i in range(len(y) - WINDOW - 1)])
Y = np.array([y[i+WINDOW]     for i in range(len(y) - WINDOW - 1)])
X = X[..., None]                       # (N, 50, 1)  (the LSTM needs a channels axis)

inputs = Input(shape=(WINDOW, 1))
x = layers.LSTM(32, return_sequences=False)(inputs)
x = layers.Dense(1)(x)

model = Model(inputs, x)
model.compile(optimizer=tf.keras.optimizers.Adam(1e-3, clipnorm=1.0),
              loss="mse", metrics=["mae"])
model.fit(X, Y, epochs=10, batch_size=32, validation_split=0.2)

# Expected: MAE ~0.08-0.12 (close to the noise floor of 0.1)

Run in Colab or locally with pip install tensorflow. Expected outputs in comments.

This is the standard recipe. The settings you tune are the hidden-state size, the window length and the depth. For multi-step forecasting, switch to seq2seq (encoder reads the window, decoder outputs N future steps) or train one model per horizon.

14. Text Generation: The Char-Level LSTM

Karpathy's 2015 demo trained a char-level LSTM on Shakespeare and sampled from it. The output sounded plausible. It is still a good teaching example.

# Pseudo-code that shows the structure, not every detail
inputs = Input(shape=(SEQ_LEN,))
x = layers.Embedding(vocab_size, 128)(inputs)
x = layers.LSTM(256, return_sequences=True)(x)
x = layers.Dense(vocab_size)(x)           # logits per position
model = Model(inputs, x)
model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3, clipnorm=1.0),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Input(*_a, **_kw):
    print('-> Input() called')
    return _AutoMock('Input()')
def Model(*_a, **_kw):
    print('-> Model() called')
    return _AutoMock('Model()')
vocab_size = _AutoMock('vocab_size')
SEQ_LEN = _AutoMock('SEQ_LEN')
layers = _AutoMock('layers')
tf = _AutoMock('tf')

Temperature sampling at generation time: divide logits by a temperature $\tau$ before softmax.

$$
P_i = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}
$$

• $\tau < 1$: sharper distribution. Output is more confident, more repetitive, "safer".
• $\tau = 1$: model's native distribution.
• $\tau > 1$: flatter distribution. Output is more diverse, more creative, also more incoherent.

def sample(logits, temperature=1.0):
    logits = logits / temperature
    probs  = tf.nn.softmax(logits).numpy()
    return np.random.choice(len(probs), p=probs)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

np = _AutoMock('np')
tf = _AutoMock('tf')

Top-k and nucleus (top-p) sampling cut off the unlikely tokens before sampling. They are the modern defaults in LLMs, and temperature works the same way with them.

15. Masking: Variable-Length Sequences

Real sequences come in variable lengths. You batch them by padding to the same length, but the padding tokens shouldn't influence the loss or the hidden state. Masking tells the RNN which positions to ignore.

# Embedding layer with mask_zero=True propagates a mask for any zero-valued input
x = layers.Embedding(vocab_size, 128, mask_zero=True)(inputs)
x = layers.LSTM(64)(x)         # automatically respects the propagated mask

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

inputs = _AutoMock('inputs')
vocab_size = _AutoMock('vocab_size')
layers = _AutoMock('layers')

When you pad with 0, the mask layer marks those positions and the LSTM skips them. Downstream Keras layers (LSTM, Bidirectional, Dense with sparse_categorical_crossentropy) all honour the mask. In Keras, mask_zero=True on the Embedding layer handles variable-length sequences in one line.

If you need explicit control, use layers.Masking(mask_value=0.0) before the LSTM.

Common Mistakes

1. Forgetting return_sequences=True on a stacked LSTM

x = layers.LSTM(128)(x)            # returns only the final state, shape (B, 128)
x = layers.LSTM(64)(x)             # error or a silently wrong result: LSTM expects 3D input

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

layers = _AutoMock('layers')

The second LSTM needs a 3D input (batch, time, features). Without return_sequences=True on the first, you've thrown away the time dimension.

2. Comparing LSTM val loss without seeding

LSTMs have high run-to-run variance. The difference between two model variants can be smaller than the noise across random seeds. Set tf.keras.utils.set_random_seed(42) and report mean ± std over 3-5 seeds, or you're measuring noise.

3. Using LSTM for sequences past a few hundred tokens

LSTMs can technically handle long sequences, but they get gradually worse as the sequence gets longer. By 500 tokens you've lost most early context. By 1000 tokens you're just relying on the last few hundred. If your task has 1k+ tokens of relevant context, switch to a transformer.

4. No gradient clipping

Without clipnorm=1.0, RNN training often ends with a NaN loss. Even when you don't get NaN, the occasional exploding step makes the final model worse. clipnorm=1.0 (or clipvalue=5.0) costs almost nothing and prevents the worst failure.

5. Wrong loss with from_logits

The Keras SparseCategoricalCrossentropy loss has a from_logits=True parameter. If your model's last layer has no softmax activation (recommended, because softmax and log computed together are more numerically stable), you must set from_logits=True. Setting it wrong silently trains the wrong objective.

6. Bidirectional in autoregressive generation

A bidirectional RNN at position $t$ has seen future tokens. That is useful for classification, but it breaks generation: the model would be conditioned on the answer. Always use unidirectional for autoregressive tasks.

🎯 Your Turn: Bidirectional LSTM Sentiment Classifier

Build a sentiment classifier for variable-length text sequences. The model should:

• Take integer-encoded input of shape (batch, max_len), padded with zeros.
• Use an Embedding(vocab_size, embed_dim, mask_zero=True) to handle masking automatically.
• Use one Bidirectional LSTM with 64 units (so output dim is 128 after concat).
• Project to a single sigmoid output for binary sentiment.
• Compile with binary cross-entropy, AdamW with clipnorm=1.0, and the accuracy metric.

import tensorflow as tf
from tensorflow.keras import layers, Input, Model

VOCAB_SIZE = 10_000
EMBED_DIM  = 128
MAX_LEN    = 200

def build_sentiment_model():
    inputs = Input(shape=(MAX_LEN,), dtype="int32")
    # TODO 1: Embedding with mask_zero=True
    # TODO 2: Bidirectional LSTM(64) with return_sequences=False (many-to-one)
    # TODO 3: Dense(1, activation="sigmoid")
    # TODO 4: build the Model and compile it
    ...
    return model

model = build_sentiment_model()
model.summary()    # expect 1,378,945 params (~1.4M), mostly the embedding

# Smoke test with random data
import numpy as np
X = np.random.randint(1, VOCAB_SIZE, size=(32, MAX_LEN))    # 0 is the padding value that gets masked
X[:, 150:] = 0                                              # pad second half
y = np.random.randint(0, 2, size=(32,))
model.fit(X, y, epochs=1, batch_size=8, verbose=0)
print(model.predict(X[:3]).shape)    # (3, 1)

Run in Colab or locally with pip install tensorflow. Expected outputs in comments.

import tensorflow as tf
from tensorflow.keras import layers, Input, Model

VOCAB_SIZE = 10_000
EMBED_DIM  = 128
MAX_LEN    = 200

def build_sentiment_model():
    inputs = Input(shape=(MAX_LEN,), dtype="int32")

    # Embedding with masking: 0 is the padding token, and it is masked automatically
    x = layers.Embedding(VOCAB_SIZE, EMBED_DIM, mask_zero=True)(inputs)

    # Bidirectional LSTM for many-to-one (only the final joined state is used)
    x = layers.Bidirectional(
        layers.LSTM(64, return_sequences=False, dropout=0.2, recurrent_dropout=0.0)
    )(x)

    # Binary sentiment head
    outputs = layers.Dense(1, activation="sigmoid")(x)

    model = Model(inputs, outputs, name="bilstm_sentiment")
    model.compile(
        optimizer=tf.keras.optimizers.AdamW(learning_rate=1e-3, clipnorm=1.0),
        loss="binary_crossentropy",
        metrics=["accuracy"],
    )
    return model


model = build_sentiment_model()
model.summary()
# Expected:
# Embedding         (None, 200, 128)    1,280,000
# Bidirectional     (None, 128)            98,816   (LSTM(64) ×2 directions)
# Dense             (None, 1)                  129
# Total params: 1,378,945

# Smoke test
import numpy as np
X = np.random.randint(1, VOCAB_SIZE, size=(32, MAX_LEN))
X[:, 150:] = 0                                # right-pad with zeros
y = np.random.randint(0, 2, size=(32,))
model.fit(X, y, epochs=1, batch_size=8, verbose=0)
print(model.predict(X[:3]).shape)             # (3, 1)
print(model.predict(X[:3]))                   # values in (0, 1)

What You Learned

• RNNs process sequences step by step, sharing weights across time. Hidden state is the memory.
• Vanilla RNNs suffer vanishing/exploding gradients through BPTT. Gradient clipping (clipnorm=1.0) fixes explosion, and LSTM fixes vanishing.
• LSTM uses three gates (forget, input, output) and a cell state highway. Cell state updates by addition, so gradients flow far back through time.
• GRU is a simplified LSTM with two gates and no separate cell state. Often equivalent performance, fewer params, faster.
• Bidirectional RNNs read forward and backward. Use them when you have the whole sequence (classification, tagging). Never for streaming or autoregressive generation.
• Stacking LSTMs requires return_sequences=True on every layer except the last. Forgetting this is a very common mistake.
• Seq2seq uses an encoder LSTM + decoder LSTM with teacher forcing during training. The fixed context bottleneck motivated attention.
• Attention lets the decoder weight encoder states by relevance at each step. The same mechanism transformers later built on.
• Transformers replaced RNNs in NLP because of training parallelism, direct long-range dependencies, and pretrained ecosystems. See Transformer Architecture.
• RNNs are still right for time series forecasting on small data, streaming inference, and edge devices.
• Masking with Embedding(mask_zero=True) handles variable-length sequences in one line.
• Temperature sampling controls the diversity-vs-coherence trade-off in generation. Top-k and top-p extend the idea.

Next in this track: Transfer Learning, where you reuse a pretrained model instead of training from scratch. Or go back to Transformer Architecture to see why the field moved from RNNs to transformers.