GANs & Generative Models: From Adversarial Training to Diffusion

Runtime note: Keras GAN training and Hugging Face diffusers Stable Diffusion both require a local Python environment with a GPU for serious work. CPU runs the diffusers example but takes minutes per image. Neither runs in the Pyodide sandbox.

Discriminative models answer "what is this?". Generative models answer "what does data from this distribution look like?". Early generative models made images that were too blurry to be useful (VAEs) or very slow to sample (autoregressive pixel models). In 2014, Ian Goodfellow proposed training two networks against each other in a game, and GANs produced the best generated images for several years. Then, around 2021, diffusion models replaced GANs as the state of the art for almost all image generation.

This lesson covers the GAN game, why GAN training is unstable, the improvements that followed, the VAE alternative (well-founded in theory, but with blurry samples), and the diffusion models that replaced GANs. A summary of which approach wins where in 2026 is at the end.

1. The GAN Game

A GAN has two networks playing a zero-sum game:

• Generator G takes a random vector z (the "latent") and tries to produce a sample that looks like real training data.
• Discriminator D takes a sample (real or generated) and outputs a probability that it's real.

You train them alternately. D's gradient teaches it to spot the fakes. G's gradient teaches it to produce fakes that fool D. As D improves, G must produce better fakes. As G improves, D must look more closely. Both keep improving until, in the ideal case, G's outputs cannot be told apart from real data and D can only guess 50/50.

That is the whole idea. There is no likelihood function, no explicit density model and no maximum-likelihood objective. Each network improves by exploiting the other's weaknesses.

The Min-Max Intuition (No Proofs)

The training objective written formally is a min-max game:

min_G max_D  E_real[log D(x)] + E_fake[log(1 - D(G(z)))]

D is trying to maximise the expression (correctly label reals as 1 and fakes as 0). G is trying to minimise it (make D label fakes as 1). At equilibrium, G's distribution matches the data distribution and D outputs 0.5 everywhere. In practice you never reach the equilibrium. You train until the results are good enough, and stop before training becomes unstable.

2. A Vanilla GAN in Keras

The structure for a small image dataset (think MNIST-scale):

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

LATENT_DIM = 100

def build_generator():
    return keras.Sequential([
        layers.Dense(7 * 7 * 128, input_shape=(LATENT_DIM,)),
        layers.LeakyReLU(0.2),
        layers.Reshape((7, 7, 128)),
        layers.Conv2DTranspose(128, 4, strides=2, padding="same"),
        layers.LeakyReLU(0.2),
        layers.Conv2DTranspose(128, 4, strides=2, padding="same"),
        layers.LeakyReLU(0.2),
        layers.Conv2D(1, 7, padding="same", activation="tanh"),
    ])

def build_discriminator():
    return keras.Sequential([
        layers.Conv2D(64, 4, strides=2, padding="same", input_shape=(28, 28, 1)),
        layers.LeakyReLU(0.2),
        layers.Conv2D(128, 4, strides=2, padding="same"),
        layers.LeakyReLU(0.2),
        layers.Flatten(),
        layers.Dropout(0.3),
        layers.Dense(1, activation="sigmoid"),
    ])

generator = build_generator()
discriminator = build_discriminator()

# Two separate optimisers. Sharing one optimiser is a very common bug
g_opt = keras.optimizers.Adam(1e-4, beta_1=0.5)
d_opt = keras.optimizers.Adam(1e-4, beta_1=0.5)
bce = keras.losses.BinaryCrossentropy()

@tf.function
def train_step(real_images):
    batch = tf.shape(real_images)[0]
    z = tf.random.normal((batch, LATENT_DIM))

    # 1. Train discriminator on real + fake
    with tf.GradientTape() as tape:
        fake_images = generator(z, training=True)
        d_real = discriminator(real_images, training=True)
        d_fake = discriminator(fake_images, training=True)
        d_loss = bce(tf.ones_like(d_real), d_real) + bce(tf.zeros_like(d_fake), d_fake)
    d_grads = tape.gradient(d_loss, discriminator.trainable_weights)
    d_opt.apply_gradients(zip(d_grads, discriminator.trainable_weights))

    # 2. Train the generator, which wants D to label fakes as real
    z = tf.random.normal((batch, LATENT_DIM))
    with tf.GradientTape() as tape:
        fake_images = generator(z, training=True)
        d_fake = discriminator(fake_images, training=True)
        g_loss = bce(tf.ones_like(d_fake), d_fake)
    g_grads = tape.gradient(g_loss, generator.trainable_weights)
    g_opt.apply_gradients(zip(g_grads, generator.trainable_weights))

    return d_loss, g_loss

Three things to notice:

• tanh output on G, with real images scaled to [-1, 1]. Don't mix sigmoid outputs with [-1, 1] inputs. This is a common bug, and it raises no error.
• Two separate optimisers, both with beta_1=0.5. The default Adam beta_1=0.9 is too high for GANs and contributes to mode collapse.
• Conv2DTranspose for upsampling. The alternative, UpSampling2D + Conv2D, avoids the "checkerboard artifact" you sometimes see in early GAN papers.

That is the smallest GAN that works. Sometimes it will train well on MNIST, and sometimes it will mode-collapse. This instability is the main problem with GANs.

3. Why GAN Training Is Hard

Three failure modes you'll hit:

• Mode collapse. G discovers one or a few outputs that fool D reliably and stops producing variety. You generate 1,000 samples and they're all the same digit. Symptom: the samples look almost the same. The loss curves can look normal, so you have to look at the samples.
• Vanishing gradients on G. D gets too good. It assigns 0.0001 to every fake, the BCE gradient becomes tiny, and G has almost nothing to learn from. Symptom: D loss near zero, G loss flat or rising.
• Hyperparameter sensitivity. Change the LR by 2× and the result is very different. Change the batch size and the architecture can seem "broken". Reproducing a paper's results is hard, which is why GAN papers started to publish their seeds and full configurations.

Researchers responded with a series of fixes, each published as a paper.

4. The GAN Improvement Timeline

The pattern is the same as everywhere else in deep learning: better architectures, better losses, more compute, better results. Then a completely different approach (diffusion) came along.

5. Conditional GANs and Image-to-Image

A conditional GAN (cGAN) takes a label alongside the random latent and produces an output conditioned on it. "Generate a 7" instead of "generate some digit". G and D both see the label.

pix2pix generalises this: input one image, output another. Edge map → photograph, day photo → night photo, sketch → finished art. It is trained with paired data: you need both the input image and the target output.

CycleGAN removes the paired-data requirement. It uses two GANs and a cycle consistency loss: translating A → B → A should give back the original. This made unpaired domain translation possible: horses to zebras, summer to winter, oil paintings to photographs. Still used in research and specific production tasks (medical image augmentation, style transfer pipelines) even after diffusion took over general image generation.

6. VAEs: The Principled Alternative

A Variational Autoencoder is the principled, likelihood-based generative model that GANs were largely replacing. The architecture is an autoencoder with a probabilistic twist:

• Encoder maps an input x to a distribution over the latent z: it outputs a mean and a log-variance.
• Sample z from that distribution.
• Decoder maps z back to a reconstruction of x.
• Loss = reconstruction error + KL divergence pulling the encoder distribution toward a standard normal.

The KL term forces the latent space to be smooth: sample any z ~ N(0, I), decode it, and you get a plausible image. That's something a vanilla autoencoder doesn't give you.

Trade-offs vs GANs:

VAEs are still useful when you care about the latent space (anomaly detection, smooth interpolation, controllable generation), but they could not match GANs on image sharpness. One interesting detail: modern latent diffusion models (Stable Diffusion) use a VAE encoder to compress images to a small latent, then run diffusion in that latent space. So a VAE is an essential part of state-of-the-art image generation in 2026.

7. Diffusion: What Replaced GANs

The idea is very simple. Take an image, add Gaussian noise, and repeat until it is pure noise. Then train a network that removes one step of noise. To generate an image, start from pure noise and apply the denoiser many times until you have a clean image.

Formally there are two processes:

• Forward process (no learning): add a tiny bit of noise at each of T steps until the image is indistinguishable from N(0, I). T is typically 1000 during training.
• Reverse process (the learned model): predict the noise that was added at each step, subtract a fraction of it, and repeat.

The network is usually a U-Net (or, more recently, a transformer, called DiT) that takes (noisy image, current timestep) and outputs the predicted noise. Training is regression with MSE, which is far easier than the adversarial min-max game.

Why it beat GANs:

• Training stability: single loss, single optimiser. No mode collapse.
• Mode coverage: diffusion naturally captures the full diversity of the training distribution.
• Steerability: classifier-free guidance lets you trade off prompt fidelity against diversity by changing one number at inference.
• Compositionality: text conditioning works very well, using cross-attention over text embeddings.

The cost is inference speed. A GAN samples in one forward pass, but diffusion needs 20–50 iterative denoising steps even with the best samplers. Faster samplers such as DPM-Solver, and distillation, solved most of this. Distilled models can generate a good image in 1–4 steps.

By 2026 the major image generators (DALL-E 3, Stable Diffusion 3, Midjourney, Imagen) are all diffusion-based. So are the leading video models (Sora, Veo, Runway) and a growing share of audio synthesis (AudioLDM, Stable Audio).

8. Stable Diffusion in Five Lines

The diffusers library wraps Stable Diffusion behind a Hugging Face-style pipeline:

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

image = pipe(
    "a watercolour painting of a snowy mountain at dawn, soft pastel colours",
    num_inference_steps=25,
    guidance_scale=7.5,
).images[0]

image.save("mountain.png")

That's the full code. The first call downloads ~5 GB of weights. After that, each image takes seconds on a recent GPU. CPU works but takes minutes per image.

The two settings that matter most:

• num_inference_steps: fewer steps = faster but lower quality. 20–30 is the standard range for SD2/SD3. Modern distilled models (SDXL-Turbo, SD3.5-Turbo) work at 1–4 steps.
• guidance_scale (CFG): how strongly to follow the prompt. 7.5 is the usual default. Higher values follow the prompt more closely but give less variety. Lower values give more variety but follow the prompt less closely.

For image-to-image, inpainting, ControlNet (conditioning on edges, depth, poses), or LoRA fine-tuning, the diffusers library has a pipeline for each. They use the same from_pretrained pattern with a different class.

9. Which Approach Wins Where in 2026

GANs are still in use. They are the right tool for low-latency inference, image-to-image with unpaired data, and a number of specialised research niches. But for the average "I want to generate good-looking images" problem in 2026, the answer is "use a pretrained diffusion model from diffusers".

10. Ethics: The Same Tools Can Cause Harm

The same model that produces a watercolour mountain produces a convincing deepfake. The same audio model that dubs your video into ten languages clones a senator's voice. The same image model that fills in your concept art trains on web-scraped art without consent.

The standard mitigations are partial:

• Watermarking: invisible signals embedded in outputs that detectors can find. Re-encoding the file can often remove them. Better than nothing, but not a solution.
• Content provenance (C2PA): cryptographic signatures that record how a piece of media was produced. Useful for checking where media came from, but useless if the platform doesn't display the signature.
• Model-level safety: the model refuses to generate certain content. If the weights are open, fine-tuning can remove these refusals.
• Opt-out registries for training data. These are voluntary, and they arrived after many models were already trained.

This topic has its own lesson: ethics.

Common Mistakes

1. Training a GAN without seeding.

GAN training is so sensitive to initialisation that two runs with different seeds can produce visibly different results. Always tf.random.set_seed(42) (or the framework equivalent) and log the seed in your config. Otherwise you can never reproduce the good run.

2. Same learning rate for G and D.

D usually learns faster than G, because the discriminative task is easier than the generative one. If they share an LR, D pulls ahead and G's gradients vanish. Use different LRs, or different update frequencies (for example, train G twice for every D step). beta_1=0.5 instead of Adam's default 0.9 also helps.

3. Over-training the discriminator.

If D reaches near-perfect accuracy, BCE gradients vanish and G stops improving. The classic recipe is n_critic updates of D per update of G (n=1 by default, n=5 for WGAN). Watch the loss curves: if D loss is near zero while G loss climbs, D is too strong. Either reduce D's LR, add noise to D's inputs, or switch to WGAN-GP whose objective is better-behaved.

4. Mismatched output activation and pixel range.

tanh output → train on images in [-1, 1]. sigmoid output → train on images in [0, 1]. If they don't match, you'll see gradient problems that look like mode collapse but are really caused by the wrong normalisation.

5. Trusting a single quality number.

GAN papers report FID, IS, and KID. None of them fully captures sample quality. Always look at a grid of samples yourself. In the past, papers that showed only a few hand-picked good samples have hidden many mediocre models.

🎯 Your Turn: Generate an Image with Stable Diffusion

Use the diffusers library to generate an image with Stable Diffusion in under ten lines. Save it to disk. Requirements:

• Use StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5").
• Use torch.float16 and .to("cuda") if a GPU is available. Otherwise, use the CPU.
• A prompt of your choice (something visually concrete works best).
• num_inference_steps=25, guidance_scale=7.5.
• Save the result to generated.png.

Skeleton:

from diffusers import StableDiffusionPipeline
import torch

# TODO 1: choose the device: "cuda" if torch.cuda.is_available() else "cpu"
device = ...

# TODO 2: load the pipeline; use float16 on GPU, float32 on CPU; .to(device)
pipe = ...

# TODO 3: write a prompt
prompt = ...

# TODO 4: call the pipe with num_inference_steps=25, guidance_scale=7.5
result = ...

# TODO 5: save result.images[0] as "generated.png"

from diffusers import StableDiffusionPipeline
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=dtype,
).to(device)

prompt = (
    "a watercolour painting of a snowy mountain at dawn, "
    "soft pastel colours, gentle morning light"
)

result = pipe(prompt, num_inference_steps=25, guidance_scale=7.5)
result.images[0].save("generated.png")
print("Saved generated.png")

What You Learned

• A GAN trains a Generator and a Discriminator against each other. G tries to produce realistic fakes, and D tries to spot them.
• GAN training is unstable: mode collapse, vanishing gradients, hyperparameter sensitivity. Research produced a decade of fixes (DCGAN → WGAN → WGAN-GP → StyleGAN).
• Conditional GANs and pix2pix/CycleGAN extend the idea to controllable and image-to-image generation.
• VAEs are the principled, likelihood-based alternative: stable to train, smooth latent space, but blurrier samples.
• Diffusion models train a noise predictor and sample by iterative denoising. They beat GANs on stability, mode coverage, and prompt-following, at the cost of multi-step inference.
• Stable Diffusion in five lines via the diffusers library. num_inference_steps and guidance_scale are the main settings.
• In 2026, diffusion is the leading approach for image, video and most audio generation. GANs remain useful for real-time inference, unpaired image-to-image, and specific super-resolution tasks.
• Generative models have serious ethical consequences: deepfakes, copyright, consent. Watermarking and provenance are partial mitigations, not solutions. Continued in ethics.

Next: Model Deployment: taking a trained model from a notebook to a production API that handles real traffic, real failures, and real cost.