Deploy Your Neural Network

A model in a Jupyter notebook is an experiment. A model behind an HTTPS endpoint that answers requests in 80ms is a product. Most of the work between the two is connecting the parts (saving, serving, monitoring), and that work decides whether anyone uses your model.

This lesson goes through the chain train → serialise → serve → monitor and shows the practical decisions at each step.

The Keras snippets need TensorFlow. Run in Colab or pip install tensorflow fastapi uvicorn. Outputs are shown inline.

1. The Deployment Chain

trained model  →  serialise  →  serve  →  monitor
   (notebook)      (.keras)    (HTTP API)   (logs, drift, alerts)

Each step can fail in its own way:

• If you serialise it wrong, the model won't load on the server.
• If you serve it without preprocessing, it gives confident but wrong predictions.
• If you skip monitoring, you may find out six months later that your input data changed in March.

Most production ML teams spend most of their time on stages 2–4. Training is the easy part.

2. Save a Keras Model: One File

# After training
model.save("sentiment.keras")

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')

A single file (its size depends on the model) that contains:

• Architecture (the layer graph)
• Weights (the learned parameters)
• Optimiser state (so training can resume)
• Compile config (loss, metrics)

Loading works the same way in reverse, on any machine with a compatible TensorFlow version (ideally the exact same version):

from tensorflow import keras
model = keras.models.load_model("sentiment.keras")
preds = model.predict(some_input)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

some_input = _AutoMock('some_input')

That is the smallest thing you can deploy.

Pin your TensorFlow version in production. A model trained with TF 2.15 may not load cleanly in TF 2.17. Lock the version in requirements.txt / pyproject.toml.

3. The Simplest Serving: A Python Script

Before the web framework, the simplest deploy is a script that reads from stdin and writes to stdout. This is useful for batch jobs and as a quick first test.

# predict.py
import json, sys
import numpy as np
from tensorflow import keras

model = keras.models.load_model("sentiment.keras")

for line in sys.stdin:
    payload = json.loads(line)
    text = payload["text"]
    score = float(model.predict([text], verbose=0)[0][0])
    print(json.dumps({"score": score, "label": "positive" if score > 0.5 else "negative"}))

$ echo '{"text": "loved it"}' | python predict.py
{"score": 0.978, "label": "positive"}

You can deploy this. It won't scale to many users, but it works.

4. Serve Over HTTP: FastAPI in 25 Lines

For anything user-facing, you want an HTTP API. FastAPI is the usual modern choice in Python. It supports async code, checks request types, and generates OpenAPI docs automatically.

# server.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from tensorflow import keras

app = FastAPI()
model = keras.models.load_model("sentiment.keras")

class TextRequest(BaseModel):
    text: str

class TextResponse(BaseModel):
    score: float
    label: str

@app.post("/predict", response_model=TextResponse)
def predict(req: TextRequest):
    if not req.text or len(req.text) > 5000:
        raise HTTPException(400, "text must be 1–5000 chars")
    score = float(model.predict([req.text], verbose=0)[0][0])
    return TextResponse(score=score,
                        label="positive" if score > 0.5 else "negative")

Run it:

$ uvicorn server:app --host 0.0.0.0 --port 8000

Send it a request:

$ curl -X POST localhost:8000/predict \
    -H "Content-Type: application/json" \
    -d '{"text": "absolutely brilliant film"}'
# → {"score": 0.987, "label": "positive"}

Pydantic validates the request shape for you. Bad JSON gets a 422 response before your model is even called. See exceptions for the broader pattern of validating at boundaries.

5. Preprocessing: Ship It With the Model

The most common deployment bug is preprocessing drift. You normalise pixels by dividing by 255 in training. In production someone forgets this step, the model receives 0–255 inputs, and accuracy drops sharply without any error.

Two fixes:

Option A: Preprocessing inside the model

Use Keras preprocessing layers (TextVectorization, Rescaling, Normalization) inside the Sequential model. They get saved as part of model.save() and run automatically at inference.

model = Sequential([
    layers.Rescaling(1./255, input_shape=(28, 28, 1)),
    Conv2D(32, (3, 3), activation='relu'),
    # ...
])

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Sequential(*_a, **_kw):
    print('-> Sequential() called')
    return _AutoMock('Sequential()')
def Conv2D(*_a, **_kw):
    print('-> Conv2D() called')
    return _AutoMock('Conv2D()')
layers = _AutoMock('layers')

This is the recommended modern approach. The model is now a self-contained text → prediction or raw_pixels → prediction function.

Option B: sklearn-style Pipeline

If you have complex preprocessing in Python (a custom tokeniser, feature engineering), wrap everything in a sklearn.pipeline.Pipeline and pickle the whole thing, or use joblib.dump. Load both pipeline and model on the server.

from joblib import dump, load
dump(preproc_pipeline, "preproc.joblib")

# server-side
pipeline = load("preproc.joblib")
model    = keras.models.load_model("model.keras")

def predict(raw_input):
    features = pipeline.transform(raw_input)
    return model.predict(features)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

preproc_pipeline = _AutoMock('preproc_pipeline')
keras = _AutoMock('keras')

The main rule in both cases: preprocessing code is part of the model artefact, not a side script.

6. Format Conversions: When .keras Is Not Enough

.keras is fine for Python-based servers. When you need to run elsewhere, convert:

Conversion is usually one function call:

# To TFLite for mobile
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open("model.tflite", "wb").write(tflite_model)

# To ONNX (needs tf2onnx)
# python -m tf2onnx.convert --keras model.keras --output model.onnx

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

model = _AutoMock('model')
tf = _AutoMock('tf')

Don't convert until you have a clear reason. After every conversion, you must test that the new model gives the same numbers as the original.

7. Making It Faster

Start with the cheapest improvements:

• Batch requests: model.predict(X_batch) is much faster than N × model.predict(x_single). On busy servers, accumulate incoming requests for a few ms and process as a batch.
• GPU vs CPU: GPUs are better for batched inference and large models, but CPUs are often better for small models that need low latency. Profile before paying for a GPU.
• Quantisation: replace 32-bit floats with 8-bit integers. The model becomes about 4× smaller and often faster, usually with only a small loss in accuracy. TFLite supports this directly.
• Pruning: set small weights to zero. Less common in deploy than quantisation.
• Use a smaller model: train a small model to copy the large one (this is called distillation). This often gives the biggest improvement, and people often forget it.

8. Hosting Options

For your first deploy: HuggingFace Spaces for a demo, FastAPI on a VPS for a real product with predictable traffic, Modal if traffic is spiky.

9. Containerise: A Minimal Dockerfile

To make a deploy reproducible, you ship the model together with its exact runtime environment. Docker is the standard way to do this today.

FROM python:3.11-slim

WORKDIR /app

# Pin everything
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Model artefact and code
COPY sentiment.keras .
COPY server.py .

EXPOSE 8000
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]

requirements.txt:

tensorflow==2.17.0
fastapi==0.115.0
uvicorn==0.30.0
pydantic==2.9.0

Build, run, deploy:

$ docker build -t sentiment-api .
$ docker run -p 8000:8000 sentiment-api

Pin every version. "It works on my laptop" is not enough for production.

10. Monitoring: The Part Most Teams Skip

Once the model is serving, you need to know when it stops being useful. Three things to log:

• Inputs: save a sample of incoming requests. You'll need them to debug regressions and detect drift.
• Predictions: the model's output and confidence. Watch the distribution shift over time.
• Outcomes, when you can get them: the eventual ground truth (the user clicked / didn't click, the doctor agreed / disagreed). This lets you measure the real accuracy.

Drift detection: compare the distribution of recent inputs to the training distribution. If your model was trained on 2024 product reviews and 2026 reviews are talking about features that didn't exist, the model is operating out of its training distribution. Several tools (Evidently AI, WhyLabs, Arize), some of them open-source, automate this.

11. The Production Checklist

Before you call a model "deployed":

• [ ] Reproducible build: pinned versions, single command to rebuild the image
• [ ] Preprocessing inside the artefact: no preprocessing scripts that could drift
• [ ] Input validation at the boundary: clear errors for bad input, not crashes
• [ ] Versioned model file: sentiment-v3.2.keras, not model.keras
• [ ] Versioned data: you can rebuild the training set six months from now
• [ ] Health endpoint: GET /health returns 200 if the model loaded
• [ ] Logged inputs and predictions: at least a sample
• [ ] Drift monitor: even a weekly summary email is better than nothing
• [ ] Fallback path: what happens when the model service is down? (Cache last result? Return a default? Fail the request?)
• [ ] Rollback plan: can you revert to the previous model in one command?

Without these, the model is running, but you can't rely on it.

Common Mistakes

• Forgetting to ship preprocessing. Training divides by 255, production doesn't. Predictions look "almost right", which makes this bug very hard to notice.
• No input validation. Your /predict endpoint will receive empty strings, 100 MB images and malformed JSON. Validate at the boundary and return clear errors (see exceptions).
• Unpinned dependencies. A pip install tensorflow six months later picks up a new major version and your model file no longer loads. Pin versions, rebuild rarely, and test that each rebuild still passes.
• No monitoring. When the input data drifts, models get worse without any error. You won't know until someone complains, which can mean months of bad predictions in the meantime.
• Optimising before measuring. Quantising a model that's already serving in 20ms with 99th-percentile latency under 100ms is wasted effort. Measure first.

🎯 Your Turn: A predict.py Script

Write a predict.py that:

1. Loads a saved Keras model from model.keras.
2. Reads a single JSON object from stdin: {"text": "..."}.
3. Validates that text is present, is a string, and is 1–5000 characters.
4. Calls model.predict and writes a JSON response to stdout: {"score": float, "label": "positive" | "negative"}.
5. On validation failure, writes {"error": "<message>"} to stdout and exits with code 1.

Skeleton:

# predict.py
import json
import sys
from tensorflow import keras

# TODO 1: load the model

def predict_one(payload):
    # TODO 2: validate payload["text"] exists, is a string, length 1–5000
    # TODO 3: run model.predict and build the response dict
    pass

if __name__ == "__main__":
    raw = sys.stdin.read()
    try:
        payload = json.loads(raw)
        result = predict_one(payload)
        print(json.dumps(result))
    except Exception as e:
        # TODO 4: print error JSON, exit 1
        ...

# predict.py
import json
import sys
from tensorflow import keras

MODEL_PATH   = "model.keras"
MAX_TEXT_LEN = 5000

model = keras.models.load_model(MODEL_PATH)

def predict_one(payload: dict) -> dict:
    if "text" not in payload:
        raise ValueError("missing field: text")
    text = payload["text"]
    if not isinstance(text, str):
        raise ValueError("text must be a string")
    if not (1 <= len(text) <= MAX_TEXT_LEN):
        raise ValueError(f"text length must be 1–{MAX_TEXT_LEN}")

    score = float(model.predict([text], verbose=0)[0][0])
    return {
        "score": round(score, 4),
        "label": "positive" if score > 0.5 else "negative",
    }

if __name__ == "__main__":
    raw = sys.stdin.read()
    try:
        payload = json.loads(raw)
        result = predict_one(payload)
        print(json.dumps(result))
    except (json.JSONDecodeError, ValueError) as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)

$ echo '{"text": "absolutely loved it"}' | python predict.py
{"score": 0.9881, "label": "positive"}

$ echo '{}' | python predict.py
{"error": "missing field: text"}

$ echo 'not even json' | python predict.py
{"error": "Expecting value: line 1 column 1 (char 0)"}

What You Learned

• The chain: train → serialise (.keras) → serve (FastAPI) → monitor.
• Preprocessing ships with the model, as Keras preprocessing layers or a pickled sklearn Pipeline.
• Validate at the boundary: Pydantic in FastAPI, explicit checks in scripts.
• Pin versions in requirements.txt and a Dockerfile. Unpinned ML dependencies break quickly.
• Monitor inputs and predictions for drift, and have a fallback and rollback plan.
• Pick a host based on your traffic pattern: VPS for steady, Modal/Replicate for spiky, SageMaker/Vertex for enterprise scale.

That completes the deep learning fundamentals path: neuron, framework, training, vision, language, deploy. Next come the specialist tracks: deeper into CNNs, RNNs, transfer learning, transformers, and the ethics (ethics) you should think about before any of these ship to real users.