Project: Markdown to HTML Converter

You'll write your own Markdown-to-HTML converter, without the markdown or mistune library. It will handle headers, bold and italic text, lists, code blocks, links and images. By the end, you'll have written a small translator from one language to another, learned the state-machine pattern for line-by-line parsers, and learned where HTML escaping must happen.

What you'll practise: regex pipelines, line-by-line state machines, the small-interpreter pattern, HTML escaping, class-based stateful processing.

Step 1: Inline Replacements

Start with the easy parts: **bold**, *italic* and `code`. Regex can handle all three.

python

import re

def render_inline(text):
    # Order matters: bold (**) before italic (*) so that ** is not matched as two single *
    text = re.sub(r"\*\*(.+?)\*\*", r"<strong>\1</strong>", text)
    text = re.sub(r"\*(.+?)\*",     r"<em>\1</em>",         text)
    text = re.sub(r"`(.+?)`",       r"<code>\1</code>",     text)
    return text

# Test
print(render_inline("This is **bold** and *italic* and `code`."))
print(render_inline("Mix **bold *italic* stuff**."))

.+? is non-greedy: it matches as few characters as possible. Without the ?, **a** and **b** would match as one single <strong> block. Use non-greedy matching whenever the same delimiter can appear more than once in a line.

This first version has one problem: the * rules also run inside code, so `a*b*c` becomes <code>a<em>b</em>c</code>. The final version in Step 6 fixes this by taking the code spans out of the text before the other rules run.

Step 2: Headers

# Header becomes <h1>Header</h1>, and so on for all six levels. We check one line at a time with one regex.

python

import re

def render_header(line):
    m = re.match(r"^(#{1,6})\s+(.+?)\s*$", line)
    if not m:
        return None        # not a header
    level = len(m.group(1))
    return f"<h{level}>{m.group(2)}</h{level}>"

# Test
for line in [
    "# Top-level title",
    "## Section",
    "###### Tiny header",
    "####### Too many hashes",       # 7 hashes, so not a header
    "not a header",
]:
    out = render_header(line)
    print(f"  {line!r:<35} → {out!r}")

The parts of the regex:

^(#{1,6}): start of line, then 1-6 hashes (captured)

\s+: at least one space (so #foo is NOT a header)

(.+?)\s*$: the title, without the whitespace at the end

When the line is not a header, the function returns None, and the caller can try the next rule. Many parsers are built this way.

Step 3: Lists (Where You Need a State Machine)

Here is the difficult part of Markdown: a list is a block, not a single line. <ul> opens before the first item and closes after the last one. To know which item is the last, the code must remember what it has already seen. That memory is called state.

python

import re

def render(lines):
    """Convert lines of markdown to lines of HTML, handling lists."""
    out = []
    in_list = None                # None, 'ul', or 'ol'

    for line in lines:
        # Bullet list item?
        m_ul = re.match(r"^\s*[-*]\s+(.+)$", line)
        # Numbered list item?
        m_ol = re.match(r"^\s*\d+\.\s+(.+)$", line)

        if m_ul:
            if in_list != "ul":
                if in_list: out.append(f"</{in_list}>")
                out.append("<ul>")
                in_list = "ul"
            out.append(f"  <li>{m_ul.group(1)}</li>")
        elif m_ol:
            if in_list != "ol":
                if in_list: out.append(f"</{in_list}>")
                out.append("<ol>")
                in_list = "ol"
            out.append(f"  <li>{m_ol.group(1)}</li>")
        else:
            if in_list:
                out.append(f"</{in_list}>")
                in_list = None
            out.append(line)

    # End-of-document: don't forget to close any open list
    if in_list:
        out.append(f"</{in_list}>")
    return "\n".join(out)

# Test
md = [
    "Some intro text.",
    "- apples",
    "- bananas",
    "- cherries",
    "",
    "1. first",
    "2. second",
    "",
    "Some closing text.",
]
print(render(md))

in_list is the state variable. It has three states (None, "ul", "ol"), and each input line can change the state. Any block-level parser can use the same structure: a state variable, changes of state when a pattern matches, and a final cleanup step at the end of the document.

Step 4: Code Blocks (Another State Machine)

Fenced code blocks (```...```) need state too. The content inside the fence is copied exactly as it is. It is NOT processed as Markdown.

python

def render_with_code(lines):
    out = []
    in_code = False

    for line in lines:
        if line.strip().startswith("```"):
            if not in_code:
                out.append("<pre><code>")
                in_code = True
            else:
                out.append("</code></pre>")
                in_code = False
            continue

        if in_code:
            # Copy the line as it is: no markdown processing inside code blocks
            out.append(line)
        else:
            out.append(line)        # would run inline/header rules here

    if in_code:
        out.append("</code></pre>")  # close a code block that was never closed
    return "\n".join(out)

md = [
    "Here's some code:",
    "```",
    "def greet(name):",
    "    return f'hello, {name}'",
    "```",
    "And we're back to prose with **bold**.",
]
print(render_with_code(md))

This on/off switch (in_code = not in_code would also work) is the simplest possible state machine: two states, one transition. A parser with ten states works on the same principle: "Which state am I in, and how does this input change the state?"

Step 5: Links and Images

These are inline patterns again, but now with capture groups for the parts.

python

import re

def render_links(text):
    # Images FIRST: they start with ! but otherwise use the same syntax as links
    text = re.sub(
        r"!\[([^\]]*)\]\(([^)]+)\)",
        r'<img alt="\1" src="\2">',
        text,
    )
    # Then links
    text = re.sub(
        r"\[([^\]]+)\]\(([^)]+)\)",
        r'<a href="\2">\1</a>',
        text,
    )
    return text

# Test
samples = [
    "See [the docs](https://docs.python.org).",
    "An image: ![diagram](/img/arch.png) inline.",
    "Mixed [link](http://x.com) and ![pic](pic.jpg).",
]
for s in samples:
    print(f"  {s}")
    print(f"  → {render_links(s)}\n")

The order matters again: process images before links. Otherwise the link regex matches the [alt](src) part of an image and leaves the ! behind.

[^\]]+ means "one or more characters that aren't ]", so the match stops at the closing bracket of the label. In the same way, [^)]+ makes the URL match stop at the closing parenthesis.

This version does not check the URLs yet. A link like [click](javascript:alert(1)) would run code when someone clicks it. The final version in Step 6 adds that check.

Step 6: Polished Final Version (a `Markdown` Class)

Now put it all together: paragraphs, all the state machines working together, and HTML escaping for both normal text and code blocks.

python

import re
from html import escape

class Markdown:
    """Tiny subset-of-CommonMark converter from Markdown to HTML.

    Handles: headers, paragraphs, bold/italic/code, lists, code blocks,
    links and images. HTML chars are escaped everywhere, including inside
    fenced code.
    """

    HEADER_RE = re.compile(r"^(#{1,6})\s+(.+?)\s*$")
    UL_RE     = re.compile(r"^\s*[-*]\s+(.+)$")
    OL_RE     = re.compile(r"^\s*\d+\.\s+(.+)$")
    FENCE_RE  = re.compile(r"^\s*```")

    def to_html(self, text):
        lines = text.splitlines()
        out = []
        self._in_list = None
        self._in_code = False
        self._para = []

        for line in lines:
            if self.FENCE_RE.match(line):
                self._flush_para(out)
                self._close_list(out)
                if not self._in_code:
                    out.append("<pre><code>")
                    self._in_code = True
                else:
                    out.append("</code></pre>")
                    self._in_code = False
                continue

            if self._in_code:
                # Escape inside code (avoids breaking <pre>), but no markdown.
                out.append(escape(line))
                continue

            if not line.strip():
                self._flush_para(out)
                self._close_list(out)
                continue

            m_h = self.HEADER_RE.match(line)
            if m_h:
                self._flush_para(out)
                self._close_list(out)
                level = len(m_h.group(1))
                out.append(f"<h{level}>{self._inline(m_h.group(2))}</h{level}>")
                continue

            m_ul = self.UL_RE.match(line)
            m_ol = self.OL_RE.match(line)
            if m_ul or m_ol:
                self._flush_para(out)
                kind = "ul" if m_ul else "ol"
                if self._in_list != kind:
                    self._close_list(out)
                    out.append(f"<{kind}>")
                    self._in_list = kind
                text = (m_ul or m_ol).group(1)
                out.append(f"  <li>{self._inline(text)}</li>")
                continue

            # Default: paragraph text. Accumulate until blank line.
            self._close_list(out)
            self._para.append(line)

        # End of document: write out anything still open
        self._flush_para(out)
        self._close_list(out)
        if self._in_code:
            out.append("</code></pre>")

        return "\n".join(out)

    def _inline(self, text):
        # Escape FIRST, then re-introduce our HTML via the replacements.
        # (Escaping afterwards would break <strong> and the other tags.)
        text = escape(text)
        # Split out `code` spans first, so the * and link rules never change them.
        # Because the pattern is in ( ), split() keeps the code spans: they are the odd items.
        parts = re.split(r"(`[^`]+`)", text)
        for i, part in enumerate(parts):
            if i % 2 == 1:
                parts[i] = f"<code>{part[1:-1]}</code>"
            else:
                parts[i] = self._format(part)
        return "".join(parts)

    def _format(self, text):
        text = re.sub(r"!\[([^\]]*)\]\(([^)]+)\)",
                      lambda m: f'<img alt="{m[1]}" src="{self._safe_url(m[2])}">', text)
        text = re.sub(r"\[([^\]]+)\]\(([^)]+)\)",
                      lambda m: f'<a href="{self._safe_url(m[2])}">{m[1]}</a>', text)
        text = re.sub(r"\*\*(.+?)\*\*", r"<strong>\1</strong>", text)
        text = re.sub(r"\*(.+?)\*",     r"<em>\1</em>",         text)
        return text

    def _safe_url(self, url):
        # escape() does not stop "javascript:alert(1)" from running when clicked,
        # so a URL with a scheme (the part before ":") must be http, https or mailto.
        # Relative URLs such as "pic.png" or "/docs/" have no scheme and are kept.
        has_scheme = ":" in url.split("/")[0]
        if has_scheme and not url.lower().startswith(("http:", "https:", "mailto:")):
            return "#"
        return url

    def _flush_para(self, out):
        if self._para:
            body = " ".join(self._para)
            out.append(f"<p>{self._inline(body)}</p>")
            self._para = []

    def _close_list(self, out):
        if self._in_list:
            out.append(f"</{self._in_list}>")
            self._in_list = None


# Demo
SAMPLE = """# Hello, Markdown

This is a **bold** intro with *italic* and `inline code`. We also
support [links](https://example.com) and ![images](pic.png).

## Lists

- one
- two
- three

1. first
2. second

## Code

```
def safe():
    return "<script>alert('xss')</script>"
```

The line above is escaped. & < > all become entities.
"""

print(Markdown().to_html(SAMPLE))

The security point: escape() runs on normal text, so <script>alert(1)</script> in user input becomes <script>..., which is safe to put in a page. But escaping is not enough for links: [click](javascript:alert(1)) contains no < or &, and the browser would still run the code when someone clicks it. That is why _safe_url replaces any URL with a scheme other than http:, https: or mailto: with #. escape() ALSO runs on the content of code blocks, because otherwise a < in the code would break the HTML around it. A common bug is to escape the normal text but forget the code blocks. Then a < in a code sample breaks the HTML without any error. Another bug is to escape the text AFTER converting ** to <strong>: the escaping turns your own tags into visible text.

Stretch Goals

1. GitHub-flavoured tables: | col1 | col2 |\n|------|------|\n| a | b | → <table>. This is another state machine: detect the header line, then the separator line, then the rows.
2. Task lists: - [ ] todo and - [x] done → <input type="checkbox" disabled> inside the <li>. Change the list-item text before the inline rules run.
3. Syntax highlighting: pass the code blocks to Pygments (pip install pygments). Wrap the fence in <pre class="highlight"><code class="language-python">...</code></pre>.
4. Footnote support: text[^1] ... [^1]: explanation → numbered footnotes at the bottom of the document. Use two passes: first collect the definitions, then render.
5. Stream a huge file: process the file line by line with a generator, and yield each HTML line when it is ready. Then you can convert a 100 MB Markdown file without loading all of it into memory.

🎯 Your Turn: `slugify_headers(html)`

In a GitHub README, you can link directly to a header: #installation jumps to the <h2>Installation</h2>. For this to work, the converter must add id="installation" to the header. Build a function that adds these ids to the finished HTML.

python

import re

def slugify_headers(html):
    """Add id='...' to every <h1>-<h6> tag based on its text content.

    Rules for the slug:
      - lowercase the header text
      - replace runs of whitespace with a single hyphen
      - strip any chars that aren't letters, digits, or hyphens

    Example:
      <h2>Getting Started!</h2>  →  <h2 id="getting-started">Getting Started!</h2>
      <h1>Step 3 — Setup</h1>    →  <h1 id="step-3--setup">Step 3 — Setup</h1>
    """
    # TODO 1: write a helper slugify(s) that applies the three rules
    # TODO 2: use re.sub with a CALLBACK function to find each <hN>TEXT</hN>
    #         The callback receives the match, computes the slug from
    #         match.group(2), and returns the replacement string.
    # TODO 3: think carefully about the pattern:
    #           <(h[1-6])>([^<]+)</\1>
    pass


# Test
html = """<h1>Project Title</h1>
<p>Some intro.</p>
<h2>Step 1 — Setup</h2>
<p>...</p>
<h2>Step 2: The Plan</h2>
<h3>A Sub-section!</h3>"""

print(slugify_headers(html))

Hint 1: Building the slug

s = s.lower(), then s = re.sub(r"\s+", "-", s), then s = re.sub(r"[^a-z0-9-]", "", s). The order matters. Make the text lowercase before you remove the other characters. Replace whitespace before you remove characters, otherwise the words of a multi-word header run together.

Hint 2: Regex with a callback

re.sub accepts a function as the replacement. The function receives the Match object and returns the replacement string: re.sub(r"<(h[1-6])>([^<]+)</\1>", lambda m: f'<{m[1]} id="{slugify(m[2])}">{m[2]}</{m[1]}>', html). The \1 backreference in the pattern ensures the opening and closing tags match (you don't slugify <h1>...</h2>).

Show full solution

python

import re

def slugify(s):
    s = s.lower()
    s = re.sub(r"\s+", "-", s)
    s = re.sub(r"[^a-z0-9-]", "", s)
    return s

def slugify_headers(html):
    def replace(m):
        tag, text = m.group(1), m.group(2)
        return f'<{tag} id="{slugify(text)}">{text}</{tag}>'
    return re.sub(r"<(h[1-6])>([^<]+)</\1>", replace, html)


html = """<h1>Project Title</h1>
<p>Some intro.</p>
<h2>Step 1 — Setup</h2>
<p>...</p>
<h2>Step 2: The Plan</h2>
<h3>A Sub-section!</h3>"""

print(slugify_headers(html))

Output:

python

<h1 id="project-title">Project Title</h1>
<p>Some intro.</p>
<h2 id="step-1--setup">Step 1 — Setup</h2>
<p>...</p>
<h2 id="step-2-the-plan">Step 2: The Plan</h2>
<h3 id="a-sub-section">A Sub-section!</h3>

Note the double hyphen in step-1--setup. The dash character was removed, but the spaces around it became hyphens. Some slugifiers also replace several hyphens in a row with one: s = re.sub(r"-+", "-", s). Add that step if you want shorter ids.

A callback function is one of the most useful features of re.sub. Use it whenever the replacement depends on the content of the match, and not only on the order of the capture groups.

What You Learned

Inline regex pipelines: non-greedy .+?, and passes in the right order (bold before italic, images before links).
The state-machine pattern for block-level parsers: a state variable, changes of state for each input line, and a cleanup step at the end of the document.
HTML escaping rules: escape normal text, escape code blocks, and NEVER escape after you have added your own tags.
re.sub with a callback, for when the replacement depends on the content of the match.
A class that holds the state: self._in_list, self._in_code and self._para, instead of passing these variables to every helper function.

You just built a small translator from Markdown to HTML. It splits the input into pieces (with regex), keeps state (the flags and the paragraph buffer) and produces output (the HTML lines). Compilers, template engines and config parsers have the same three parts, so they will be easier to understand now.

The next two projects in this path are an image resizer and a chatbot. After that, the next levels cover web development (Flask, FastAPI), data (pandas, NumPy) and async code. All of them use the skills you have practised in these projects.

Next: Project: Image Batch Resizer.

← Back to Academy

1 · The lesson

Step 1: Inline Replacements

Step 2: Headers

Step 3: Lists (Where You Need a State Machine)

Step 4: Code Blocks (Another State Machine)

Step 5: Links and Images

Step 6: Polished Final Version (a `Markdown` Class)

Stretch Goals

🎯 Your Turn: `slugify_headers(html)`

What You Learned

Practice this

Project: Markdown to HTML Converter

Step 1: Inline Replacements

Step 2: Headers

Step 3: Lists (Where You Need a State Machine)

Step 4: Code Blocks (Another State Machine)

Step 5: Links and Images

Step 6: Polished Final Version (a Markdown Class)

Stretch Goals

🎯 Your Turn: slugify_headers(html)

What You Learned

Step 6: Polished Final Version (a `Markdown` Class)

🎯 Your Turn: `slugify_headers(html)`