Project: Word Frequency Counter

You'll build a program that reads some text and reports its most common words. A writer can run it on a draft to find words they use too often. An analyst can run it on a few thousand customer reviews to see which topics people mention most.

What you'll practise: string methods (lower, split, strip), dicts, collections.Counter, regex basics, sorting.

Step 1: The Three-Line Version

python

text = "the quick brown fox jumps over the lazy dog the fox is quick"

counts = {}
for word in text.split():
    counts[word] = counts.get(word, 0) + 1

print(counts)

dict.get(key, 0) returns 0 when the key isn't there yet, so there's no need for a separate if word in counts check.

Output:

python

{'the': 3, 'quick': 2, 'brown': 1, 'fox': 2, 'jumps': 1, 'over': 1, 'lazy': 1, 'dog': 1, 'is': 1}

Step 2: Use `collections.Counter`

Counting things comes up so often that the standard library has a class for it.

python

from collections import Counter

text = "the quick brown fox jumps over the lazy dog the fox is quick"

counts = Counter(text.split())
print(counts)
print(counts.most_common(3))

Counter is a dict with two extra features:

A missing key reads as 0, so you don't need .get(k, 0)

.most_common(n) gives you the top n, already sorted

Output:

python

Counter({'the': 3, 'quick': 2, 'fox': 2, 'brown': 1, ...})
[('the', 3), ('quick', 2), ('fox', 2)]

Most data-processing projects use Counter somewhere.

Step 3: Real Text Is Messy

Real text has capital letters, punctuation and contractions. To text.split(), "Fox", "fox," and "fox" are three different words.

python

from collections import Counter

text = "The fox is quick. The fox is brown! THE FOX..."

# Too simple, and wrong
print(Counter(text.split()).most_common(3))
# [('The', 2), ('fox', 2), ('is', 2)]   ("The" and "THE" are counted separately)

# Fix step 1: lowercase
text = text.lower()

# Fix step 2: strip punctuation. Two options:
import string

# Option A: str.translate to delete punctuation
table = str.maketrans("", "", string.punctuation)
cleaned = text.translate(table)
print(Counter(cleaned.split()).most_common(3))
# [('the', 3), ('fox', 3), ('is', 2)]

# Option B: regex (more flexible)
import re
words = re.findall(r"[a-zA-Z']+", text)
print(Counter(words).most_common(3))
# [('the', 3), ('fox', 3), ('is', 2)]

The regex [a-zA-Z']+ matches runs of letters and apostrophes, so a contraction like "don't" stays one word.

Step 4: Remove Common "Stop Words"

The most common English words ("the", "a", "is", "of", ...) are at the top of every count, so they tell you nothing about the text. Remove them, and the words left at the top are the ones that describe this particular text.

python

from collections import Counter
import re

STOP_WORDS = {
    "the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for",
    "of", "with", "by", "from", "as", "is", "was", "are", "were", "be",
    "been", "being", "have", "has", "had", "do", "does", "did", "will",
    "would", "could", "should", "may", "might", "must", "can", "this",
    "that", "these", "those", "i", "you", "he", "she", "it", "we", "they",
    "what", "which", "who", "whom", "this", "that", "am", "if", "not",
    "no", "yes", "so", "than", "then", "there", "their", "them", "its",
}

text = """
Python is a high-level, general-purpose programming language. Its design
philosophy emphasizes code readability with the use of significant indentation.
Python is dynamically typed and garbage-collected. It supports multiple
programming paradigms, including structured (particularly procedural),
object-oriented and functional programming.
"""

words = re.findall(r"[a-zA-Z']+", text.lower())
meaningful = [w for w in words if w not in STOP_WORDS and len(w) > 1]

counts = Counter(meaningful)
for word, n in counts.most_common(5):
    print(f"  {word:<15} {n}")

Output:

python

  programming     3
  python          2
  high            1
  level           1
  general         1

Search engines and NLP libraries such as NLTK ship their own stop-word lists. This is the same idea, written by hand.

Step 5: Read from a File

Now make it work on real files.

python

from collections import Counter
import re
import io

# In real code:
#   with open("article.txt", "r", encoding="utf-8") as f:
#       text = f.read()
# In the browser, simulate with a string:
text = """
The quick brown fox jumps over the lazy dog. The dog barks at the fox.
The fox runs away. Foxes are quick; dogs are loyal.
"""

STOP_WORDS = {"the", "a", "an", "is", "are", "at", "over", "and", "of"}

words = re.findall(r"[a-zA-Z']+", text.lower())
filtered = [w for w in words if w not in STOP_WORDS and len(w) > 2]
counts = Counter(filtered)

print(f"Total words: {len(words)}")
print(f"Unique words: {len(set(words))}")
print(f"\nTop 5 (after filtering):")
for word, n in counts.most_common(5):
    bar = "█" * n
    print(f"  {word:<10} {bar} {n}")

"█" * n repeats the block character n times, which gives you a rough bar chart in the console.

Step 6: Comparison Mode

Compare two texts. Which words appear in both? Which are unique to each?

python

from collections import Counter
import re

def words_of(text):
    return re.findall(r"[a-zA-Z']+", text.lower())

text_a = "the cat sat on the mat the cat was happy"
text_b = "the dog sat in the sun the dog was content"

counts_a = Counter(words_of(text_a))
counts_b = Counter(words_of(text_b))

# Counter supports set-like operations
shared = counts_a & counts_b           # the smaller count of each word
only_a = counts_a - counts_b           # words that appear more often in A
only_b = counts_b - counts_a

print("Shared:", shared.most_common())
print("Only in A:", only_a.most_common())
print("Only in B:", only_b.most_common())

Counter also supports + and | (max). Subtraction drops anything that falls to zero or below, which is why only_a leaves out words that appear equally often in both.

Step 7: A Polished Final Version

python

from collections import Counter
import re

STOP_WORDS = {
    "the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for",
    "of", "with", "by", "from", "as", "is", "was", "are", "were", "be",
    "have", "has", "had", "i", "you", "he", "she", "it", "we", "they",
    "this", "that", "these", "those", "if", "not", "so", "than", "then",
    "there", "their", "them", "its", "am", "do", "does", "did", "will",
}

def analyze_text(text, *, min_word_length=2, top_n=10, remove_stop_words=True):
    """Return a sorted list of (word, count) tuples."""
    words = re.findall(r"[a-zA-Z']+", text.lower())
    if remove_stop_words:
        words = [w for w in words if w not in STOP_WORDS]
    words = [w for w in words if len(w) >= min_word_length]
    return Counter(words).most_common(top_n)

def print_report(text, label="Text"):
    print(f"\n=== {label} ===")
    results = analyze_text(text, top_n=8)
    max_count = max((c for _, c in results), default=1)
    for word, count in results:
        bar = "█" * int(20 * count / max_count)
        print(f"  {word:<14} {bar} {count}")

sample = """
Python is a high-level, general-purpose programming language. Its design philosophy emphasizes
code readability with significant indentation. Python is dynamically typed and garbage-collected.
"""

print_report(sample, "Python description")

Stretch Goals

1. Multiple files: aggregate counts across several files in a folder.
2. N-grams: count two-word phrases, not just single words ("quick brown", "the cat").
3. Sentence length stats: average words per sentence, longest sentence, etc.
4. Word cloud data: output JSON in the shape word-cloud libraries expect.
5. Live monitoring: watch a log file and update the counter as new lines come in (use time.sleep and seek).
6. CLI: accept a filename argument with sys.argv or argparse.

🎯 Your Turn: N-Grams (Two-Word Phrases)

Counting single words has a limit. If you count pairs of words that appear next to each other, you find the phrases a text repeats, like "the cat" or "machine learning".

Build a function that counts the top N two-word phrases (bigrams) in a text.

python

from collections import Counter
import re

def top_bigrams(text, top_n=5):
    """Return the top N two-word phrases in the text.

    Example: 'the cat sat on the mat' has bigrams:
        ('the','cat'), ('cat','sat'), ('sat','on'), ('on','the'), ('the','mat')
    """
    # TODO 1: tokenize to lowercase words
    # TODO 2: build a list of (word_i, word_{i+1}) tuples for i = 0..len-2
    # TODO 3: feed to Counter and return .most_common(top_n)
    pass

# Test
text = "the cat sat on the mat the cat was happy the cat sat again"
print(top_bigrams(text, 3))
# Expected roughly [(('the', 'cat'), 3), (('cat', 'sat'), 2), ...]

Hint 1: Generating pairs from a list

If words = ['a', 'b', 'c', 'd'], then list(zip(words, words[1:])) gives [('a','b'), ('b','c'), ('c','d')].

Hint 2: Feeding tuples to Counter

Counter works on any iterable. Pass it a list of tuples and you get tuple → count.

Show full solution

python

from collections import Counter
import re

def top_bigrams(text, top_n=5):
    words = re.findall(r"[a-zA-Z']+", text.lower())
    if len(words) < 2:
        return []
    bigrams = list(zip(words, words[1:]))
    return Counter(bigrams).most_common(top_n)

text = "the cat sat on the mat the cat was happy the cat sat again"
for (a, b), n in top_bigrams(text, 5):
    print(f"  '{a} {b}': {n}")

Bigram counts are also the oldest kind of language model. You record which word follows which, and then generate text by choosing each next word based on those counts.

What You Learned

dict.get(key, default) for default values
collections.Counter for counting anything
String cleaning: .lower(), .translate(), string.punctuation
re.findall(r"[a-zA-Z']+", text) to pull out words
List comprehensions for filtering ([w for w in words if condition])
Counter arithmetic (&, -) for comparison

Next time you catch yourself writing counts[x] = counts.get(x, 0) + 1, use a Counter instead.

Next: Mad Libs Generator, where you fill a story template with the reader's words.

1 · The lesson

Step 1: The Three-Line Version

Step 2: Use `collections.Counter`

Step 3: Real Text Is Messy

Step 4: Remove Common "Stop Words"

Step 5: Read from a File

Step 6: Comparison Mode

Step 7: A Polished Final Version

Stretch Goals

🎯 Your Turn: N-Grams (Two-Word Phrases)

What You Learned

Practice this

Project: Word Frequency Counter

Step 1: The Three-Line Version

Step 2: Use collections.Counter

Step 3: Real Text Is Messy

Step 4: Remove Common "Stop Words"

Step 5: Read from a File

Step 6: Comparison Mode

Step 7: A Polished Final Version

Stretch Goals

🎯 Your Turn: N-Grams (Two-Word Phrases)

What You Learned

Step 2: Use `collections.Counter`