Tokenization: Vocabulary, One-Hot and Byte Pair Encoding in Python

The concept in plain words

Neural networks only understand numbers, so text is first split into tokens and each token gets an id. Language models such as GPT use byte pair encoding, which builds tokens from frequent pieces of words.

Turn text into numbers the way language models do: build a vocabulary, one-hot encode words and learn subword tokens with BPE.

Key ideas

A token is a unit of text, such as a word, part of a word or a character.
One-hot vectors are simple but become very long for large vocabularies.
BPE handles rare and new words by splitting them into known pieces.
Models then map each token id to a learned vector called an embedding.

When should you use it?

Understand what happens inside neural networks and AI libraries.
Learn the idea before you implement it in the matching challenges.
Prepare for machine learning courses, interviews and projects.

See it in action

Type your own sentence to build a vocabulary, then press Step to watch byte pair encoding merge the most frequent pair.

Example code

Run code →

main.py

from collections import Counter

text = "the cat sat on the mat"

# 1. Build a vocabulary: every distinct word gets an id.
words = text.split()
vocab = {word: i for i, word in enumerate(sorted(set(words)))}
print("vocab:", vocab)
print("ids:", [vocab[w] for w in words])

# 2. One-hot encoding: a vector with a single 1 at the word's id.
def one_hot(word):
    return [1 if vocab[word] == i else 0 for i in range(len(vocab))]
print("one-hot 'cat':", one_hot("cat"))

# 3. Byte pair encoding: repeatedly merge the most frequent pair of symbols.
tokens = list("banana")
for _ in range(2):
    pairs = Counter(zip(tokens, tokens[1:]))
    best = max(pairs, key=pairs.get)
    merged, i = [], 0
    while i < len(tokens):
        if i + 1 < len(tokens) and (tokens[i], tokens[i + 1]) == best:
            merged.append(tokens[i] + tokens[i + 1])
            i += 2
        else:
            merged.append(tokens[i])
            i += 1
    tokens = merged
    print("merge", best, "->", tokens)

Expected output

vocab: {'cat': 0, 'mat': 1, 'on': 2, 'sat': 3, 'the': 4}
ids: [4, 0, 3, 2, 4, 1]
one-hot 'cat': [1, 0, 0, 0, 0]
merge ('a', 'n') -> ['b', 'an', 'an', 'a']
merge ('b', 'an') -> ['ban', 'an', 'a']

How it works

The vocabulary maps each distinct word to an id, and the sentence becomes a list of ids. A one-hot vector has a 1 only at the word's id. BPE counts neighbouring pairs of symbols and merges the most frequent one, first ('a', 'n') and then ('b', 'an').

Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.

Practice exercises

Modify the runnable example with the exercises below to build understanding beyond copying the result.

Change the inputs or weights and predict the output before running.
Print the intermediate values to follow each step of the calculation.
Solve the linked challenges below to implement the idea yourself.

Run in Python IDE →

Practice with a challenge

Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.

All challenges