Why word-level tokenizers break

A language model never sees your text. It sees a list of integers. Before any attention or matrix multiply happens, something has to turn "Shall I hear more" into [8761, 5032, 4705, 6430] and back again. That something is the tokenizer, and its whole job is two functions:

• encode(text) -> [ids] - text to a list of integer ids
• decode([ids]) -> text - ids back to text

It is tempting to read that pair as a lossless round-trip, where decode(encode(x)) returns x. It usually isn't. A tokenizer first normalizes text - it may lowercase, drop punctuation, or collapse whitespace - and only then assigns ids. So decode(encode(x)) returns a normalized version of x, not x itself. How much gets lost in that normalization, and why, is the whole subject of this post.

The simplest tokenizer that satisfies the contract

To make the loss concrete we need the smallest thing that actually implements encode and decode. The design is deliberately minimal:

• Word-level. A token is a whole word. This keeps the vocabulary human-readable and the code short - you can see exactly what every id stands for.
• Lowercased. One less axis of variation, so Shall and shall don't consume two separate ids.
• Trained on tiny-shakespeare - the ~1 MB single-file Shakespeare corpus from Karpathy's char-rnn. Small enough to inspect by hand, real enough to behave like language.

"Training" here is not gradient descent - it is just collecting the vocabulary. The tokenizer walks the corpus once, and the sorted set of words it sees becomes the vocabulary, plus two special tokens:

import re
from typing import List
 
 
class SimpleTokenizer:
    """Word-level tokenizer: lowercases, splits on whitespace/punctuation, maps to integer ids."""
 
    # Punctuation is used as a delimiter (not kept as a token), text is lowercased,
    # and the Shakespeare-specific "&c" abbreviation is stripped.
    SPLIT_RE = re.compile(r"[ \n.'!,:;?\-]")
 
    def __init__(self, text: str):
        tokens = self._tokenize(text)
        vocab = sorted(set(tokens)) + ["<|unk|>", "<|endoftext|>"]
        self.token_to_id = {t: i for i, t in enumerate(vocab)}
        self.id_to_token = {i: t for t, i in self.token_to_id.items()}
 
    def _tokenize(self, text: str) -> List[str]:
        cleaned = re.sub(r"&c", "", text.lower())
        return [t for t in self.SPLIT_RE.split(cleaned) if t]

Every normalization decision lives in _tokenize: lowercase, strip the &c abbreviation, then split on punctuation and whitespace. Splitting on those characters means they act as separators and are thrown away - a choice that comes back to bite us shortly.

With the vocabulary fixed, the contract itself is four lines. encode looks each token up; anything not in the vocabulary falls back to <|unk|>. decode reverses the map:

    def encode(self, text: str) -> List[int]:
        unk = self.token_to_id["<|unk|>"]
        return [self.token_to_id.get(t, unk) for t in self._tokenize(text)]
 
    def decode(self, ids: List[int]) -> str:
        return " ".join(self.id_to_token[i] for i in ids)

The full class, with the trained vocabulary, lives in the source notebook. That .get(t, unk) fallback on the encode side is small, but it is where most of the trouble starts.

Run it yourself

Reading the code tells you what should happen; running it lets you feel it. The box below runs the real SimpleTokenizer above - trained on the full tiny-shakespeare vocabulary, executing entirely in your browser. Type a sentence and watch three things: the ids it produces, the decoded round-trip, and a diff of what changed between input and output.

Try a sentence with capitals and punctuation and watch the round-trip flatten it. Then try your own name, or any word Shakespeare never wrote, and watch it collapse to a single <|unk|>.

The four ways it loses information

Everything the playground shows you traces back to a specific line of the code above. Three of the losses happen in _tokenize, before an id is ever assigned; the fourth happens in the vocabulary lookup:

• Case is collapsed - text.lower(). Shall and shall share one id, so the round-trip can never recover the capital.
• Punctuation is a delimiter, not a token - SPLIT_RE. The split consumes . ' ! , : ; ? -, so they are never stored. "more," becomes more; the comma is gone for good.
• Contractions and hyphenates get chopped - the same regex splits on ' and -, so don't → ['don', 't'] and well-known → ['well', 'known'].
• Anything unseen becomes <|unk|> - the .get(t, unk) fallback. A name, a typo, a word that simply never appeared in tiny-shakespeare all map to the same id, and decode cannot tell them apart.

The first three are normalization choices you could argue about. The fourth is different in kind - it is not a choice, it is a wall.

Why out-of-vocabulary is inevitable, not unlucky

You might hope OOV is a rare case you can train away with more data. It isn't - it is baked into how language is distributed. Count how often each token appears in tiny-shakespeare and a few tokens dominate while the tail stretches out forever, a pattern known as Zipf's law:

A fixed word-level vocabulary has to draw a line somewhere in that tail - everything past the line is <|unk|>. And because the tail never stops growing, a bigger corpus just adds more rare words rather than closing the gap. This is the structural reason word-level vocabularies get large and still miss words. It is not a quirk of Shakespeare: it is exactly the rare-word problem that motivated subword tokenization for neural machine translation (Sennrich, Haddow & Birch, 2016).

Where this goes next

None of the four losses above are bugs - they are the honest consequences of representing text as whole-word ids. The fix is to stop using whole words. Byte-Pair Encoding works with subword pieces built up from single bytes, so nothing is ever out-of-vocabulary and the round-trip stays faithful. The next post, Building BPE from scratch, builds it from scratch - paste the same sentence into its playground and watch the <|unk|> disappear.

For more on tokenization beyond these two posts, the Hugging Face NLP Course, Ch. 6 is a thorough, approachable reference.