Jul 16, 2026 · Tokenization · 7 min
Building BPE from scratch
Byte-Pair Encoding fixes word-level tokenization by working with subword pieces: it starts from single-byte characters - so text in its range is never out-of-vocabulary - and repeatedly merges the most frequent adjacent pair. Here it is built and trained from scratch, with a live playground.
tokenization · bpe · nlp · from-scratch