Writing
Essays
Shipping AI from R&D and PoC to production at scale.
Categories
Tag
Jul 16, 2026 · Tokenization · 7 min
Building BPE from scratch
Byte-Pair Encoding fixes word-level tokenization by working with subword pieces: it starts from single-byte characters - so text in its range is never out-of-vocabulary - and repeatedly merges the most frequent adjacent pair. Here it is built and trained from scratch, with a live playground.
tokenization · bpe · nlp · from-scratch
Jul 16, 2026 · Tokenization · 6 min
Why word-level tokenizers break
A model never sees text - it sees integer ids from a fixed vocabulary. Building the simplest possible tokenizer makes the encode/decode contract concrete, and shows exactly where word-level tokenization loses information.
tokenization · nlp · from-scratch