https://decodedbykanishk.codes/blogEssays on shipping AI from R&D and PoC to production at scale.https://decodedbykanishk.codes/blog/building-bpe-from-scratchhttps://decodedbykanishk.codes/blog/building-bpe-from-scratchThu, 16 Jul 2026 00:00:00 GMTByte-Pair Encoding fixes word-level tokenization by working with subword pieces: it starts from single-byte characters - so text in its range is never out-of-vocabulary - and repeatedly merges the most frequent adjacent pair. Here it is built and trained from scratch, with a live playground.https://decodedbykanishk.codes/blog/why-word-tokenizers-breakhttps://decodedbykanishk.codes/blog/why-word-tokenizers-breakThu, 16 Jul 2026 00:00:00 GMTA model never sees text - it sees integer ids from a fixed vocabulary. Building the simplest possible tokenizer makes the encode/decode contract concrete, and shows exactly where word-level tokenization loses information.