Interactive BPE TokenizerInteractive BPE Tokenizer

Byte pair encoding sits underneath every modern language model, but the merge process is usually invisible. This makes it steppable.

I defined the REST API contract from scratch, implemented the encoder in Python behind Django REST Framework, and iterated the React interface until the merge process was legible to someone seeing it for the first time.

The interface work was the harder half. Getting a first-time user to understand what a merge does, without explaining it to them, took more revisions than the encoder did.