How Byte-Pair Encoding (BPE) Tokenizes Text
Large Language Models (like GPT-4o, Claude 3.5, and Llama 3) do not process raw text or full words directly. Instead, they convert strings into sequences of integer tokens using Byte-Pair Encoding (BPE). Common words become single tokens (e.g. " architect"), while rare words, numbers, code syntax, and emojis are decomposed into sub-word byte fragments.
Key Tokenization Characteristics
- Whitespace Attachment: In modern BPE tokenizers (like
cl100k_baseando200k_base), leading whitespace is merged into the subsequent word token (e.g." hello"is a different token ID from"hello"). - Numbers & JSON: Numbers and JSON punctuation (
{,:,") often consume individual tokens, making structured JSON prompts denser than plain prose. - Character Compression Ratio: For standard English prose, 1 token roughly equals 4 characters (~0.75 words). For code and non-Latin languages, token density is higher.