What is Tokenization

How It Works

Tokenizers break input text into a sequence of tokens using learned or rule-based splitting strategies. Subword tokenizers like BPE (Byte Pair Encoding) and WordPiece start with characters and iteratively merge frequent pairs to build a vocabulary of subword units. Each token maps to an integer ID used as input to neural network models. Special tokens mark sequence boundaries, padding, and separation.

Technical Details

Modern tokenizers use BPE (GPT family), WordPiece (BERT), Unigram (T5), or SentencePiece (language-agnostic). Vocabulary sizes range from 32K to 128K tokens. Subword tokenization handles out-of-vocabulary words by decomposing them into known subwords. Tokenizer choice directly affects model behavior: the same text may produce different token counts and representations with different tokenizers.

Best Practices

• Always use the tokenizer that matches your model, as mismatched tokenizers cause silent errors
• Account for tokenization when setting maximum sequence lengths in your pipeline
• Test tokenization on domain-specific text to ensure important terms are tokenized sensibly
• Pre-tokenize and cache token sequences for repeated processing of the same documents

Common Pitfalls

• Mixing tokenizers between encoding and decoding, producing garbled text
• Ignoring that the same word may tokenize differently depending on surrounding context
• Not accounting for special tokens when calculating effective sequence length
• Assuming whitespace tokenization is equivalent to model-specific subword tokenization

Advanced Tips

• Train custom tokenizers on domain-specific corpora for specialized vocabularies (medical, legal)
• Use tiktoken for fast, efficient tokenization compatible with OpenAI models
• Implement token-level alignment between text and other modalities for fine-grained fusion
• Monitor token-per-word ratios to estimate costs and latency for API-based language models