Generative AI · Foundational · Beginner

Tokenization

Also known as: Tokens

Splitting text into the sub-word units a language model actually reads, and the basis of context limits and pricing.

What Tokenization is

Models do not see characters or words; they see integer token IDs. A token is roughly three-quarters of an English word, but far less efficient for many non-Latin scripts.

How it works

Byte-pair encoding and similar algorithms learn a vocabulary of frequent sub-word pieces from a corpus. Text is greedily segmented into those pieces, mapped to IDs, and embedded into vectors before the first layer.

Why it matters

Tokens are the billing unit and the context unit. Tokenisation also explains classic oddities such as poor character counting and spelling manipulation.

Common uses

→Cost estimation for API usage
→Context budgeting in RAG pipelines
→Multilingual efficiency analysis

Strengths

✓Handles unseen words gracefully
✓Compact vocabulary

Watch for

✓Unequal cost across languages
✓Breaks character-level tasks

Continue exploring

Context Window Embeddings Inference Prompt Engineering