Tokenization: How LLMs Read Text

Learn how Large Language Models convert human language into tokens and token IDs, including vocabulary, subword tokenization, special tokens, BPE, WordPiece, SentencePiece, context length, chat templates, and why tokenization matters.

25 minBeginner

The Core Idea: An LLM does not receive a sentence as a sequence of human-readable words. Before the model can process text, a tokenizer converts the text into smaller units called tokens, and those tokens are mapped to numerical token IDs. This conversion is the first major step between human language and the mathematical operations performed by the model.

  1. Beginner: What Is Tokenization?

Tokenization is the process of breaking text into smaller pieces called tokens. The tokenizer then maps those pieces to integer IDs from a vocabulary that the model understands.

A token is not necessarily a complete word. Depending on the tokenizer, a token may represent a whole word, part of a word, punctuation, whitespace-related information, or another frequently occurring text pattern.

Human Language → Model Input

Human-readable text
"Large language models are useful."
↓
Tokenizer

Splits the text according to the tokenizer's vocabulary and rules.

↓
Tokens
Largelanguagemodelsareuseful.
↓
Token IDs
[ ... , ... , ... , ... , ... , ... ]

The exact IDs depend on the tokenizer and vocabulary.

Step #1
Text

The input written by a human or produced by another system.

Step #2
Tokens

The pieces of text selected by the tokenizer.

Step #3
Token IDs

Integer identifiers used to identify entries in the tokenizer vocabulary.

Important: Tokenization happens before the Transformer processes the sequence. The Transformer does not directly operate on raw human-readable text.
⚡ Knowledge Check

What does a tokenizer primarily do?

A) Convert text into tokens and token IDs▼
✓ Correct!The tokenizer converts text into the token representation expected by the model.
B) Train all of the model's neural-network weights▼
✕ Incorrect.Tokenization is a text-processing step. It does not train the model parameters.

  1. Beginner: Words Are Not the Same as Tokens

One of the most common beginner assumptions is that one word equals one token. Modern LLM tokenizers generally do not work that way.

Character-Level

Each character can become a separate unit. This gives flexible coverage but can create very long sequences.

Word-Level

Each word can be treated as a unit. This is intuitive but creates a large vocabulary and struggles with rare words.

Subword-Level

Words can be represented using smaller recurring pieces. This provides a practical balance between vocabulary size and sequence length.

Modern language models commonly use subword-based or subword-inspired tokenization. As a result, a familiar word may be one token while another word may be split into several tokens.

A Conceptual Example

Text
unbelievable
→
Possible Pieces
unbelievable

This is a conceptual illustration. The exact split depends on the tokenizer and vocabulary.

Why Subwords Work Well

Subword tokenization lets a tokenizer reuse common pieces across many words. A rare word does not necessarily need a completely new vocabulary entry; it can often be represented using smaller pieces that already exist in the vocabulary.

  1. Beginner: Vocabulary and Token IDs

A tokenizer has a vocabulary: a collection of token entries that the tokenizer knows how to map to integer IDs. The model later uses those IDs as the entry point for learned token representations.

TokenToken IDPurpose
hello...A vocabulary entry representing a token piece.
world...Another token entry with its own integer identifier.
....Punctuation can also be represented as a token.

The exact numeric IDs are model and tokenizer specific. You should not memorize a particular ID. What matters is the pipeline:

Text→Tokens→Token IDs→Embedding Lookup→Vectors
Think of the vocabulary as an index. The tokenizer maps text pieces to IDs, and the model uses those IDs to retrieve learned representations. The ID itself has no mathematical meaning such as "larger number = more important token."
⚡ Knowledge Check

What does a token ID represent?

A) An integer identifier for a token in the vocabulary▼
✓ Correct!The token ID identifies a vocabulary entry.
B) A numerical score showing how important the token is▼
✕ Incorrect.The ID is an identifier, not an importance score.

  1. Beginner: Subword Tokenization

Subword tokenization is a practical compromise between character-level and word-level tokenization. Instead of requiring every possible word to appear in the vocabulary, the tokenizer can represent many words using reusable pieces.

Frequent Word

A common word may be represented efficiently as one token or a small number of tokens.

Rare Word

A rare or unusual word can be divided into smaller pieces so the model can still represent it using known vocabulary entries.

This is especially useful for technical terminology and newly formed words. The tokenizer does not need a separate vocabulary entry for every possible word combination.

Why Subwords Help

Common Pieces

Reusable Vocabulary

Many Possible Words

Remember: A tokenizer's exact splitting behavior is model-specific. Never assume that a word will always correspond to one particular set of pieces across different models.

  1. Intermediate: BPE, WordPiece, and SentencePiece

Modern tokenizers use algorithms that construct useful vocabulary pieces from training data. Three names you will frequently encounter are Byte Pair Encoding (BPE), WordPiece, and SentencePiece.

BPE

Builds a vocabulary by repeatedly combining frequently occurring symbol or subword pairs according to the tokenizer's training procedure.

WordPiece

Builds subword vocabularies using a likelihood-based training objective and is associated with models such as BERT.

SentencePiece

A tokenizer framework that can train subword models directly from raw text without requiring whitespace-based pre-tokenization.

These names should not be treated as interchangeable. They describe different algorithms or tokenizer frameworks, and real model tokenizers can include additional preprocessing, normalization, byte-level behavior, special tokens, or model-specific rules.

Conceptual Tokenizer Construction

Raw Text
Learn / Build Pieces
Vocabulary

  1. Intermediate: Special Tokens

Tokenizers can contain special tokens that are not ordinary pieces of natural language. These tokens help the model represent boundaries, padding, unknown values, or other control information.

BOS
Beginning-of-sequence information used by some tokenizers or models.
EOS
End-of-sequence information that can indicate that generation should stop.
PAD
Padding used by some batching strategies so sequences can share a common length.
UNK
An unknown-token representation used by tokenizers that need one.
Important: Not every modern LLM uses the same special tokens, and some models do not need every category listed above. Always inspect the tokenizer configuration for the model you are using.

  1. Intermediate: Tokenization and Context Length

LLMs process a finite sequence of tokens at a time. This available input capacity is commonly described as the model's context window or context length.

Context Window

Token
Token
Token
Token
Token
Limit

The model sees a sequence of tokens, not a fixed number of English words.

More Tokens
Longer sequences consume more of the available context and generally require more computation.
Fewer Tokens
More information can fit inside a fixed context budget when the tokenizer represents the input efficiently.
Practical Effect
Token count matters for context limits, latency, memory usage, and many API pricing systems.
⚡ Knowledge Check

Two prompts contain the same number of words. Can they use different numbers of tokens?

A) Yes, because token boundaries depend on the tokenizer▼
✓ Correct!Words can split into different numbers of tokens depending on the tokenizer, language, spelling, and surrounding characters.
B) No, one word always equals one token▼
✕ Incorrect.A word can correspond to one token, multiple tokens, or interact with whitespace and punctuation depending on the tokenizer.

  1. Intermediate: Tokenization Across Languages & Symbols

Tokenization is not equally efficient for every language or every type of content. Vocabulary coverage, script, training data, normalization, and tokenizer design can all affect how text is divided.

Multilingual Text

Different languages can require different numbers of tokens for similar semantic content. Script and vocabulary coverage strongly influence token efficiency.

Source Code

Brackets, operators, identifiers, indentation-related patterns, and common code fragments can all influence token boundaries.

Symbols & Punctuation

Numbers, punctuation, URLs, emojis, and unusual character sequences can tokenize differently from ordinary natural-language words.

Practical implication: Token counts should be measured with the tokenizer associated with the actual model you plan to use. Word counts are only rough estimates.

  1. Intermediate: Tokenization in Chat Models

A chat interface may look like it sends a simple conversation to the model, but the underlying system still needs to convert that conversation into a token sequence. Modern instruction-tuned models often use chat templates to format roles and messages before tokenization.

Conversation → Model Input

SystemYou are a helpful assistant.UserExplain tokenization.↓ chat template

Formatted conversation → Tokenizer → Token IDs

This explains why two chat models can receive the same visible messages but internally construct different token sequences. Their special tokens, role formatting, separators, and chat templates may differ.

Why Developers Should Care

When using an instruction-tuned model, the tokenizer and chat template are part of the model's expected interface. Passing the wrong format can change the model input and affect output quality.

  1. Advanced Preview: Tokenization Trade-Offs

Tokenization looks like preprocessing, but its design affects the entire LLM system. Vocabulary size, token sequence length, multilingual coverage, memory usage, context capacity, and inference cost are connected to the tokenizer.

Larger Vocabulary
  • • More complete pieces can be represented directly.
  • • May reduce token counts for some inputs.
  • • Increases vocabulary-related parameters.
  • • Does not automatically improve every language.
Smaller Vocabulary
  • • Uses fewer vocabulary entries.
  • • Can represent text using more pieces.
  • • May increase sequence length.
  • • Can still provide strong coverage through reusable subwords.
The Central Trade-Off

A tokenizer tries to balance vocabulary size, sequence length, and coverage. There is no single tokenization scheme that is universally optimal for every language, domain, and model.

⚡ Knowledge Check

Why can tokenizer design affect the cost and context capacity of an LLM application?

A) Because the model processes token sequences, not raw word counts▼
✓ Correct!More tokens can consume more context capacity and computation.
B) Because token IDs directly determine the model's intelligence▼
✕ Incorrect.Token IDs are identifiers. Their numeric values do not directly measure intelligence.

  1. Code: Tokenizing Text with Python

Now let's turn the concepts into code. In practice, you normally use the tokenizer associated with the model you are using. The example below uses Hugging Face Transformers to inspect how a sentence becomes tokens and token IDs.

InstallTerminal
pip install transformers
Example: Inspect Tokens and Token IDs
tokenize_text.pyPython 3.11+ · Hugging Face Transformers
from transformers import AutoTokenizer # Load a tokenizer associated with a pretrained model.tokenizer = AutoTokenizer.from_pretrained("gpt2") text = "Large language models are useful." # Convert text into token IDs.token_ids = tokenizer.encode(    text,    add_special_tokens=False) # Convert IDs back into readable token pieces.tokens = tokenizer.convert_ids_to_tokens(token_ids) print("Tokens:")print(tokens) print("\nToken IDs:")print(token_ids) print("\nNumber of tokens:", len(token_ids))
Output
Tokens:['Large', 'Ġlanguage', 'Ġmodels', 'Ġare', 'Ġuseful', '.'] Token IDs:[model-specific integer IDs] Number of tokens: 6

What to notice: The exact token IDs depend on the tokenizer. The important pipeline is text → tokens → token IDs.

encode()
Converts text into integer token IDs.
convert_ids_to_tokens()
Lets you inspect the token pieces represented by those IDs.
len(token_ids)
Counts how many tokens the input consumed for this tokenizer.
Try changing the text. Test punctuation, technical words, source code, URLs, multilingual text, and longer prompts. Token boundaries and token counts can change substantially between inputs.
A Tiny Conceptual Tokenizer

Before using a real model tokenizer, it can be useful to see the basic idea with a deliberately simple example. This is not how production LLM tokenizers work; it is only a learning aid.

tiny_tokenizer.pyPython 3.11+ · Conceptual Example
text = "hello world" vocabulary = {    "hello": 0,    "world": 1,} tokens = text.split()token_ids = [vocabulary[token] for token in tokens] print("Tokens:", tokens)print("Token IDs:", token_ids)
Output
Tokens: ['hello', 'world']Token IDs: [0, 1]

What to notice: The toy tokenizer looks up each word in its vocabulary and replaces it with the corresponding integer ID.

Why this toy example is limited: It assumes spaces define words and requires every word to exist in the vocabulary. Real tokenizers use much more sophisticated rules and algorithms so they can handle rare words, punctuation, code, multilingual text, and unseen combinations.

  1. Visual Explanation: From Text to the Transformer

Tokenization is only the beginning of the LLM pipeline. Once text has been converted into token IDs, those IDs can be used to retrieve learned token representations. The resulting vectors are then processed by the Transformer layers.

HUMAN TEXT / USER PROMPT
↓
TOKENIZERVocabulary + tokenization rules
↓
TOKEN SEQUENCESubwords + punctuation + special tokens
↓
TOKEN IDsInteger identifiers
↓
EMBEDDING LOOKUPToken IDs → learned vectors
↓
TRANSFORMERAttention + feed-forward processing
You Learned

How raw text becomes a numerical sequence that an LLM can process.

Next Module

Embeddings: how token IDs become learned numerical vectors.

Remember

The exact tokenization behavior is model-specific. Always use the tokenizer associated with the model.

  1. Beginner: Key Terms to Remember

Before moving to Embeddings, make sure these foundational terms are clear. They will appear repeatedly throughout the rest of the LLM course.

TermMeaning
TokenA unit produced by a tokenizer and represented by a token ID.
VocabularyThe collection of token entries known by a tokenizer.
Token IDAn integer identifier for a token vocabulary entry.
SubwordA reusable piece of a word or text sequence used by many modern tokenizers.
Special TokenA tokenizer entry used for control or structural information rather than ordinary text.
Context WindowThe amount of tokenized context a model can process in a particular request.
Chat TemplateA model-specific format that converts structured conversation messages into the sequence expected by the tokenizer and model.
Next Step: Now that you understand how an LLM converts text into tokens and token IDs, the next module is Embeddings & Representations. You will learn how those IDs become numerical vectors that the Transformer can actually process.
Module Complete

You now understand how an LLM turns human language into model-readable tokens.

You should now be comfortable with tokenization, token IDs, vocabulary, subword tokenization, BPE, WordPiece, SentencePiece, special tokens, context length, chat templates, and the relationship between tokenization and the rest of the LLM pipeline.