The Core Idea: Tokenization converts text into token IDs, but token IDs themselves are only identifiers. An LLM needs a richer numerical representation. Embeddings map token IDs to dense vectors that the neural network can process and transform through its layers.
- Beginner: What Is an Embedding?
An embedding is a numerical vector used to represent an item in a continuous mathematical space. In an LLM, token embeddings provide a learned vector representation for each token in the model's vocabulary.
Remember the previous module: a tokenizer converts text into token IDs. Those IDs are not meaningful numbers by themselves. The embedding layer uses each ID as an index into a learned table of vectors.
Token ID → Embedding Vector
An integer used to identify a vocabulary entry.
A learned matrix containing one vector for each vocabulary entry.
A dense numerical representation that can enter the Transformer.
What is the main purpose of a token embedding?
A) Convert a token ID into a learned dense vector▼
B) Convert a vector back into the original sentence▼
- Beginner: The Embedding Matrix
The token embedding layer can be viewed as a matrix. Each row corresponds to one vocabulary entry, and each row contains the learned vector associated with that token.
Conceptual Embedding Matrix
If a vocabulary contains V token entries and the embedding dimension is D, the conceptual shape of the embedding matrix is V × D.
One row for each vocabulary entry.
The dimensions of each embedding vector.
A token ID selects one row rather than requiring a full matrix multiplication.
- Beginner: Why Vectors Instead of Token IDs?
Token IDs are categorical identifiers. If one token has ID 100 and another has ID 101, the numerical difference between those IDs does not mean the two tokens are semantically close.
Why Raw IDs Are Not Meaningful Coordinates
Dense vectors give the neural network a continuous numerical representation. Their geometry can then participate in operations such as dot products, projections, additions, normalization, and attention.
A lookup key. It tells the model which vocabulary entry is being referenced.
A learned numerical representation that can participate in neural-network computation.
If token A has ID 500 and token B has ID 501, what can you conclude?
A) They are semantically very similar▼
B) They are simply two vocabulary entries▼
- Beginner: Embedding Dimensions
Every token embedding has a fixed number of dimensions for a particular model. If the model's hidden size is D, each token starts with a vector containing D numerical values.
One Token → One Vector
A larger embedding dimension provides more numerical capacity for representing information, but it also increases parameter count and computation. The dimension is an architectural choice, not something chosen independently for each token.
Every token starts with vectors of the same dimensionality inside a given model.
The values are learned parameters rather than manually assigned features.
Different model architectures can use different embedding dimensions.
- Intermediate: Semantic Similarity and Vector Geometry
One useful property of learned representations is that relationships can be studied geometrically. A common operation is cosine similarity, which compares the direction of two vectors.
Conceptual Vector Space
vector A
vector B
vector C
conceptual only — not a literal 2D embedding
If two vectors point in similar directions, their cosine similarity can be high. This can be useful in information retrieval and embedding-based search, although token embeddings inside an LLM should not be treated as a perfect semantic dictionary.
Vectors point in relatively similar directions.
Vectors point in substantially different directions.
A representation's meaning depends on the model, layer, context, and training objective.
Dedicated embedding models are often designed to produce useful vectors for retrieval, clustering, or similarity search. An LLM's internal token embeddings serve a different role: they are the starting representations that are transformed repeatedly as information moves through the network.
- Intermediate: Contextual Representations
A token's initial embedding is not the final representation used by the Transformer. As the sequence passes through attention and feed-forward layers, the representation becomes contextual.
Same Token, Different Context
The important distinction is static starting embeddings vs contextual hidden states. The embedding lookup initially associates each token ID with a learned vector. Attention and the rest of the Transformer then transform those vectors based on the entire sequence.
- Intermediate: Position and Embeddings
Token embeddings tell the model what token is present, but a sequence also needs information about where each token occurs. This is the role of positional information.
Token Information + Position Information
Different Transformer architectures handle positional information differently. Later in the course, you will encounter methods such as learned positional embeddings, sinusoidal positional encodings, and Rotary Positional Embeddings (RoPE).
- Intermediate: Embeddings in the Full LLM Pipeline
You can now connect the previous Tokenization module to the upcoming Transformer concepts. The embedding layer sits directly between token IDs and the neural-network computations that follow.
HUMAN TEXT
TOKENIZER Text → tokens → token IDs
TOKEN IDs
TOKEN EMBEDDING LOOKUP IDs → dense vectors
POSITIONAL INFORMATION
TRANSFORMER LAYERS Attention + feed-forward processing
- Intermediate: Token Embeddings vs Sentence Embeddings
The word embedding is used in several contexts, so it is important not to confuse token embeddings inside an LLM with embeddings produced by dedicated embedding models.
- • Usually one initial vector per token position.
- • Stored as learned model parameters.
- • Feed into the Transformer computation.
- • Become transformed contextual representations.
- • Usually represent a larger piece of text with one vector.
- • Often produced by dedicated embedding models.
- • Common in semantic search and RAG.
- • Designed to make text similarity useful for downstream applications.
- Code: Inspecting Token Embeddings with Python
You can inspect the embedding matrix of a small pretrained Transformer using the Hugging Face Transformers library. This example is intended for understanding the mechanism, not for training a model.
pip install transformers torchimport torchfrom transformers import AutoTokenizer, AutoModel model_name = "distilbert-base-uncased"tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModel.from_pretrained(model_name) text = "Large language models are useful."inputs = tokenizer(text, return_tensors="pt") input_ids = inputs["input_ids"]embedding_layer = model.get_input_embeddings() with torch.no_grad(): embeddings = embedding_layer(input_ids) print("Input IDs shape:", input_ids.shape)print("Embedding shape:", embeddings.shape)print("First token vector:", embeddings[0, 0])Input IDs shape: torch.Size([1, N])Embedding shape: torch.Size([1, N, 768])First token vector: tensor([...])What to notice: The middle dimension represents the token sequence length, while 768 is the hidden/embedding dimension of this particular DistilBERT architecture. The exact sequence length depends on tokenization.
weight = embedding_layer.weightprint("Embedding table shape:", weight.shape)
- Visual Explanation: From Token ID to Representation
The entire idea can be summarized as a lookup followed by neural-network processing. The token ID selects a row from the embedding matrix; the resulting vector becomes part of the model's initial representation of the sequence.
TOKEN ID e.g. 1234
EMBEDDING MATRIX Vocabulary × Embedding Dimension
SELECTED VECTOR [x₁, x₂, x₃, ..., xD]
POSITIONAL INFORMATION
TRANSFORMER LAYERS Attention transforms representations using context
- Beginner: Key Terms to Remember
Before moving to Transformers and Attention, make sure these concepts are clear.
| Term | Meaning |
|---|---|
| Embedding | A dense numerical vector representation learned by a model. |
| Embedding Matrix | A learned table containing one embedding vector for each vocabulary entry. |
| Embedding Dimension | The number of numerical values in each token embedding vector. |
| Token ID | An integer identifier used to select a vocabulary entry and its embedding. |
| Contextual Representation | A representation transformed by the network using information from the surrounding sequence. |
| Positional Information | Information that helps the architecture distinguish token positions in a sequence. |
| Sentence Embedding | A vector representing a larger text span, commonly used in retrieval and similarity applications. |
You now understand how token IDs become learned numerical representations.
You should now be comfortable with embeddings, embedding matrices, dimensions, token IDs, vector geometry, contextual representations, positional information, token embeddings versus text embeddings, and the role of embeddings in the LLM pipeline.