Embeddings & Representations

Learn how token IDs become dense numerical vectors, why embeddings represent semantic information, how embedding tables work, positional information, similarity, dimensions, and how representations flow into Transformer layers.

30 minBeginner

The Core Idea: Tokenization converts text into token IDs, but token IDs themselves are only identifiers. An LLM needs a richer numerical representation. Embeddings map token IDs to dense vectors that the neural network can process and transform through its layers.

  1. Beginner: What Is an Embedding?

An embedding is a numerical vector used to represent an item in a continuous mathematical space. In an LLM, token embeddings provide a learned vector representation for each token in the model's vocabulary.

Remember the previous module: a tokenizer converts text into token IDs. Those IDs are not meaningful numbers by themselves. The embedding layer uses each ID as an index into a learned table of vectors.

Token ID → Embedding Vector

Token
"language"
↓
Tokenizer
token ID = 1234
↓
Embedding Table Lookup
row 1234
↓
Dense Vector
[0.12, -0.48, 0.77, ...]
The vector has a fixed dimensionality chosen by the model architecture.
Input
Token ID

An integer used to identify a vocabulary entry.

Lookup
Embedding Table

A learned matrix containing one vector for each vocabulary entry.

Output
Vector

A dense numerical representation that can enter the Transformer.

Important: The embedding vector is learned during model training. Its individual dimensions are not normally assigned simple human-readable meanings such as "grammar" or "friendliness."
⚡ Knowledge Check

What is the main purpose of a token embedding?

A) Convert a token ID into a learned dense vector▼
✓ Correct!The embedding lookup provides the numerical vector representation used by the model.
B) Convert a vector back into the original sentence▼
✕ Incorrect.Embeddings are representations used inside the model; they are not a direct sentence decoder.

  1. Beginner: The Embedding Matrix

The token embedding layer can be viewed as a matrix. Each row corresponds to one vocabulary entry, and each row contains the learned vector associated with that token.

Conceptual Embedding Matrix

Token
d₁
d₂
d₃
...
hello
0.21
-0.14
0.73
...
language
-0.42
0.58
0.11
...
model
0.08
-0.67
0.32
...
...
...
...
...
...

If a vocabulary contains V token entries and the embedding dimension is D, the conceptual shape of the embedding matrix is V × D.

Rows

One row for each vocabulary entry.

Columns

The dimensions of each embedding vector.

Lookup

A token ID selects one row rather than requiring a full matrix multiplication.

Why this matters: The embedding matrix is part of the model's learned parameters. During training, its values are updated so that the representations become useful for the model's overall objective.

  1. Beginner: Why Vectors Instead of Token IDs?

Token IDs are categorical identifiers. If one token has ID 100 and another has ID 101, the numerical difference between those IDs does not mean the two tokens are semantically close.

Why Raw IDs Are Not Meaningful Coordinates

Token A
ID 100
≠ semantic distance
Token B
ID 101

Dense vectors give the neural network a continuous numerical representation. Their geometry can then participate in operations such as dot products, projections, additions, normalization, and attention.

Token ID

A lookup key. It tells the model which vocabulary entry is being referenced.

Embedding Vector

A learned numerical representation that can participate in neural-network computation.

⚡ Knowledge Check

If token A has ID 500 and token B has ID 501, what can you conclude?

A) They are semantically very similar▼
✕ Incorrect.The numerical IDs are identifiers and do not encode semantic distance.
B) They are simply two vocabulary entries▼
✓ Correct!The IDs tell us which vocabulary entries are selected, not how close their meanings are.

  1. Beginner: Embedding Dimensions

Every token embedding has a fixed number of dimensions for a particular model. If the model's hidden size is D, each token starts with a vector containing D numerical values.

One Token → One Vector

Token ID→Embedding Lookup→[x₁, x₂, x₃, ..., xD]

A larger embedding dimension provides more numerical capacity for representing information, but it also increases parameter count and computation. The dimension is an architectural choice, not something chosen independently for each token.

Fixed Per Model

Every token starts with vectors of the same dimensionality inside a given model.

Learned Values

The values are learned parameters rather than manually assigned features.

Model-Specific

Different model architectures can use different embedding dimensions.

  1. Intermediate: Semantic Similarity and Vector Geometry

One useful property of learned representations is that relationships can be studied geometrically. A common operation is cosine similarity, which compares the direction of two vectors.

Conceptual Vector Space

vector A

vector B

vector C

conceptual only — not a literal 2D embedding

If two vectors point in similar directions, their cosine similarity can be high. This can be useful in information retrieval and embedding-based search, although token embeddings inside an LLM should not be treated as a perfect semantic dictionary.

High Similarity

Vectors point in relatively similar directions.

Low Similarity

Vectors point in substantially different directions.

Important Caveat

A representation's meaning depends on the model, layer, context, and training objective.

Embedding Models vs LLM Token Embeddings

Dedicated embedding models are often designed to produce useful vectors for retrieval, clustering, or similarity search. An LLM's internal token embeddings serve a different role: they are the starting representations that are transformed repeatedly as information moves through the network.

  1. Intermediate: Contextual Representations

A token's initial embedding is not the final representation used by the Transformer. As the sequence passes through attention and feed-forward layers, the representation becomes contextual.

Same Token, Different Context

Context A
"river bank"
The representation of "bank" is influenced by its surrounding context.
Context B
"bank account"
The same token or token pieces can receive different contextual information.

The important distinction is static starting embeddings vs contextual hidden states. The embedding lookup initially associates each token ID with a learned vector. Attention and the rest of the Transformer then transform those vectors based on the entire sequence.

Initial
Token embedding selected from the learned embedding table.
Context
Attention allows information from other positions to influence the representation.
Hidden State
The resulting representation carries information shaped by the current context.

  1. Intermediate: Position and Embeddings

Token embeddings tell the model what token is present, but a sequence also needs information about where each token occurs. This is the role of positional information.

Token Information + Position Information

Token Embedding
"cat" → learned vector
+
Positional Information
position 5 → position representation
↓
Transformer Input Representation
A representation containing token and positional information according to the architecture.

Different Transformer architectures handle positional information differently. Later in the course, you will encounter methods such as learned positional embeddings, sinusoidal positional encodings, and Rotary Positional Embeddings (RoPE).

Connection to the next modules: Embeddings answer "what is represented?" while positional mechanisms provide information needed to distinguish different positions in a sequence. Attention then uses these representations to model relationships between positions.

  1. Intermediate: Embeddings in the Full LLM Pipeline

You can now connect the previous Tokenization module to the upcoming Transformer concepts. The embedding layer sits directly between token IDs and the neural-network computations that follow.

HUMAN TEXT

↓

TOKENIZER Text → tokens → token IDs

↓

TOKEN IDs

↓

TOKEN EMBEDDING LOOKUP IDs → dense vectors

↓

POSITIONAL INFORMATION

↓

TRANSFORMER LAYERS Attention + feed-forward processing

Before Embeddings
The model has token IDs, which are identifiers rather than useful continuous features.
Embedding Stage
Token IDs select learned dense vectors from the embedding table.
After Embeddings
The resulting representations enter the Transformer computation.

  1. Intermediate: Token Embeddings vs Sentence Embeddings

The word embedding is used in several contexts, so it is important not to confuse token embeddings inside an LLM with embeddings produced by dedicated embedding models.

Token Embeddings
  • • Usually one initial vector per token position.
  • • Stored as learned model parameters.
  • • Feed into the Transformer computation.
  • • Become transformed contextual representations.
Sentence / Text Embeddings
  • • Usually represent a larger piece of text with one vector.
  • • Often produced by dedicated embedding models.
  • • Common in semantic search and RAG.
  • • Designed to make text similarity useful for downstream applications.
Why this distinction matters: When you later learn RAG, you will work with text embeddings and vector databases. Those application-level embeddings should not automatically be assumed to be the same thing as the token embedding matrix inside a generative LLM.

  1. Code: Inspecting Token Embeddings with Python

You can inspect the embedding matrix of a small pretrained Transformer using the Hugging Face Transformers library. This example is intended for understanding the mechanism, not for training a model.

Install the library
pip install transformers torch
Example: Token ID to Embedding Vector
import torchfrom transformers import AutoTokenizer, AutoModel model_name = "distilbert-base-uncased"tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModel.from_pretrained(model_name) text = "Large language models are useful."inputs = tokenizer(text, return_tensors="pt") input_ids = inputs["input_ids"]embedding_layer = model.get_input_embeddings() with torch.no_grad():    embeddings = embedding_layer(input_ids) print("Input IDs shape:", input_ids.shape)print("Embedding shape:", embeddings.shape)print("First token vector:", embeddings[0, 0])
Output
Input IDs shape: torch.Size([1, N])Embedding shape: torch.Size([1, N, 768])First token vector: tensor([...])

What to notice: The middle dimension represents the token sequence length, while 768 is the hidden/embedding dimension of this particular DistilBERT architecture. The exact sequence length depends on tokenization.

input_ids
Integer token IDs produced by the tokenizer.
embedding_layer
The learned lookup layer that maps IDs to vectors.
embeddings
A tensor containing one vector for each token position.
Inspecting the Embedding Table Shape
weight = embedding_layer.weightprint("Embedding table shape:", weight.shape)
Expected idea: the embedding table has shape [vocabulary size, embedding dimension]. The first dimension tells you how many vocabulary entries can be looked up; the second tells you the size of each vector.

  1. Visual Explanation: From Token ID to Representation

The entire idea can be summarized as a lookup followed by neural-network processing. The token ID selects a row from the embedding matrix; the resulting vector becomes part of the model's initial representation of the sequence.

TOKEN ID e.g. 1234

↓

EMBEDDING MATRIX Vocabulary × Embedding Dimension

↓

SELECTED VECTOR [x₁, x₂, x₃, ..., xD]

↓

POSITIONAL INFORMATION

↓

TRANSFORMER LAYERS Attention transforms representations using context

Tokenization
Answers: "Which vocabulary entries are present?"
Embedding
Answers: "What learned vector starts the representation for each token?"
Transformer
Answers: "How should these representations interact given the context?"

  1. Beginner: Key Terms to Remember

Before moving to Transformers and Attention, make sure these concepts are clear.

TermMeaning
EmbeddingA dense numerical vector representation learned by a model.
Embedding MatrixA learned table containing one embedding vector for each vocabulary entry.
Embedding DimensionThe number of numerical values in each token embedding vector.
Token IDAn integer identifier used to select a vocabulary entry and its embedding.
Contextual RepresentationA representation transformed by the network using information from the surrounding sequence.
Positional InformationInformation that helps the architecture distinguish token positions in a sequence.
Sentence EmbeddingA vector representing a larger text span, commonly used in retrieval and similarity applications.
Next Step: You now know how text becomes token IDs and how those IDs become learned vectors. The next step is to understand Transformers, including self-attention, queries, keys, values, positional information, and how representations change as they pass through the network.
Module Complete

You now understand how token IDs become learned numerical representations.

You should now be comfortable with embeddings, embedding matrices, dimensions, token IDs, vector geometry, contextual representations, positional information, token embeddings versus text embeddings, and the role of embeddings in the LLM pipeline.