The Core Idea: A model must fit in GPU memory. Memory = parameters × bytes per number, so using fewer bits (quantization) lets the same model fit on a much smaller GPU with only a small quality loss.
- Beginner: Why Memory Is the First Question
Before asking how smart is this model?, practitioners ask will it even fit on my GPU? A model is just billions of numbers (parameters), and every number must live in the GPU's fast memory, called VRAM, while the model runs.
So a model with 7 billion parameters needs 7 billion times the size of one number. How big one number is depends on its precision.
Analogy: Think of a photo album. Each photo is a parameter. A high-resolution print takes more shelf space than a thumbnail. Lowering precision is like storing thumbnails: you can fit many more photos, but each loses some detail.
- Beginner: Number Formats (Precision)
| Format | Bits | Bytes per number | 7B model weights | Typical use |
|---|---|---|---|---|
| float32 (FP32) | 32 | 4 | 28 GB | Older training, scientific work |
| float16 / bfloat16 | 16 | 2 | 14 GB | Standard for training and inference |
| int8 | 8 | 1 | 7 GB | Quantized inference |
| int4 | 4 | 0.5 | 3.5 GB | Aggressive quantization, local models |
bfloat16 keeps the same range as float32 (so training rarely overflows) but with less precision, which is why it became the favorite for deep learning. float16 has more precision but a smaller range.
Easy mental math: Weights in gigabytes ≈ parameters in billions × bytes per number. A 13B model in 16-bit is about 26 GB.
- Medium: What Else Uses Memory?
Weights are only part of the story. Running a model also needs:
| Item | What it is | Grows with |
|---|---|---|
| Weights | The learned parameters | Model size and precision |
| KV cache | Stored keys and values of earlier tokens | Context length and number of users |
| Activations | Temporary results inside the layers | Batch size |
| Overhead | Framework buffers and fragmentation | Roughly 10 to 20 percent extra |
Training needs far more than inference. With the common Adam optimizer in mixed precision, each parameter needs about 16 bytes: 2 for the weights, 2 for the gradient, 4 for a 32-bit master copy and 8 for the two optimizer states. A 7B model therefore needs about 112 GB to train fully, versus 14 GB to run.
Rule of thumb for inference: Plan for weights × 1.2, plus the KV cache for your longest conversations.
- Medium: Quantization, Shrinking the Numbers
Quantization stores weights with fewer bits. Instead of a precise decimal like 0.7312, we keep a small integer and one shared scale number, then multiply to recover an approximation:
Why it works: Neural network weights are noisy and redundant. A small rounding error in each weight usually changes the output very little, so quality drops only slightly.
- Advanced: Group-Wise Methods and the Quality Trade-Off
A single scale for a whole matrix is crude: one huge weight forces a coarse scale on everyone. Group-wise quantization uses a separate scale for each small group (for example 64 or 128 weights), which fits the values much better and is used by most 4-bit methods.
| Approach | Idea | Where you will see it |
|---|---|---|
| Post-training quantization | Quantize an already trained model with no retraining | GPTQ, AWQ, GGUF files |
| 4-bit NormalFloat (NF4) | A 4-bit format designed for bell-shaped weights | QLoRA fine-tuning |
| Quantization-aware training | Train while simulating the rounding | Best quality, higher cost |
| KV-cache quantization | Compress the cache, not the weights | Long-context serving |
- Visual Explanation: Bytes, Bits and Rounding
The figure first compares one 7-billion-parameter model at three precisions, then shows the quantize and dequantize round trip for a single weight.
Click to see why group-wise is betterCompare ▼
Part 1 reads left to right: fewer bytes per number means a smaller model. Part 2 shows that quantizing stores a small integer and a scale, and using it multiplies back to an approximation. Open the bar to see why groups help.
- Python Implementation: Memory Math and Quantization From Scratch
This script calculates how much memory 7B and 70B models need at four precisions, quantizes a random weight matrix to int8 and int4, and compares one global scale with group-wise scales.
import numpy as np # 1. Memory needed just to store the weightsdef weights_gb(params_billion, bits): return params_billion * 1e9 * bits / 8 / 1e9 for bits, name in [(32, "float32"), (16, "float16 / bfloat16"), (8, "int8"), (4, "int4")]: print(name.ljust(18), "7B model:", round(weights_gb(7, bits), 1), "GB | 70B model:", round(weights_gb(70, bits), 1), "GB") # 2. Symmetric int8 quantization of a weight matrixrng = np.random.default_rng(0)W = rng.normal(size=(256, 256)).astype(np.float32) def quantize(W, bits): qmax = 2 ** (bits - 1) - 1 # 127 for int8, 7 for int4 scale = np.abs(W).max() / qmax q = np.clip(np.round(W / scale), -qmax, qmax) return q, scale def error_after(W, bits): q, scale = quantize(W, bits) return float(np.mean((W - q * scale) ** 2)) print("Mean squared error int8:", format(error_after(W, 8), ".2e"))print("Mean squared error int4:", format(error_after(W, 4), ".2e")) # 3. Group-wise quantization: one scale per small group of 64 weights fits betterdef error_groupwise(W, bits, group=64): flat = W.reshape(-1, group) qmax = 2 ** (bits - 1) - 1 scales = np.abs(flat).max(axis=1, keepdims=True) / qmax q = np.clip(np.round(flat / scales), -qmax, qmax) return float(np.mean((flat - q * scales) ** 2)) print("int4 with one scale for everything:", format(error_after(W, 4), ".2e"))print("int4 with one scale per 64 weights:", format(error_groupwise(W, 4), ".2e"))The first four lines are the memory table. Then the error numbers show int8 is about 300 times more accurate than int4 in this test, and that group-wise scales cut the int4 error to about a third.
Interactive Knowledge Check
Key Points
The Big Picture
Fewer bits per number → smaller model → fits on smaller hardware → small quality cost
Knowing the memory math lets you choose a model and a GPU in minutes instead of by trial and error.
Up Next: Now that you can size a model, you need to talk to one. In Prompt Engineering you will learn how to write instructions that reliably get the answers you want.