Quantization

Quantization is a compression technique that reduces size and computation by lowering the precision of the numbers inside a model. Instead of storing weights as 16-digit decimals, you store them as 4- to 8-bit integers — similar to lowering a photo's quality a notch to shrink its file size drastically.

The key point is that performance barely drops even after cutting precision. The fact that LLMs can run on laptops and phones, and the community culture of running open models on personal PCs, all exist thanks to quantization. When you see a phrase like "4-bit quantized version released," read it as "now you can run this on your home computer."

Both get described as 'making a model lighter,' so they look like the same technique. But one means training a separate, smaller model to learn from a larger one's abilities, while the other means shrinking the size of an existing model by coarsening its numbers. They're often combined too — distill down to a small model, then quantize it further.