What is a Model Quantization?

Full-precision models store each weight as a 16- or 32-bit float, which for a large model can mean tens of gigabytes of memory just to hold the weights. Quantization maps those values onto a smaller set of discrete levels, cutting memory and often compute requirements roughly in proportion to the bit-width reduction, at some cost to output quality. Common schemes include post-training quantization (applied to an already-trained model, fast but slightly lossy) and quantization-aware training (the model learns to compensate for reduced precision during training itself, better quality but more expensive to produce). The practical payoff is running larger models on smaller hardware: a 4-bit quantized 70B model can fit on a single consumer GPU where the full-precision version would need multiple data-center cards. This matters most for on-device and edge deployment, and for cost-sensitive inference at scale where every percentage of memory and latency reduction compounds across millions of requests.