What is an Inference?

A model's lifecycle splits cleanly into two phases. Training is expensive and happens rarely: the model's weights are iteratively adjusted against a large dataset until it captures useful patterns. Inference is what happens every single time the model is actually used afterward — a prompt goes in, weights don't change, and an output comes out. When you send a message to a chatbot or an agent calls a model to decide its next step, that's an inference call. Inference cost and latency are what most production AI systems are actually optimized around day to day, since training is a one-time (or periodic) event but inference happens on every single request at scale. Techniques like batching, caching, smaller distilled models, and quantization all exist specifically to make inference cheaper or faster without touching the training process at all. When people talk about a model's “speed” or “cost per token” in a live product, they are almost always talking about inference, not training.