What is an AI Inference Cost?

Unlike training cost, which is a large one-time expense to produce a model, inference cost recurs on every single call and scales with usage, making it the dominant ongoing expense for any product built on top of an LLM. Cost is driven by token volume in both directions: a long input context (documents, chat history, tool results fed back to the model) and a long generated output both add up, and agentic systems that make many sequential tool-augmented calls within one task can rack up substantially more tokens than a single chatbot turn. Managing inference cost in practice means routing tasks to the cheapest model capable of handling them well rather than defaulting every request to the largest, most expensive model, caching repeated context so it isn't re-processed on every call, and setting sensible limits on how many steps or retries an agentic loop can take before it's flagged for review. None of this should ever mean adding a timeout that cuts off an in-progress LLM call, cost control and call-completion guarantees are separate concerns.