What is Rate Limiting?

Most APIs enforce rate limits using algorithms like token bucket or sliding window: each client gets an allotment of requests per second or minute, and requests beyond that allotment are rejected (typically with an HTTP 429 status) or queued until capacity frees up. This protects backend infrastructure from being overwhelmed by a single noisy client and ensures fair access across many consumers of a shared API. For systems that call external LLM or third-party APIs, rate limits are a first-class operational concern rather than an edge case: exceeding a provider's limit can silently degrade an entire pipeline if it isn't handled deliberately. Well-built integrations read the rate-limit headers a provider returns, back off and retry with jitter rather than hammering the endpoint, and often implement their own internal throttling to stay comfortably under a provider's ceiling — especially important when many concurrent agent sessions all call the same upstream model. Rate limiting differs from simple retry logic in intent: retries handle transient failures, while rate limiting proactively paces requests to avoid triggering those failures in the first place.