What is Top-p Sampling?

Rather than fixing a hard number of candidate tokens, top-p adapts to the shape of the distribution: when the model is highly confident, the nucleus might contain only one or two tokens; when it is uncertain across many plausible continuations, the nucleus widens to include more of them. A common setting like p=0.9 means the model samples only from tokens that together account for the top 90% of probability mass, discarding the long tail of unlikely options. Top-p is usually used together with, or instead of, temperature, and the two interact: temperature reshapes the probability distribution first, and top-p then trims which tokens are even eligible to be sampled from that reshaped distribution. Lowering p tightens output toward the most probable continuations and reduces the chance of an odd, low-probability token derailing a response; raising it allows more variety.