What is an RLHF (Reinforcement Learning from Human Feedback)?

A base language model pretrained on internet text can produce fluent but unhelpful, unsafe, or off-target responses, because next-token prediction alone doesn't encode what makes an answer genuinely good for a person. RLHF closes that gap in a few stages: human labelers rank multiple candidate responses to the same prompt from best to worst, a separate reward model is trained to predict those human preference rankings, and then the language model itself is fine-tuned with reinforcement learning to maximize the score that reward model assigns — effectively teaching the model to produce the kind of response humans preferred, generalized across far more examples than the human raters directly reviewed. This is the technique that turned raw pretrained models into the helpful, instruction-following assistants that ship in production — it's a large part of why a model that has gone through RLHF follows instructions, refuses harmful requests, and matches a conversational tone far more reliably than the same model's untuned base checkpoint. RLHF has known limitations: the reward model can be gamed (a phenomenon called reward hacking, where the policy learns to produce outputs the reward model over-scores without the improvement being real), and the technique bakes in whatever biases existed in the human raters' preferences, both of which are active areas of alignment research alongside newer methods like RL from AI feedback and reward-free preference optimization.