What is Pre-training?

During pre-training, a model (typically a transformer) is trained on a next-token-prediction objective across a huge, diverse corpus — web text, books, code, and more — often trillions of tokens, using enormous compute budgets measured in thousands of GPU or accelerator-years. This phase is where the model acquires most of what looks like "general knowledge" and language competence; it has no notion yet of following instructions, being helpful, or refusing harmful requests. After pre-training, models go through further stages — supervised fine-tuning on instruction-following examples, and typically reinforcement learning from human (or AI) feedback — to shape the raw pre-trained model into something that behaves like a helpful assistant rather than just a very capable text-completion engine. The pre-trained model is sometimes called the "base model," and it behaves quite differently from the aligned, instruction-tuned model most users actually interact with. Pre-training is also where a model's factual knowledge cutoff comes from: whatever was in the training corpus up to the point data collection stopped is what the model "knows" without external tools, which is why retrieval and tool use matter for anything beyond that cutoff.