What is a Model Evaluation?

Evaluation typically compares a model's outputs against reference answers or scoring rubrics using metrics like accuracy, F1 score, BLEU/ROUGE for text, or task-completion rate for agentic systems. For large language models, evaluation increasingly relies on a mix of automated benchmarks (MMLU, HumanEval, GSM8K) and LLM-as-judge scoring, since many outputs (summaries, code, agent plans) don't have a single correct answer that a simple string match can grade. Good evaluation practice separates offline evaluation (run before a model change ships, against a fixed test set) from online evaluation (monitoring live traffic for regressions, drift, or unexpected failure modes after deployment). Without both, teams either ship regressions that offline tests missed or fail to notice quality decay as usage patterns shift over time.