What is an Agent Evaluation?

Unlike testing deterministic software, evaluating an agent has to account for variability — the same prompt can produce a differently worded but equally correct response, so evaluation criteria usually judge outcomes and behavior (did it call the right tool, did it reach a factually correct answer, did it avoid an unsafe action) rather than checking for an exact string match. Evaluation sets typically include both "golden" tasks with known correct answers and adversarial cases designed to probe failure modes like prompt injection or unsafe tool use. Evaluation isn't a one-time gate before launch — it has to run continuously, because an underlying model update, a new tool integration, or a changed prompt can silently shift behavior. Mature teams re-run their evaluation suite on every meaningful change and track pass rates over time the same way they'd track a test-coverage metric for regular code, treating a regression in agent evaluation results as seriously as a broken CI build.