Also known as: evaluation (eval)
Evaluation is the process of measuring the quality, behavior, or performance of a model, system, or change against defined criteria. In machine learning it may use validation and test data, while LLM evaluation may also include safety, factuality, robustness, and user-impact assessments.
The process of measuring a model's quality or comparing different models against each other. To evaluate a supervised machine learning model, you typically judge it against a validation set and a test set. Evaluating a LLM typically involves broader quality and safety assessments.
Amazon Machine Learning: The process of measuring the predictive performance of a machine learning (ML) model. Also a machine learning object that stores the details and result of an ML model evaluation.
An eval, short for "evaluation", is a type of experiment in which logged or synthetic queries are sent through two Search stacks--an experimental stack that includes your change and a base stack without your change. Evals produce diffs and metrics that let you evaluate the impact, quality, and other effects of your change on search results and other parts of the Google user experience. Evals are used during tuning, or iterations, on your change. They are also used as part of launching a change to live user traffic.
A side-by-side comparison of Model Monitoring and Evaluation. It explains how continuous observation of a deployed model differs from measuring a model, system, or change against defined criteria.
A side-by-side comparison of Evaluation and Benchmark. Understand how the concepts differ, when each term applies, and why the distinction matters for AI governance, evaluation, or system design.