Automatic evaluation is the use of software or models to assess the quality, correctness, safety, or usefulness of a system's outputs. It may rely on programmatic checks, reference answers, metrics such as BLEU or ROUGE, or model-based evaluators called autoraters. Automatic evaluation is often used alongside human evaluation when outputs are complex or context-dependent.
Using software to judge the quality of a model's output. When model output is relatively straightforward, a script or program can compare the model's output to a golden response. This type of automatic evaluation is sometimes called programmatic evaluation . Metrics such as ROUGE or BLEU are often useful for programmatic evaluation. When model output is complex or has no one right answer, a separate ML program called an autorater sometimes performs the automatic evaluation. Contrast with human evaluation.