An evaluation hub for measuring model quality before deployment and monitoring model behavior after deployment.
31
Terms
10
Comparisons
8
Questions
Model evaluation and monitoring help teams decide whether a system is good enough to deploy and whether it remains safe and useful after release. The topic covers metrics, benchmarks, test design, thresholds, drift, and post-deployment evidence.
Poor evaluation can hide discriminatory outcomes, false alarms, unsafe recommendations, or exaggerated vendor claims. Monitoring closes the gap between laboratory results and real-world performance.
Useful for model cards, audit evidence, procurement review, high-risk system controls, post-market monitoring, and incident investigation.
Foundational concepts for measuring model quality and selecting evidence.
Terms for separating training, validation, testing, and generalization evidence.
Concepts for post-deployment monitoring and evaluating generated answers.
A side-by-side comparison of Precision and Recall. Understand how precision evaluates the correctness of predicted positives, while recall evaluates how many actual positives were found.
A side-by-side comparison of Accuracy and Precision. Understand why overall correctness can hide weak positive-class performance and why precision matters when false positives are costly.
A side-by-side comparison of Accuracy and F1 Score. Understand how overall correctness differs from a balanced measure of precision and recall.
A side-by-side comparison of F1 Score and Area Under the ROC Curve. Understand how a threshold-dependent balance of precision and recall differs from ranking discrimination across thresholds.
A side-by-side comparison of False Positive and False Negative. Understand which error type flags something that is absent and which error type misses something that is present.
A side-by-side comparison of Metric and Benchmark. Understand how the concepts differ, when each term applies, and why the distinction matters for AI governance, evaluation, or system design.
A side-by-side comparison of Evaluation and Benchmark. Understand how the concepts differ, when each term applies, and why the distinction matters for AI governance, evaluation, or system design.
A side-by-side comparison of Classification Threshold and Decision Threshold. Understand how threshold language applies to class prediction and broader discrete model decisions.
A side-by-side comparison of Model Monitoring and Evaluation. It explains how continuous observation of a deployed model differs from measuring a model, system, or change against defined criteria.
A side-by-side comparison of Data Drift and Model Drift. Understand how changes in input-data distribution differ from changes or degradation in model performance over time.
The F1 score is the harmonic mean of precision and recall, used to evaluate classification and information extraction performance. It ranges from 0 to 1, with higher values indicating a better balance between correctly identified positives and missed positives.
Use False Positive for an incorrect positive flag; use False Negative for an incorrect failure to detect a real positive condition.
A classification threshold is a chosen cutoff value used to convert a model's score or probability into a class prediction. Changing the threshold can alter the balance between false positives and false negatives, making threshold selection an important governance and performance decision.
Validation data is data used to evaluate a model during development and tune choices such as hyperparameters, thresholds, or learning processes. It should be distinct from training and test data to help detect underfitting, overfitting, and generalization problems.
A test set is a reserved dataset used to provide an independent evaluation of a trained model or AI system. It should be separated from training and validation data so that performance estimates better reflect behavior on unseen data before deployment or release.
Model monitoring is the continuous observation of a deployed model's performance, inputs, outputs, drift, errors, latency, and operational health. It helps detect degradation, misuse, bias, security issues, and conditions that require retraining or intervention.
Groundedness is the property of a model output being supported by specific source material or provided context. In generative AI, groundedness helps assess whether an answer is traceable to evidence rather than unsupported model generation.
UCR (unsupported-claim rate) is the percentage of claims in a model response that are not grounded in supporting evidence. A high UCR indicates that the system may be producing too many unsupported or hallucinated assertions.