Intermediate
What is an F1 score?
The F1 score is the harmonic mean of precision and recall, used to evaluate classification and information extraction performance. It ranges from 0 to 1, with higher values indicating a better balance between correctly identified positives and missed positives.
How are false positives and false negatives different?
Use False Positive for an incorrect positive flag; use False Negative for an incorrect failure to detect a real positive condition.
What is a classification threshold?
A classification threshold is a chosen cutoff value used to convert a model's score or probability into a class prediction. Changing the threshold can alter the balance between false positives and false negatives, making threshold selection an important governance and performance decision.
What is validation data?
Validation data is data used to evaluate a model during development and tune choices such as hyperparameters, thresholds, or learning processes. It should be distinct from training and test data to help detect underfitting, overfitting, and generalization problems.
What is a test set?
A test set is a reserved dataset used to provide an independent evaluation of a trained model or AI system. It should be separated from training and validation data so that performance estimates better reflect behavior on unseen data before deployment or release.
What is model monitoring?
Model monitoring is the continuous observation of a deployed model's performance, inputs, outputs, drift, errors, latency, and operational health. It helps detect degradation, misuse, bias, security issues, and conditions that require retraining or intervention.
Advanced
What is groundedness in LLM evaluation?
Groundedness is the property of a model output being supported by specific source material or provided context. In generative AI, groundedness helps assess whether an answer is traceable to evidence rather than unsupported model generation.
What is unsupported-claim rate?
UCR (unsupported-claim rate) is the percentage of claims in a model response that are not grounded in supporting evidence. A high UCR indicates that the system may be producing too many unsupported or hallucinated assertions.