Equalized odds is a fairness criterion requiring comparable error behavior across groups for both positive and negative outcomes. In classification, it generally requires the model's true positive and false positive rates, or equivalent conditional error rates, to be the same across protected groups.
A fairness metric to assess whether a model is predicting outcomes equally well for all values of a sensitive attribute with respect to both the positive class and negative class---not just one class or the other exclusively. In other words, both the true positive rate and false negative rate should be the same for all groups. Equalized odds is related to equality of opportunity, which only focuses on error rates for a single class (positive or negative). For example, suppose Glubbdubdrib University admits both Lilliputians and Brobdingnagians to a rigorous mathematics program. Lilliputians' secondary schools offer a robust curriculum of math classes, and the vast majority of students are qualified for the university program. Brobdingnagians' secondary schools don't offer math classes at all, and as a result, far fewer of their students are qualified. Equalized odds is satisfied provided that no matter whether an applicant is a Lilliputian or a Brobdingnagian, if they are qualified, they are equally as likely to get admitted to the program, and if they are not qualified, they are equally as likely to get rejected. Suppose 100 Lilliputians and 100 Brobdingnagians apply to Glubbdubdrib University, and admissions decisions are made as follows: Table 3. Lilliputian applicants (90% are qualified) Qualified 45 45 90 --- Table 4. Brobdingnagian applicants (10% are qualified): Qualified 5 5 10 --- Equalized odds is satisfied because qualified Lilliputian and Brobdingnagian students both have a 50% chance of being admitted, and unqualified Lilliputian and Brobdingnagian have an 80% chance of being rejected. [!NOTE] Note: While equalized odds is satisfied here, demographic parity is not satisfied. Lilliputian and Brobdingnagian students are admitted to Glubbdubdrib University at different rates; 47% of Lilliputian students are admitted, and 23% of Brobdingnagian students are admitted. Equalized odds is formally defined in "Equality of Opportunity in Supervised Learning" as follows: "predictor satisfies equalized odds with respect to protected attribute A and outcome Y if and A are independent, conditional on Y." [!NOTE] Note: Contrast equalized odds with the more relaxed equality of opportunity metric.