Also known as: WSC
WSC, or Winograd Schema Challenge, is an evaluation format and dataset for testing whether a language model can resolve pronoun references using contextual reasoning. Each example presents a short passage, a target pronoun, and candidate noun phrases, with the model expected to identify the correct referent. It is used as part of broader benchmarks such as SuperGLUE.
A format (or dataset conforming to that format) for evaluating an LLM's ability to determine the noun phrase that a pronoun refers to. Each entry in a Winograd Schema Challenge consists of: - A short passage, which contains a target pronoun - A target pronoun - Candidate noun phrases, followed by the correct answer (a Boolean). If the target pronoun refers to this candidate, the answer is True. If the target pronoun does not refer to this candidate, the answer is False. For example: - Passage: Mark told Pete many lies about himself, which Pete included in his book. He should have been more truthful. - Target pronoun: He - Candidate noun phrases : - Mark: True, because the target pronoun refers to Mark - Pete: False, because the target pronoun doesn't refer to Peter The Winograd Schema Challenge is a component of the SuperGLUE ensemble.