Also known as: WSC
WSC, или Winograd Schema Challenge, — это evaluation format и dataset для проверки способности language model разрешать pronoun references с использованием contextual reasoning. Каждый example содержит short passage, target pronoun и candidate noun phrases, а модель должна identify correct referent. Он используется как часть broader benchmarks, таких как SuperGLUE.
A format (or dataset conforming to that format) for evaluating an LLM's ability to determine the noun phrase that a pronoun refers to. Each entry in a Winograd Schema Challenge consists of: - A short passage, which contains a target pronoun - A target pronoun - Candidate noun phrases, followed by the correct answer (a Boolean). If the target pronoun refers to this candidate, the answer is True. If the target pronoun does not refer to this candidate, the answer is False. For example: - Passage: Mark told Pete many lies about himself, which Pete included in his book. He should have been more truthful. - Target pronoun: He - Candidate noun phrases : - Mark: True, because the target pronoun refers to Mark - Pete: False, because the target pronoun doesn't refer to Peter The Winograd Schema Challenge is a component of the SuperGLUE ensemble.