Also known as: Pass At K (pass@k) Β· pass@k Β· Pass At K pass@k
Pass@k is an evaluation metric for generated code that estimates whether at least one of k generated solutions passes the required tests. It is useful when language models produce multiple candidate programs for the same problem.
A metric to determine the quality of code (for example, Python) that a large language model generates. More specifically, pass at k tells you the likelihood that at least one generated block of code out of k generated blocks of code will pass all of its unit tests. Large language models often struggle to generate good code for complex programming problems. Software engineers adapt to this problem by prompting the large language model to generate multiple (k ) solutions for the same problem. Then, software engineers test each of the solutions against unit tests. The calculation of pass at k depends on the outcome of the unit tests: - If one or more of those solutions pass the unit test, then the LLM Passes that code generation challenge. - If none of the solutions pass the unit test, then the LLM Fails that code generation challenge. The formula for pass at k is as follows: \\\\\\[\\\\pass at k\ = \\\\\\\\total number of passes\\ \\\\\total number of challenges\\\\\\\\] In general, higher values of k produce higher pass at k scores; however, higher values of k require more large language model and unit testing resources. Suppose a software engineer asks a large language model to generate k=10 solutions for n=50 challenging coding problems. Here are the results: - 30 Passes - 20 Fails The pass at 10 score is therefore: