The AI Cyber Challenge makes evaluation operational
The Defense Advanced Research Projects Agency's (DARPA) August 8 announcement reports the results of its Artificial Intelligence (AI) Cyber Challenge (AIxCC). In the final scored round, competing cyber reasoning systems have analyzed more than 54 million lines of code, identified 86 percent of the synthetic vulnerabilities, and patched 68 percent of those identified.
Those figures are impressive. The design of the evaluation may be more important.