Stanford paper warns policymakers about inflated claims from AI benchmarking

As policymakers look to benchmarks as a way to regulate artificial intelligence systems, they should demand evidence to support claims based on such evaluations, researchers from Stanford University’s institute for Human-Centered Artificial Intelligence said, offering a logical framework for assistance.

“AI companies often use benchmarks to test their systems on narrow tasks but then make sweeping claims about broad capabilities like ‘reasoning’ or ‘understanding.’ This gap between testing and claims is driving misguided policy decisions and investment choices,” reads the...