Benchmarking is essentially writing values into code.

All our expectations and fears about AI are forcibly embedded into those scoring tools—what constitutes progress, what should be feared, what needs to be optimized—and we end up pretending that these things can be precisely quantified. The problem is, some things simply can't be measured. Behind the selected metrics, there are often the designer's own assumptions. The choices you make in testing are equivalent to defining what AI should become. Conversely, the things that are not chosen might actually be the most important.