Goodhart's law

Goodhart's law says that when a yardstick used to measure something becomes a goal in itself, it stops reflecting the real state of things. Economist Charles Goodhart originally coined this to describe policy, but it's now cited across any field that involves evaluation and measurement.

Here's an analogy. Suppose you decide to measure a student's ability by test scores. At first, the score is a decent proxy for actual ability. But once the score itself becomes the goal, students start focusing on test-taking tricks and memorized answer patterns instead of building real skill. Scores keep climbing, but they no longer mean what they used to.

The same thing happens with AI. If you train an AI to score well on some benchmark measuring performance, the AI may not actually get better at the underlying task — it may just learn tricks that look good on that specific test. This gap between apparent performance and real performance is exactly the trap Goodhart's law warns about.

Terence Tao invoked Goodhart's law regarding the growing volume of AI-generated math proofs. His point: if the AI industry rewards easily measurable outcomes like citation counts or benchmark scores, AI ends up chasing results that merely look good rather than real understanding or contribution. It's worth noting this law wasn't created with AI in mind — it's an old principle originally used to explain economic policy, and Tao is simply borrowing it to talk about how mathematical research gets evaluated.