Sunil Venkataram
Back to garden
•
🌰 seedling
Goodhart's Law in AI Adoption Metrics
type/noteaiorganizationeconomics

Goodhart’s Law in AI Adoption Metrics

The mechanism

Goodhart’s Law states that when a measure becomes a target, it ceases to be a good measure. In AI adoption programs, this plays out through a specific escalation pattern:

  1. Mandate phase — Leadership announces AI transformation. Every team must adopt AI tools and report results.
  2. Competition phase — One team reports 40% productivity gains. The next reports 60%. A third claims 80% automation. Nobody checks methodology because nobody has a baseline.
  3. Gamification phase — Leaderboards appear tracking prompts per week, AI-generated code percentage, agent skills shipped. Individual contributors are ranked against peers.
  4. Inversion phase — The entire organization optimizes to make the adoption metric look good rather than to make the company better. The metric was the instrument; now it’s the objective.

Why this is worse than typical Goodhart failure

Standard Goodhart scenarios involve a known-good metric that gets gamed. AI adoption metrics were never good measures to begin with. “Percentage of code AI-generated” conflates quality with quantity. “Number of agent skills built” rewards complexity. “Time saved” is self-reported with no control group. The metric was born corrupted and then made a target.

The grain-reporting parallel

Hanchung Lee draws an explicit analogy to the Great Leap Forward’s grain reporting, where provinces competed to report impossible yields. Officials staged photographs of rice so dense children could stand on it. The central government, pleased by the numbers, increased requisitions based on reported yields. Farmers starved eating the difference between the real number and the fantasy.

The corporate equivalent: leadership increases AI investment based on reported gains, reallocates headcount based on claimed automation, and sets next quarter’s targets based on this quarter’s fiction.

Key Takeaways

  1. AI adoption metrics are uniquely vulnerable to Goodhart failure because no established baselines exist for most knowledge work.
  2. The competitive dynamic between teams creates an escalation spiral where each report must outdo the last.
  3. Self-reported productivity gains without methodology review are not data — they are political signaling.
  4. The corrective is measuring outcomes (revenue, error rates, customer satisfaction) rather than inputs (AI usage, prompts written, skills shipped).
Backlinks (1)
Connected Notes