A position paper being presented at a workshop on Agentic Software Engineering co-located with the 32nd ACM SIGKDD Conference argues that the coding benchmarks the industry uses to rank AI coding agents are measuring something meaningfully different from what actually matters in real software engineering work. Benchmarks like the SWE-bench family score an agent on whether it can resolve a pre-defined, already-scoped GitHub issue against a fixed test suite, a task that resembles a take-home coding exercise more than it resembles the ambiguous, evolving, judgment-heavy work real engineers do, where requirements shift mid-task, the right fix is often a judgment call rather than a pass or fail test, and success depends as much on architectural decisions and effective collaboration with reviewers as on producing code that satisfies a test harness. The paper's core argument is that optimizing models and agent harnesses against these benchmarks risks a Goodhart's-law problem: labs get very good at the specific, narrow shape of task the benchmark rewards, without that progress necessarily transferring to messier real-world engineering, and worse, some of that benchmark progress may reflect memorization or contamination rather than genuine capability gains, an issue several other papers this year have separately raised. For engineering teams evaluating which coding agent or model to adopt, the practical takeaway is to treat published benchmark leaderboard positions as a weak, easily-gamed signal and instead pilot tools against your own team's actual, messier backlog of tickets, including ones with ambiguous requirements or that require touching unfamiliar parts of a codebase, before trusting a benchmark score to predict real-world performance.