Tencent's WorkBuddy Bench targets a real weakness in coding-agent benchmarks: contamination

A persistent, underappreciated problem with coding-agent benchmarks is that many of them draw tasks straight from public GitHub issues and pull requests, which means a sufficiently well-trained model doesn't need to actually solve the underlying problem: it just needs to recognize the specific issue and recall the fix that's already sitting in its training data or reachable via a quick web search. Tencent's new WorkBuddy Bench is built specifically to close that loophole. Rather than lifting issue text directly, every task in the benchmark is reverse-engineered from a real commit, pull request, or business scenario and then rewritten as a short, conversational, role-played request, deliberately obscuring the connection back to the original issue thread so it can't be looked up online even if a model tries. The benchmark spans four genuinely distinct domains that map to how coding agents actually get used in practice: repository-level engineering work, front-end development, office and business workflow automation, and offensive and defensive security tasks, each scored with its own domain-appropriate verification method rather than a single generic pass or fail check. Just as importantly, the whole thing, task directories, environment images, the evaluation harness, and reference solutions, is released completely openly, so any team can re-run every task themselves and audit exactly what's being measured rather than trusting a vendor's self-reported number on a closed test set. For developers deciding which coding agent to actually adopt, this matters because a lot of the benchmark numbers vendors publish are vulnerable to exactly the contamination problem this benchmark is designed to avoid, meaning a model's real-world reliability on genuinely novel problems can be meaningfully worse than its benchmark score suggests. A reproducible, contamination-resistant benchmark like this one is a more trustworthy signal to weigh when comparing agent harnesses than marketing claims built on benchmarks a model may have effectively already seen.

Source

View on ShipDigest