What coding agents install, measured run by run
Every coding agent gets the same real request in a fresh, version-pinned test app, eight times per scenario, and every pick rate shows its run count and likely range.
Version-pinned test apps
Agents work in real apps: an empty folder or an official framework starter, pinned to an exact version. Every run starts in a fresh sandbox.
The request each agent gets
Each agent gets plain requests: add an integration, recommend a tool and, where the category has one, use a hosted service. Each ends the same way: decide alone, and assume the team creates any accounts.
Eight runs per scenario
Agents don't choose the same tool every time, so a full test runs each scenario eight times. A scenario reads as a lead or a loss only when the gap is clear at 90%.
Pick rate and install rate
The install rate counts runs that wrote the tool into the code. The pick rate also counts runs that named it as their choice, in tests that recognise tools from a confirmed list.
Re-tests and model releases
Public leaderboards are re-tested on every major model release. Your own tests run on demand or on a daily, weekly or monthly schedule, and every run records its model and classifier version.