# How we test

Canonical: https://devtune.ai/product/method

## What coding agents install, measured run by run

Every coding agent gets the same real request in a fresh, version-pinned test app, eight times per scenario, and every pick rate shows its run count and likely range.

#### Version-pinned test apps

Agents work in real apps: an empty folder or an official framework starter, pinned to an exact version. Every run starts in a fresh sandbox.

#### The request each agent gets

Each agent gets plain requests: add an integration, recommend a tool and, where the category has one, use a hosted service. Each ends the same way: decide alone, and assume the team creates any accounts.

#### Eight runs per scenario

Agents don't choose the same tool every time, so a full test runs each scenario eight times. A scenario reads as a lead or a loss only when the gap is clear at 90%.

#### Pick rate and install rate

The install rate counts runs that wrote the tool into the code. The pick rate also counts runs that named it as their choice, in tests that recognise tools from a confirmed list.

#### Re-tests and model releases

Public leaderboards are re-tested on every major model release. Your own tests run on demand or on a daily, weekly or monthly schedule, and every run records its model and classifier version.

- [Pricing](https://devtune.ai/pricing?product=agent-picks)
- [Documentation](https://docs.devtune.ai/)