How we score
Every tool is scored per use case out of 100. The same tool can rank differently in different use cases.
| Criterion | Weight | Evidence |
|---|---|---|
| Task success on our standard test suite | 30 | 3 fixed tasks per use case, run hands-on |
| Output quality / maintainability | 15 | Lint, tests, pinned LLM-judge review of the diff |
| Pricing value and transparency | 15 | Vendor pricing page, cost to complete the suite |
| Ease of onboarding | 10 | Time and steps to first working result |
| Ecosystem and integrations | 10 | Git, IDEs, MCP, hosting, DB/auth |
| Privacy, security, ownership | 10 | Data-training policy, export, compliance docs |
| Momentum and reliability | 10 | Changelog cadence, status-page incidents |
Preliminary vs tested scores
Preliminary · based on public info means the score is an editorial estimate from vendor documentation and public reporting. The hands-on criteria (task success, output quality, onboarding time) are estimated, not measured. When we run the full test suite with our reference setup, we replace the score, change the badge to "Hands-on tested" with a date, and publish the evidence log. Partly hands-on tested means we ran some fixed tasks (for example, with a free local model) and published the raw logs, but the score is still the preliminary estimate. As of October 14, 2026, no score has been fully re-derived from hands-on tests.
Rules
- Every sub-score cites a source URL and access date.
- A score moves more than 5 points only with new evidence, recorded in the public changelog (the git history of our data files).
- Affiliate status is never an input to scores.
- Prices are re-checked weekly; each price shows the date it was last checked against the vendor page, or "unverified".
- A human reviews every change to a #1 pick.
Fixed test tasks
The terminal/CLI agent suite (v1) is published here, with the exact prompts and pass criteria. Suites for other use cases are in progress.