The automated research benchmark

AutoArena

Evaluating automated research for efficient AI

A benchmark for automated methods that improve the efficiency of language models.
Nine targets, fixed experiment budgets, and reproducible results.

Code Tasks Submit a result

— methods / 9 targets / — accepted runs

Overall leaderboard

Compare the best accepted results across all nine efficiency targets.

Data

Methods ranked by overall Elo, with open-source availability, best value and rank for every target
#	Method	Overall Elo ?	Open source

Values ↓ lower is better · #1 best on target · Click a value to inspect its run

Missing targets are excluded from Elo; coverage appears below each method.

How overall Elo is calculated

Each pair of methods is compared on the final target values of its best run. A lower value wins; equal values tie. Every included target receives the same nominal weight: 1/9 across all nine targets. Missing targets are excluded.

Training time is rounded to the nearest 30-second checkpoint with 30 × floor(seconds / 30 + 0.5) for ranking, tables and curves. Matching checkpoints tie; other targets retain their task’s tiebreaks. Downloaded submission JSON preserves raw measurements. Only selected endpoints enter Elo, without adjustments for search budget or reference scores.

A comparison’s update reflects its proportional improvement: the absolute log ratio is divided by the target’s median nonzero pairwise log gap, transformed with log₂(1 + gap), and capped at 3. Exact ties make no update.

Ratings start at 1500 and use standard Elo updates with K = 32 and a 400-point scale. We average 4,000 shuffled comparison orders with seed 20260902. Equal and opposite updates keep the cohort mean at 1500; there is no fixed maximum. Download full-precision ratings and calculation settings. The exported match-order standard deviation describes sensitivity to comparison order, not experimental uncertainty.

Performance by target

Best qualifying result as the experiment budget is spent.

Individual experiments

Steps count charged launches, including the reference and failures. Each line ends when that run ends.

Run records

All accepted runs for the selected target
Method / run	Best value	Budget used	Research model	Evidence