Submit to AutoArena

1. Prepare your record and artifacts

Start with the exported submission.json. If you have not exported your completed run yet:

python3 -m engine submission \
  --run runs/my-method-params \
  --output runs/my-method-params/submission

When artifact downloads are ready, archive the export and upload it to a file or artifact host:

tar -czf runs/my-method-params/artifacts.tar.gz \
  -C runs/my-method-params/submission .

Get a public HTTPS download link. Keep the code, measurement logs and reproduction files at their exported paths inside the archive. The files do not need to be served individually. Use a link without credentials or expiring authentication parameters.

2. Add the JSON to your PR

Fork the repository, clone your fork, and create a branch. From its root, install the check dependencies and run:

python3 -m pip install -r site/requirements.txt
git switch -c submit/my-method

python3 scripts/leaderboard.py add \
  --submission runs/my-method-params/submission/submission.json \
  --output site/submissions/my-method/params

python3 scripts/leaderboard.py check \
  site/submissions/my-method/params/submission.json

Replace the paths with yours. The helper converts the exported results to the common autoarena/site-submission/1 format and creates an empty artifacts.json. Add --artifacts-url URL when the code and logs archive is available. The page shows only supplied download links.

site/submissions/my-method/
  params/
    submission.json
    artifacts.json
  flops/
    submission.json
    artifacts.json

Repeat for each completed run. Use a new folder for each run, such as params/, flops/, or params-run-2/. If code and logs are separate downloads, use --code-url URL --logs-url URL instead of --artifacts-url. You may add --report-url URL for an existing HTML report; it is optional.

3. Open a pull request

git add site/submissions/my-method/
git commit -m "Submit my-method results"
git push -u origin submit/my-method

On GitHub, open one pull request for the method’s results across its targets. Use the title Submit: my-method and include:

1. Your public method name and a link to its code or description. Include a license link if available for the Open source column.
2. Artifact download links, when available.
3. Relevant method and evaluation details.

Commit both JSON files for each run. The build checks result consistency and recalculates rankings and curves without contacting artifact hosts. Publish the generated directory to update the leaderboard.

Display names and public source/license links are recorded separately in site/methods.json, keyed by your existing method ID. You may include that small metadata edit or provide the links in your PR for maintainers to add. Open-source availability does not affect ranking or submission eligibility.

What the checks and curves mean

The checks verify the common JSON format, ordered measurement sequences, budget counts, and agreement of the winner and reference with the recorded measurements. The page uses the supplied qualification flags and ranking keys.

The overall table shows one row per method, using its best run on each target. Each cell includes the method’s rank and value. Overall Elo gives every target equal weight, uses proportional improvement margins, and averages 4,000 shuffled comparison orders. The cohort mean stays at 1500, with no fixed maximum. Training time is rounded to 30-second checkpoints for rankings, tables and curves; matching checkpoints tie. Downloaded submissions retain raw measurements. Missing targets are excluded and coverage is shown. The leaderboard includes the full rating calculation.

The horizontal axis counts charged experiment launches, including charged failures and the reference. Lines show the best qualifying value so far and end at the run’s recorded budget usage. Individual experiments can be shown with the chart toggle. All accepted runs, including repeated runs of a method, remain available under Run records.

Keep accepted artifacts available and unchanged. Submit a new run folder for a new run. If an existing record needs correction, explain it in a follow-up PR and retain the original evidence.