What it measures

Agents train and evaluate machine-learning systems on competition-style engineering tasks.

How to read the result

Report performance alongside runtime, data access, and whether the workflow respected the training and evaluation split.

Limits and comparison traps

Competition data provenance, leakage controls, hardware, and time budgets can dominate apparent model differences.

Published evaluation

Official benchmark result table. The named measure is retained in its native units; separate task or scoring measures are not combined.

Where this fits in an evaluation suite

A correct multiple-choice answer and a supported scientific contribution demand different evidence. Check whether the score rewards a proposal, an executed experiment, or a completed research target.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Official benchmark result table. The named measure is retained in its native units; separate task or scoring measures are not combined.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

  1. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — published result table ↗

    Chan, Jun Shern, Chowdhury, Neil, Jaffe, Oliver, Aung, James, Sherburn, Dane, Mays, Evan et al. · 2024

    All 12 authors

    Chan, Jun Shern, Chowdhury, Neil, Jaffe, Oliver, Aung, James, Sherburn, Dane, Mays, Evan, Starace, Giulio, Liu, Kevin, Maksin, Leon, Patwardhan, Tejal, Weng, Lilian, Mądry, Aleksander

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.