What the index measures

Epoch fits a latent capability scale from overlapping model results across its benchmark collection. Task difficulty and the pattern of results matter; ECI is not a simple average of benchmark percentages.

Epoch uncertainty and interpretation

The imported estimate and interval retain Epoch’s original ECI units. Small differences with overlapping intervals do not establish a reliable ranking. Benchmark difficulty (EDI) describes the evaluations, not model performance.

Keep domain scales explicit

Epoch also documents domain-specific capability indices. The general ECI snapshot here is kept separate from individual benchmark accuracy, Arena ratings and provider-specific composite indices.

Where this fits in an evaluation suite

Ask what the task actually requires: recall, deduction, search, or a mixture. Separate closed-book reasoning from retrieval and tool-assisted problem solving.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Compare estimates from the same ECI fit and snapshot; retain the original uncertainty bounds and model versions. Do not treat ECI values as percentages.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

  1. AI benchmarks & capabilities ↗

    Epoch AI

  2. Epoch AI · ECI methodology ↗
  3. Capabilities & benchmarking — retained external evaluation records ↗

    Epoch AI · 2026

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.