IFEval

Reported in recent model launches

These model reports include this benchmark. Their harnesses, task subsets, scoring policies, and resource budgets can differ; follow each report’s evaluation notes before joining the numbers.

What it measures

Programmatically checkable instructions test whether model outputs satisfy explicit constraints.

How to read the result

Report strict prompt-level success separately from individual constraint satisfaction.

Limits and comparison traps

A parser-visible constraint is only part of useful instruction following. Semantically wrong content can still satisfy a format checker.

Benchmark reference notes

541 prompts containing 25 types of programmatically verifiable formatting, length, keyword, punctuation, and case constraints.

Where this fits in an evaluation suite

Format compliance, preference, truthfulness, and safety require separate measurements. Keep desirable writing style from masking factual errors.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Pin the benchmark release, task split, model configuration, tools, sampling budget, and evaluation harness described by the primary source.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.