SCROLLS & ZeroSCROLLS

What it measures

Standardized CompaRison Over Long Language Sequences across GovReport, SummScreenFD, QMSum, Qasper, NarrativeQA, and QuALITY.

Reading the measurement

Length is a test condition, not a capability score. Locate the relevant evidence, then examine whether the model resolves conflicts and combines details correctly. Keep document extraction, input length and answer grading fixed.

Published evaluation

SCROLLS aggregate; SCROLLS · original baseline comparison. The source’s task split, model configuration and scoring units are retained.

Where this fits in an evaluation suite

A large context window is a capacity specification. Successful retrieval and cross-document reasoning are observed capabilities. Test both at the actual lengths your workflow needs.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

SCROLLS aggregate; SCROLLS · original baseline comparison. The source’s task split, model configuration and scoring units are retained.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.