Make the score mean what you intended

A test suite, exact-answer check, or rubric can only measure the conditions it contains. A policy may improve its score by learning the intended skill, exploiting a blind spot, or changing how it presents an answer. The design task is to make those possibilities distinguishable.

Check valid alternative solutions, missing files, malformed outputs, timeouts, and the effect of changing tests or environment state. A false negative can punish useful work; a false positive can supply a misleading training target. Review a sample of both successes and failures rather than inspecting only the headline winners.

Partial credit can make learning practical, but it should not conceal failure to meet a required condition. Publish reward components and strict task success together. Inspect whether improving one component lets a model ignore an obligation that the real user considers essential.

Reserve fresh tasks and a stronger or separately reviewed verifier for evaluation. When a gain disappears under independent checking, report the discrepancy. Keep a clear boundary between the reward used to optimize behavior and the evidence used to claim that behavior improved.