Keep the test outside the learning loop

A held-out benchmark is useful when it represents work the training and selection process did not already optimize. If the same tasks influence generated examples, reward design, checkpoint selection, and final reporting, the result becomes a development measurement rather than a clean test of transfer.

Near-duplicate problems, paraphrased solutions, source metadata, and generated variants can reveal the same underlying answer. Track provenance at the problem and source-family level. An n-gram filter can help identify copied text, but cannot establish independence by itself.

Use a development set for iteration and preserve a sealed test distribution for final comparisons. Record which benchmark results influenced decisions. When a team chooses the best of many checkpoints, include that selection process in the account of the experiment.

Public benchmarks remain useful for regression testing and historical comparison. Label their limitations and complement them with fresh tasks, private holdouts, or independent audits. Report what was checked for contamination and what remains unknown rather than claiming a dataset is contamination-free without the evidence to support it.