Learning from preferences

Preference data says that one response is preferred to another under a particular prompt and rubric. It does not automatically identify which response is true. Human feedback and model-generated feedback both require a clear target: usefulness, correctness, tone, policy adherence, or some explicit combination.

A reward-model route learns a scoring function from comparisons, then updates the language-model policy against it. Direct Preference Optimization instead fits the policy using preference pairs through a classification-style objective derived from a constrained reward formulation. These are related ideas with different training machinery.

If comparisons mainly reward verbosity, the resulting policy may become verbose without becoming more capable. Include pairs where polished but incorrect work loses to concise, supported work. Examine disagreement and annotator expertise rather than treating every pair as equally informative.

Hold out prompts, task families, and judging rubrics. Combine a preference test with objective checks where possible. Use blinded comparisons, randomize answer order, and report model and judge versions. A win rate against one opponent and one judge does not define a universal ranking.