CLoud (Critique-out-Loud) & Self-Taught Evaluators

Mechanism

CLoud: Trains a single model where the LLM base generates an explicit natural-language critique $c \sim \pi_\phi(\cdot\mid x,y)$ $c \sim π_{ϕ} (\cdot ∣ x, y)$ first, and a lightweight scalar reward head placed on top of the generated critique tokens predicts $R_\phi(x, y, c) \in \mathbb{R}$ $R_{ϕ} (x, y, c) \in R$ , combining CoT reasoning depth with continuous scalar calibration.
- Self-Taught Evaluator: Iteratively trains a generative LLM judge without human preference labels by synthesizing contrasting model outputs (correct vs. perturbed/flawed) and filtering judge CoT traces that correctly identify the winner.

Critique-out-Loud Reward Models ↗
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu · 2024
Self-Taught Evaluators ↗
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu et al. · 2024
All 10 authors
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, Xian Li