THE RESEARCH NOTEBOOK / BENCHMARKS & POST-TRAINING

Understand the work
behind the score.

Read how evaluations are built, what their results support, and how post-training changes the systems being measured.

Read the latest model-launch evaluation maps →

Post-training methods

Explore the learning loop →

SFT

Teaching a model what good looks like

Demonstrations, data quality, and the difference between learning an answer format and improving a capability.

DPO / RLHF

Learning from preferences

How preference pairs, reward models, and direct optimization shape an assistant’s behavior.

RLVR

When the reward can be checked

Executable verifiers, hidden holdouts, and the gap between maximizing a score and learning the intended task.

PPO / GRPO

From rollouts to policy updates

A conceptual map of policy optimization, relative advantages, and the compute that belongs in an experiment report.

ENV

A task is more than a prompt

Observations, actions, partial information, and why environment design determines what a model can learn.

EVAL

Did the training actually help?

Exact-base controls, contamination checks, uncertainty, and honest attribution for post-training gains.

LoRA / QLoRA

Adapt a model without updating every weight

Low-rank adapters, quantized base models, and what an efficient fine-tuning comparison should keep fixed.

DISTILLATION

What a student model learns from a teacher

Distillation transfers a selected learning signal, so teacher quality and dataset construction matter as much as model size.

PRM / ORM

Reward the reasoning, the result, or both?

Process labels and outcome checks expose different errors. A sound evaluation needs to know what each one actually verifies.

SYNTHETIC DATA

Generate more data without repeating the same mistakes

Instruction generation, filtering, diversity, and the checks that make synthetic examples useful for post-training.

SELECTION / STaR

Learn from the attempts that survive checking

Sampling, verification, and iterative self-training can turn successful trajectories into supervision—but selection changes the dataset.

KTO / ORPO / SimPO

Different preference objectives, different assumptions

Compare methods by the feedback they need and the behavior they optimize, rather than treating every preference-training label as interchangeable.

RL SYSTEMS / DAPO

The training system is part of the method

Rollout generation, reward handling, sequence lengths, and system throughput can determine what an RL experiment learns.

REWARD DESIGN

Make the score mean what you intended

Verifier coverage, partial credit, invalid episodes, and independent checks determine whether reward gains represent capability.

INFERENCE COMPUTE

Separate better learning from more attempts

Reasoning effort, tool use, sampling, and reranking can improve a system without changing its trained weights.

DATA BOUNDARIES

Keep the test outside the learning loop

Contamination includes structural leakage and benchmark-specific selection, not only copied answer text.

Multi-Task Instruction Tuning

Multi-Task Instruction Tuning (FLAN / T0 / NaturalInstructions)

Teach one model to follow many kinds of instructions by turning classification, question answering, summarization and other tasks into supervised examples.

Superficial Alignment & High-Quality Curation SFT

Superficial Alignment & High-Quality Curation SFT (LIMA / AlpaGasus / DEITA)

Study how a small, carefully curated set of demonstrations can teach a pretrained model the style and format of useful responses.

Long-CoT Cold-Start & Reasoning-Centric Minimal SFT

Long-CoT Cold-Start & Reasoning-Centric Minimal SFT (DeepSeek-R1 Cold-Start / s1 / LIMO)

Start reasoning training with a small set of checked, extended solutions that demonstrate reflection, exploration and correction.

Evol-Instruct & Complexity-Scaled Synthetic SFT

Evol-Instruct & Complexity-Scaled Synthetic SFT (WizardLM / WizardCoder / WizardMath)

Expand instruction data by asking a teacher model to add constraints, deepen reasoning and introduce new kinds of tasks.

Self-Instruct, Magpie & Persona-Driven Synthetic SFT

Self-Instruct, Magpie & Persona-Driven Synthetic SFT (Alpaca / Magpie / PersonaHub)

Use a teacher model to generate instruction–response pairs, then filter and diversify them for supervised training.

Explanation Tuning & System-Prompt Trace Distillation

Explanation Tuning & System-Prompt Trace Distillation (Orca / Orca 2 / AgentInstruct)

Augments standard teacher outputs with rich system prompts ("Explain like I'm five", "Think step-by-step and justify every assumption") to elicit detailed intermediate reasoning explanations from a frontier teacher, then…

Instruction Backtranslation & Self-Alignment

Instruction Backtranslation & Self-Alignment (Humpback / Kun)

Trains an inverse model π_inv(x | y) to predict candidate user instructions x from raw, unlabelled web/textbook documents y, then uses the forward model to score and filter (x, y) pairs by quality,…

Multi-Stage Curriculum SFT & Specialized Research SFT

Multi-Stage Curriculum SFT & Specialized Research SFT (Annealing SFT / ArxivNet A-SFT)

Splits SFT into a two-stage or multi-stage curriculum: Stage 1 trains on broad, diverse general instructions and domain-foundational reasoning; Stage 2 anneals the learning rate on high-precision, long-context domain research…

Noisy Embedding & Token-Weighted SFT

Noisy Embedding & Token-Weighted SFT (NEFTune / Prompt-Loss Weighting)

Modifies the SFT forward/loss pass to prevent memorization and improve conversational generation.

Dual-Mode / Hybrid Thinking-NonThinking Fusion SFT

Dual-Mode / Hybrid Thinking-NonThinking Fusion SFT (Qwen3 / DeepSeek-V3.1+)

After training a reasoning RL policy, conducts a fusion SFT stage that mixes long-CoT trajectories (reasoning traces triggered by /think or system flags) with fast, direct non-thinking responses (reasoning traces…

Critique-and-Repair SFT / Reflection-Tuning

Critique-and-Repair SFT / Reflection-Tuning (Self-Critique SFT / CRITIC)

Rather than training only on clean (x, y^) gold trajectories, trains the model on multi-step trajectories (x, y_draft, c_critique, y_repaired^) containing realistic model failures,…

Tool-Integrated Reasoning

Tool-Integrated Reasoning (TIR) SFT (ToRA / NuminaMath-TIR)

Supervises interleaved natural-language deduction and executable code/tool blocks: x → r_1 → a_1 (Python/SymPy/Bash) → obs_1 → r_2 → … → y,…

Formal Proof-State & Tactic SFT

Formal Proof-State & Tactic SFT (LeanDojo / DeepSeek-Prover / UlamAI Prover)

Extracts (proof_state, premise_context) -> tactic pairs (or whole-proof generation with natural-language informal sketch comments) from formal theorem provers (Lean 4, Isabelle/HOL, Coq) and fine-tunes the LLM to emit formally…

Dynamic Fine-Tuning

Dynamic Fine-Tuning (DFT) / Probability-Stabilized SFT

Reweight supervised token losses with detached target probabilities, then measure the effect on held-out generalization.

Proximal Policy Optimization

Proximal Policy Optimization (PPO) for RLHF

Optimizes the LLM policy π_θ against a token-level reward r_t = -β log(π_θ(y_t| x,y_<t)/π_ref(y_t| x,y_<t)) + I[t=|y|]R_φ(x,y) using Generalized Advantage…

PPO-ptx

PPO-ptx (PPO with Pretraining Mix Gradients)

Augments the standard PPO RLHF objective with an auxiliary next-token log-likelihood loss on the original pretraining distribution D_pretrain to prevent the "alignment tax" (regressions on public NLP…

REINFORCE with Moving-Average / Learned Baseline for LLMs

Treats the entire generated sequence y as a single bandit action (since intermediate environment transitions in text generation are deterministic), optimizing

RLOO

RLOO (REINFORCE Leave-One-Out)

Samples K independent completions y^(1), …, y^(K) ~ π_θ(·| x) per prompt and uses the average reward of the other K-1 completions as an unbiased, low-variance prompt-specific baseline…

ReMax

ReMax (Reward Maximization with Greedy Baseline)

Train from the reward difference between a sampled response and a greedily decoded response, using the greedy response as a baseline without a separate value network.

AWR / AWAC

AWR / AWAC (Advantage-Weighted Regression / Actor-Critic)

Formulates policy improvement as weighted supervised learning where off-policy or replay-buffer trajectories (x,y) are weighted exponentially by their estimated advantage

PPG

PPG (Phasic Policy Gradient) / Decoupled Policy-Value RLHF

Separates policy updates (actor phase) and value-function distillation/training (auxiliary phase) into distinct alternating phases so value estimation can train for more epochs without corrupting shared representations or…

P3O

P3O (Pairwise Proximal Policy Optimization)

Notices that standard PPO is sensitive to additive shifts in the reward model R(x,y) + c(x) unless the value critic is perfectly calibrated.

APA

APA (Advantage-Induced Policy Alignment)

Replaces PPO's multiplicative ratio clipping with a squared-error loss between the log-policy ratio log π_θ(y_t| s_t)π_ref(y_t| s_t) and the scaled token advantage (A_t/β),…

NLPO

NLPO (Natural Language Policy Optimization)

Extends PPO for language generation with a parameterized top-p masking policy that dynamically masks out low-probability vocabulary tokens during RL exploration, preventing grammatical collapse in large action spaces (|V| >…

IMPALA V-Trace / Asynchronous Actor-Learner LLM RL

Decouples rollout actors (running high-throughput vLLM/SGLang inference engines) from gradient learners (running FSDP/Megatron), correcting for policy staleness μ ≠ π_θ using truncated importance sampling weights…

Soft Q-Learning

Soft Q-Learning (SQL) & Path Consistency Learning (PCL) for LLMs

Exploits the equivalence between maximum-entropy RL and soft Bellman consistency along any sub-trajectory (s_t, a_t:t+k, s_t+k), allowing unified off-policy and on-policy token-level updates without separate actor-critic…

RLVR

RLVR (Reinforcement Learning with Verifiable Rewards)

Replaces neural preference reward models R_φ(x,y) with deterministic, programmatic verification functions v(x,y) ∈ 0, 1 (or graded rubric scores v(x,y) ∈ [0, 1]) such as symbolic math equivalence checkers…

Pure Zero-SFT RLVR

Pure Zero-SFT RLVR (DeepSeek-R1-Zero Paradigm)

Applies RLVR (via GRPO with accuracy + format rewards) directly to a raw pretrained base checkpoint π_base with zero prior SFT, allowing the model to discover long-CoT reasoning, self-verification, and…

GRPO

GRPO (Group Relative Policy Optimization)

Samples a group of G outputs y_1, …, y_G from the old policy π_θ_old(·| x) for each prompt x, computes their rewards r_1, …, r_G, and estimates the advantage of token t in…

Dr.GRPO

Dr.GRPO (Group Relative Policy Optimization Done Right)

Remove sequence-length and reward-standard-deviation normalization from GRPO to address the length and question-difficulty biases studied in the paper.

DAPO

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization)

Introduces four synergistic modifications to GRPO for large-scale long-CoT RL: 1.

VAPO

VAPO (Value-Augmented Proximal Policy Optimization for Long-CoT)

Demonstrates that value-based PPO can outperform critic-free GRPO/DAPO on long-CoT reasoning once value-function pathologies are fixed via: (1) Value-Pretraining (warming up the critic from the reward signal before policy…

GSPO

GSPO (Group Sequence Policy Optimization)

Identifies a fundamental flaw in token-level importance ratios ρ_i,t = π_θ(y_i,t| x,y_i,<t)π_θ_old(y_i,t| x,y_i,<t): while the basic policy gradient identity holds at the…

CISPO

CISPO (Clipped Importance Sampling Policy Optimization)

Observes that PPO/GRPO/DAPO zero out the gradient of any token whose importance ratio ρ_i,t(θ) exceeds 1+ε.

REINFORCE++ & REINFORCE++-Baseline

Combines the simplicity of critic-free REINFORCE with key PPO stability optimizations: (1) token-level KL penalty added to rewards, (2) PPO-style ratio clipping, and (3) global batch advantage normalization (normalizing…

SAO / V-SAO

SAO / V-SAO (Single-Rollout Asynchronous Optimization)

Designed specifically for long-horizon asynchronous agentic RL where group-wise sampling (G=8 or 16 rollouts per prompt) creates severe tail-latency stragglers and off-policy staleness.

VinePPO

VinePPO (Monte Carlo Tree Rollout Credit Assignment)

Replaces the inaccurate neural value network V_ψ(s_t) in PPO with unbiased Monte Carlo rollouts branched directly from intermediate reasoning steps s_t (utilizing fast vLLM prefix caching to sample K fast…

PRIME

PRIME (Process Reinforcement through Implicit Rewards)

Trains an implicit Process Reward Model π_ψ online alongside the policy π_θ using only outcome labels r ∈ 0,1, where token-level implicit process rewards are defined automatically via the log-ratio…

GPG

GPG (Group Policy Gradient)

Minimalist RLVR method that strips away surrogate ratio clipping, reference-model KL divergence, and critic networks altogether, directly optimizing the group-normalized REINFORCE objective with dynamic thresholding of…

TOPR

TOPR (Tapered Off-Policy REINFORCE)

Handles off-policy lag in asynchronous RL by combining asymmetric tapered importance-sampling weights on positive-advantage trajectories with unclipped/bounded updates on negative-advantage trajectories, accelerating learning…

AGPO

AGPO (Adaptive Group Policy Optimization)

Modifies GRPO to handle two boundary cases: (1) when all G responses in a group are uniformly correct or uniformly wrong (std=0), assigns a small non-zero signed advantage (+1 or -1) rather than 0 so the model still…

CPPO

CPPO (Completion Pruning Policy Optimization)

Accelerates GRPO training by up to 4–8× by dynamically pruning low-contribution completions inside each group G before the expensive backward pass (keeping only completions with large absolute relative advantages |A_i|).

LCPO

LCPO (Length-Controlled Policy Optimization / L1)

Trains reasoning models via GRPO to obey explicit user-specified token budgets n_target passed in the prompt

Step-GRPO / S-GRPO

Step-GRPO / S-GRPO (Step-Wise & Early-Exit GRPO)

Augments GRPO's outcome reward with dense step-wise reasoning accuracy and formatting rewards (or serial early-exit rollouts truncated at intermediate reasoning checkpoints), rewarding the model for reaching correct…

Hybrid SFT-RL / Prefix-RFT

Hybrid SFT-RL / Prefix-RFT (UFT / LUFFY / Prefix-Guided RLVR)

When a hard problem has 0% pass rate under current policy sampling (pass@G = 0), pure RLVR receives zero positive signal.

DPO

DPO (Direct Preference Optimization)

Derives the closed-form optimal policy π^(y| x) = (1/Z(x))π_ref(y| x)exp((1/β)R(x,y)) of the KL-regularized RLHF objective, solving for R(x,y) = β log…

IPO

IPO (Identity Preference Optimization / ΨPO)

Shows that under deterministic or near-deterministic preferences P(y_w y_l) → 1, the Bradley-Terry transformation in DPO drives log (π_θ(y_w)/π_θ(y_l)) → ∞ regardless of the KL…

KTO

KTO (Kahneman-Tversky Optimization)

Eliminates the requirement for paired (y_w, y_l) preferences on the same prompt x.

ORPO

ORPO (Odds Ratio Preference Optimization)

Combine supervised learning and preference optimization in one objective, using response odds to separate preferred and rejected answers without a reference model.

SimPO

SimPO (Simple Preference Optimization)

Notices that at inference time, decoding is governed by the policy's own length-normalized log-likelihood (1/|y|)log π_θ(y| x), whereas DPO optimizes the unnormalized log-ratio against π_ref.

CPO

CPO (Contrastive Preference Optimization)

Approximates the ideal reference policy as a uniform prior to drop π_ref memory overhead, combining a reference-free DPO contrastive loss with an explicit SFT negative log-likelihood regularizer on y_w:…

R-DPO

R-DPO (Regularized / Length-Desensitized DPO)

Explicitly penalizes sequence-length exploitation in DPO by adding an analytical length margin α(|y_w| - |y_l|) inside the Bradley-Terry sigmoid argument

cDPO

cDPO (Conservative DPO)

Accounts for a known label-noise rate ε ∈ (0, 0.5) in human/AI preference annotations by applying label smoothing directly to the DPO cross-entropy objective

rDPO

rDPO (Robust DPO under Random Preference Flips)

Correct the DPO loss for a specified rate of random preference-label flips, under the paper’s symmetric-noise assumption.

sDPO

sDPO (Stepwise / Reference-Curriculum DPO)

Splits offline preference datasets into sequential difficulty or quality tranches and updates the reference model π_ref ← π_θ^(k-1) at each tranche boundary so the reference policy stays close…

Step-DPO & MCTS-DPO

Step-DPO & MCTS-DPO (Process-Level Stepwise DPO)

Instead of contrasting two complete responses (y_w, y_l) that share many valid early reasoning steps, identifies the exact first erroneous reasoning step s_k^l vs.

TDPO

TDPO (Token-Level Direct Preference Optimization)

Incorporates forward token-level sequential KL divergence constraints ∑_t=1^|y| D_SeqKL(x, y_<t; π_ref|π_θ) directly into the DPO objective (TDPO1 and TDPO2), regulating…

f-DPO

f-DPO (Generalized f-Divergence DPO)

Generalizes DPO from reverse KL divergence to any convex f-divergence (forward KL, Jensen-Shannon, α-divergence) by replacing log π_θπ_ref with the derivative of the convex conjugate…

β-DPO

β-DPO (Dynamic β Calibration for DPO)

Dynamically adapts the KL penalty parameter β_i per preference pair based on the empirical reward discrepancy Δ R_i relative to batch statistics, increasing β on noisy low-gap pairs and decreasing β on…

DPO-Positive and chosen-response likelihood regularization

Identifies a critical failure mode of DPO on reasoning and math datasets where y_w and y_l differ by only a few tokens (small edit distance): standard DPO reduces log π_θ(y_w| x) along with log…

NCA & InfoNCA

NCA & InfoNCA (Noise Contrastive Alignment)

Formulates LLM alignment from scalar reward annotations r(x, y_k)_k=1^K as a Noise Contrastive Estimation (NCE) problem, training the policy log-ratio β log π_θ(y_k| x){π_ref(y_k|…

BCO

BCO (Binary Classifier Optimization)

Trains a binary logistic classifier separating thumbs-up completions y^+ from thumbs-down completions y^- using density ratio matching with a dynamically shifted reward baseline δ, providing theoretical grounding…

AOT

AOT (Alignment via Optimal Transport)

Enforces first-order stochastic dominance of the implicit reward distribution of positive samples over negative samples using 1D Wasserstein optimal transport sorting across mini-batches, supporting both paired and unpaired…

APO

APO (Anchored Preference Optimization: APO-zero & APO-down)

Controls whether the chosen likelihood log π_θ(y_w) and rejected likelihood log π_θ(y_l) increase or decrease during offline alignment depending on whether the model is already stronger or weaker than the…

WPO

WPO (Weighted Preference Optimization for Off-Policy-to-On-Policy Simulation)

Reweights off-policy preference pairs (x, y_w, y_l) by their sequence probability under the current policy w(x,y) = exp((1/|y|)∑_t log π_θ(y_t| x, y_<t)), making off-policy public…

RainbowPO & MallowsPO

RainbowPO & MallowsPO (Unified Multi-Component Preference Optimization)

Integrates seven orthogonal DPO improvements—length normalization, margin calibration, reference-policy mixing (α π_θ + (1-α)π_ref), WPO reweighting, and contextual scaling (Mallows…

RRHF

RRHF (Rank Responses to Align Language Models with Human Feedback)

Samples K candidate responses y_1, …, y_K with external scores r_1, …, r_K and optimizes a pairwise ranking margin hinge loss over length-normalized conditional log-probabilities p_i =…

PRO

PRO (Preference Ranking Optimization)

Extends pairwise Bradley-Terry preference optimization to a full listwise Plackett-Luce permutation ranking over K ordered candidates y_1 y_2 … y_K

SLiC-HF

SLiC-HF (Sequence Likelihood Calibration with Human Feedback)

Directly calibrates the sequence log-likelihood log π_θ(y| x) using a large-margin rank hinge loss max(0, δ - log π_θ(y_w| x) + log π_θ(y_l| x)) regularized by cross-entropy on the…

LiPO & LiPO-λ

LiPO & LiPO-λ (Listwise Preference Optimization via Learning-to-Rank)

Adapts LambdaRank from classic Information Retrieval to LLM alignment: given a list of K responses with real-valued reward labels ψ = (s_1, …, s_K), weights each pairwise DPO log-ratio gradient by the change in…

Multi-Preference / Plackett-Luce DPO

Multi-Preference / Plackett-Luce DPO (mDPO / K-Pair DPO)

Samples K responses per prompt and constructs either the exact Plackett-Luce listwise likelihood or all 2 thresholded preference pairs where R(y_i) - R(y_j) > δ_min, maximizing data efficiency from…

DNO

DNO (Direct Nash Optimization)

Batched on-policy iterative algorithm that samples K outputs from the current policy π_θ_t, compares them against each other and the gold/teacher output using a general pairwise win-rate oracle P(y_i…

MODPO / Multi-Objective Margin Preference Optimization

Aligns a single LLM simultaneously to M competing preference dimensions (e.g., helpfulness, safety, conciseness, citation faithfulness) by incorporating scalarized reward margins ∑_m ≠ k w_m r_m(x, y) from M-1…

Cringe Loss & Contrastive Unlikelihood Training

Token-level negative-feedback objective that penalizes toxic, repetitive, or invalid tokens y_t^- by contrasting their logit directly against a top-k plausible positive token sampled from the model's own distribution at…

Iterative DPO / Online DPO

Closes the off-policy gap of vanilla DPO by alternating across T rounds: in round t, sample K responses y_1,…,y_K ~ π_θ_t(·| x), score them with a reward model or verifier to form fresh…

XPO

XPO (Exploratory Preference Optimization)

Augments Iterative Online DPO with an explicit theoretical optimism bonus α ∑_y ~ π_ref log π_θ(y| x) that encourages the policy to actively explore out-of-support responses where the…

SELM

SELM (Self-Exploring Language Models)

Bi-level online preference optimization objective that adds an optimism term to the implicit reward during online DPOpair selection, actively seeking out high-potential unseen responses rather than greedily exploiting the…

SPIN

SPIN (Self-Play Fine-Tuning)

Allows an SFT model to improve without any additional human preference data or external reward model via a two-player minimax game: at iteration t, the opponent (old policy π_θ_t) generates synthetic responses…

SPPO

SPPO (Self-Play Preference Optimization)

Formulates alignment with general, potentially intransitive preferences P(y y' | x) as a constant-sum two-player game, seeking the von Neumann winner (Nash equilibrium) via multiplicative weights / mirror…

Nash-MD

Nash-MD (Nash Learning from Human Feedback via Mirror Descent)

Replaces the reward-maximization paradigm of RLHF with finding the Nash equilibrium of a KL-regularized preference model P(y y' | x) against a geometric mixture opponent policy π_t^mix(y| x)…

SPO

SPO (Minimax Self-Play Policy Optimization)

Directly feeds the empirical multi-sample win-rate payoff (y_i y_-i| x) - (1/2) into an on-policy PPO/REINFORCE loop without fitting a parametric reward model, converging to the minimax-optimal…

INPO

INPO (Iterative Nash Policy Optimization)

Approximates the online Nash-MD / Online Mirror Descent update using an offline-compatible DPO-style direct loss over preference pairs sampled from the current policy π_θ_t, avoiding on-policy token-generation…

OAIF

OAIF (Online AI Feedback)

Samples two candidate responses (y_1, y_2) ~ π_θ(·| x) online at every training step, queries an LLM-as-a-Judge in real time to label which response is preferred, and immediately applies an online DPO/IPO…

Self-Rewarding Language Models

Unifies the actor policy and the LLM-as-a-Judge reward model into a single model π_θ: in Iteration t, π_θ_t generates new candidate prompts, samples K candidate responses per prompt, grades its own…

Meta-Rewarding Language Models

Extends Self-Rewarding LMs by adding a third Meta-Judge role to the same model: for each response, the model generates multiple candidate judge critiques/scores, and the Meta-Judge evaluates which critique is most accurate…

Online Mirror Descent

Online Mirror Descent (OMD) Policy Optimization & Long2Short RL (Kimi k1.5)

Formulates long-CoT RLVR as Online Mirror Descent with analytical partition function log Z(x) ≈ log (1/G)∑_i=1^G exp(r_i / τ), optimizing the squared score-matching loss between the log-policy ratio…

Best-of-N Rejection Sampling Fine-Tuning

Best-of-N Rejection Sampling Fine-Tuning (RS-SFT / RAFT)

Samples N candidate completions y_1, …, y_N ~ π_θ_t(·| x) at temperature T ∈ [0.7, 1.0], evaluates all N candidates with a reward model or executable verifier, selects the highest-scoring…

RSO

RSO (Statistical Rejection Sampling Optimization)

Notices that selecting the simple max_1≤ i≤ N R(x, y_i) skews the empirical distribution away from the theoretical KL-regularized optimal policy π^(y| x) ∝ π_ref(y|…

STaR

STaR (Self-Taught Reasoner)

Iterative reasoning bootstrap loop for datasets that only have questions and final answers (x_i, a_i^) without ground-truth chain-of-thought rationales: 1.

Quiet-STaR

Quiet-STaR (Token-Level Internal Thought Bootstrapping)

Generalizes STaR from curated QA datasets to arbitrary unstructured text: at every token position t in pretraining/post-training text, the model generates parallel internal "thought" continuations…

V-STaR

V-STaR (Training Verifiers for Self-Taught Reasoners)

Use both successful and unsuccessful reasoning attempts to train a verifier, then use it to rank candidate solutions at inference time.

ReST

ReST (Reinforced Self-Training: Grow & Improve)

Decouples offline data generation (Grow step: sample a large dataset of trajectories from the current policy π_θ_g) from iterative offline policy optimization (Improve step: filter the grown dataset with an…

ReST^EM

ReST^EM (Expectation-Maximization Reinforced Self-Training)

Formulates binary-reward self-training as formal Expectation-Maximization on latent correct trajectories: - E-step: Sample K completions per problem from π_θ_t and filter by binary verifier r(x,y) ∈ {0,…

ReST-MCTS*

ReST-MCTS* (Process-Reward-Guided Tree Search Self-Training)

Replaces independent flat sampling in the E-step of ReST^EM with Process-Reward-guided Monte Carlo Tree Search (MCTS^), using inferred step-level values from tree backup to simultaneously train both…

SCoRe

SCoRe (Training Language Models to Self-Correct via Multi-Turn RL)

Proves that offline SFT or naive single-turn RL on self-correction tasks fails due to distribution shift and mode collapse (the model either outputs the final answer on Turn 1 and learns nothing about Turn 2 correction, or…

RFT

RFT (Rejection Sampling Fine-Tuning with Reasoning Path Diversity)

Samples K=100 reasoning paths per math problem, verifies the final answer, clusters the correct reasoning paths by distinct intermediate equation/code signatures (discarding near-duplicate algebra paths), and trains on the…

Bradley-Terry Outcome Reward Modeling

Bradley-Terry Outcome Reward Modeling (ORM)

Replaces the final unembedding layer of a pretrained/SFT LLM with a scalar linear head R_φ(x, y) ∈ R evaluated on the final EOS token, trained via pairwise binary cross-entropy on human/AI preference…

Process Reward Modeling

Process Reward Modeling (PRM / PRM800K)

Instead of assigning a single scalar score at the end of a trajectory, assigns a correctness probability p_k = P(step s_k is valid and on-track | x, s_1:k) ∈ [0, 1] at the end of every reasoning step…

Math-Shepherd

Math-Shepherd (Automated Monte Carlo Process Supervision)

Eliminates expensive human step-by-step annotation (like PRM800K) by defining the process label of intermediate step s_k via automated Monte Carlo rollouts from a completer policy π: - Hard Estimation (HE):…

OmegaPRM & Consensus-Filtered PRM

OmegaPRM & Consensus-Filtered PRM (Qwen2.5-Math-PRM)

Scales automated process supervision efficiently using a Divide-and-Conquer Monte Carlo Tree Search (OmegaPRM) that performs binary search over a failing trajectory to locate the exact first error step in O(log T)…

PAV

PAV (Process Advantage Verifiers)

Proves that a step-level reward should measure progress (the change in success probability induced by step s_k) rather than absolute state value

GenRM

GenRM (Generative Reward Models & CoT Verifiers)

Replaces the discriminative scalar classification head R_φ(x,y) with next-token generation: the reward model first generates an explicit natural-language verification chain-of-thought z_critique comparing or…

CLoud

CLoud (Critique-out-Loud) & Self-Taught Evaluators

- CLoud: Trains a single model where the LLM base generates an explicit natural-language critique c ~ π_φ(·| x,y) first, and a lightweight scalar reward head placed on top of the generated critique tokens…

DeepSeek-GRM & SPCT

DeepSeek-GRM & SPCT (Self-Principled Critique Tuning)

Pointwise Generative Reward Model trained via Self-Principled Critique Tuning (SPCT): the model first dynamically generates evaluation principles (rubric criteria and weights tailored to the specific prompt x), then…

Constitutional AI

Constitutional AI (SL-CAI & RL-CAI) and RLAIF

Replaces human harmlessness annotations with a formal written Constitution (a list of normative, ethical, and epistemic principles): 1.

Deliberative Alignment

Deliberative Alignment (OpenAI o1 / o3 / GPT-6 Safety Reasoning)

Rather than hoping a model implicitly infers safety boundaries from millions of binary refusal/compliance examples, Deliberative Alignment explicitly teaches the model the exact text of the safety & behavior policies during…

Rule-Based Rewards

Rule-Based Rewards (RBR) & Instruction Hierarchy Training

- Rule-Based Rewards (RBR): Decomposes desired safety/behavioral style into fine-grained propositional rules (e.g., "Does not include preachy lecturing", "Clearly states inability to fulfill unsafe request", *"Provides…

SteerLM & HelpSteer Multi-Attribute Conditioned Post-Training

Replaces single-scalar reward modeling with a 5-dimensional regression Reward Model (HelpSteer / HelpSteer2 / HelpSteer3) predicting explicit human-interpretable attributes (Helpfulness, Correctness, Coherence, Complexity,…

WARM

WARM (Weight Averaged Reward Models) & Reward Ensembles

Mitigates Goodhart's Law / reward overoptimization during RLHF without paying the N× inference cost of running an ensemble of N separate reward models.

Sequence-Level Knowledge Distillation

Sequence-Level Knowledge Distillation (SeqKD / Reasoning Trace Distillation)

Samples verified reasoning trajectories y^ ~ π_teacher(·| x) from a frontier reasoning model (e.g., DeepSeek-R1 671B or Qwen3-235B) after rejection sampling against verifiers, and trains a smaller…

On-Policy & Generalized Knowledge Distillation

On-Policy & Generalized Knowledge Distillation (GKD / Strong-to-Weak Logit Distillation)

Standard offline distillation trains the student on teacher-generated sequences y ~ π_teacher, creating a train-test exposure mismatch when the student samples from its own distribution y ~ π_θ at…

MiniLLM

MiniLLM (Reverse-KL Knowledge Distillation)

Replaces forward KL divergence D_KL(π_teacher|π_θ) (which forces a small-capacity student to spread probability mass across all modes of the teacher, producing hallucinations in…

DistiLLM

DistiLLM (Skew KL & Adaptive Off-Policy Distillation)

Optimizes a parameterized Skew KL divergence D_KL(π_teacher | α π_teacher + (1-α)π_θ) (which has bounded, smooth gradients unlike raw reverse KL) paired with an…

Weak-to-Strong Generalization

Weak-to-Strong Generalization (Superalignment Supervision)

Studies how a weak supervisor model π_weak can supervise a much stronger pretrained student π_strong without the student merely imitating the weak supervisor's errors, using an auxiliary confidence…

Model Soups & Multi-Specialist Checkpoint Averaging

Trains multiple SFT, DPO, or RL checkpoints θ_1, …, θ_M with different hyperparameter seeds, data mixes, or domain specializations (e.g., a math/code specialist, a long-context specialist, a general chat…

Task Arithmetic, TIES-Merging & DARE

Benchmark articles

Compare their evaluation contracts →

Loading benchmark articles…

Understand the workbehind the score.