ErdősBench · 139-item controlled slice
Qwen3.6-27B
ArxivNet SFT + Verified Reasoning V-SAO
Untouched base2.155 / 4
Post-trained2.655 / 4
+0.500 mean points / 4
139 evaluation items · average judgment
THE LEARNING LOOP / POST-TRAINING
From feedback
to capability.
Understand the training signal before choosing the method. Explore demonstrations, preferences, verifiable rewards, and how to measure what changed.
01 /
Diagnose
Find a capability gap using a held-out evaluation.
02 /
Construct
Create demonstrations, preferences, or environments.
03 /
Optimize
Update the policy against the chosen learning signal.
04 /
Verify
Compare with the exact base under matched conditions.
The method notebook
Explore the benchmarks →Training eras, model-report notes & objective reference →
What changed after training?
Explore training lift →ErdősBench · 139-item controlled slice
Qwen3.6-27B
Verified Reasoning V-RLVR
Untouched base2.155 / 4
Post-trained2.219 / 4
+0.064 mean points / 4
139 evaluation items · average judgment
SimoBench · 126 proofs
Qwen3.5-4B-Base
OlympiadNet O-RLVR
Untouched base2.929 / 7
Post-trained3.135 / 7
+0.206 mean points / 7
126 evaluation items · mean proof credit
369 → 395 proof points out of 882. The means above divide these totals by 126.
Ulam powers RLTasks. These are publisher-reported results, not independent replications. Ulam AI · Frontier Data for Frontier Models — Training Lift Scorecards ↗
148 of 148 method notes
SFT
Teaching a model what good looks like ↗
Demonstrations, data quality, and the difference between learning an answer format and improving a capability.
DPO / RLHF
Learning from preferences ↗
How preference pairs, reward models, and direct optimization shape an assistant’s behavior.
RLVR
When the reward can be checked ↗
Executable verifiers, hidden holdouts, and the gap between maximizing a score and learning the intended task.
PPO / GRPO
From rollouts to policy updates ↗
A conceptual map of policy optimization, relative advantages, and the compute that belongs in an experiment report.
ENV
A task is more than a prompt ↗
Observations, actions, partial information, and why environment design determines what a model can learn.
EVAL
Did the training actually help? ↗
Exact-base controls, contamination checks, uncertainty, and honest attribution for post-training gains.
LoRA / QLoRA
Adapt a model without updating every weight ↗
Low-rank adapters, quantized base models, and what an efficient fine-tuning comparison should keep fixed.
DISTILLATION
What a student model learns from a teacher ↗
Distillation transfers a selected learning signal, so teacher quality and dataset construction matter as much as model size.
PRM / ORM
Reward the reasoning, the result, or both? ↗
Process labels and outcome checks expose different errors. A sound evaluation needs to know what each one actually verifies.
SYNTHETIC DATA
Generate more data without repeating the same mistakes ↗
Instruction generation, filtering, diversity, and the checks that make synthetic examples useful for post-training.
SELECTION / STaR
Learn from the attempts that survive checking ↗
Sampling, verification, and iterative self-training can turn successful trajectories into supervision—but selection changes the dataset.
KTO / ORPO / SimPO
Different preference objectives, different assumptions ↗
Compare methods by the feedback they need and the behavior they optimize, rather than treating every preference-training label as interchangeable.
RL SYSTEMS / DAPO
The training system is part of the method ↗
Rollout generation, reward handling, sequence lengths, and system throughput can determine what an RL experiment learns.
REWARD DESIGN
Make the score mean what you intended ↗
Verifier coverage, partial credit, invalid episodes, and independent checks determine whether reward gains represent capability.
INFERENCE COMPUTE
Separate better learning from more attempts ↗
Reasoning effort, tool use, sampling, and reranking can improve a system without changing its trained weights.
DATA BOUNDARIES
Keep the test outside the learning loop ↗
Contamination includes structural leakage and benchmark-specific selection, not only copied answer text.
Multi-Task Instruction Tuning
Multi-Task Instruction Tuning (FLAN / T0 / NaturalInstructions) ↗
Teach one model to follow many kinds of instructions by turning classification, question answering, summarization and other tasks into supervised examples.
Superficial Alignment & High-Quality Curation SFT
Superficial Alignment & High-Quality Curation SFT (LIMA / AlpaGasus / DEITA) ↗
Study how a small, carefully curated set of demonstrations can teach a pretrained model the style and format of useful responses.
Long-CoT Cold-Start & Reasoning-Centric Minimal SFT
Long-CoT Cold-Start & Reasoning-Centric Minimal SFT (DeepSeek-R1 Cold-Start / s1 / LIMO) ↗
Start reasoning training with a small set of checked, extended solutions that demonstrate reflection, exploration and correction.
Evol-Instruct & Complexity-Scaled Synthetic SFT
Evol-Instruct & Complexity-Scaled Synthetic SFT (WizardLM / WizardCoder / WizardMath) ↗
Expand instruction data by asking a teacher model to add constraints, deepen reasoning and introduce new kinds of tasks.
Self-Instruct, Magpie & Persona-Driven Synthetic SFT
Self-Instruct, Magpie & Persona-Driven Synthetic SFT (Alpaca / Magpie / PersonaHub) ↗
Use a teacher model to generate instruction–response pairs, then filter and diversify them for supervised training.
Explanation Tuning & System-Prompt Trace Distillation
Explanation Tuning & System-Prompt Trace Distillation (Orca / Orca 2 / AgentInstruct) ↗
Augments standard teacher outputs with rich system prompts ("Explain like I'm five", "Think step-by-step and justify every assumption") to elicit detailed intermediate reasoning explanations from a frontier teacher, then…
Instruction Backtranslation & Self-Alignment
Instruction Backtranslation & Self-Alignment (Humpback / Kun) ↗
Trains an inverse model π_inv(x | y) to predict candidate user instructions x from raw, unlabelled web/textbook documents y, then uses the forward model to score and filter (x, y) pairs by quality,…
Multi-Stage Curriculum SFT & Specialized Research SFT
Multi-Stage Curriculum SFT & Specialized Research SFT (Annealing SFT / ArxivNet A-SFT) ↗
Splits SFT into a two-stage or multi-stage curriculum: Stage 1 trains on broad, diverse general instructions and domain-foundational reasoning; Stage 2 anneals the learning rate on high-precision, long-context domain research…
Noisy Embedding & Token-Weighted SFT
Noisy Embedding & Token-Weighted SFT (NEFTune / Prompt-Loss Weighting) ↗
Modifies the SFT forward/loss pass to prevent memorization and improve conversational generation.
Dual-Mode / Hybrid Thinking-NonThinking Fusion SFT
Dual-Mode / Hybrid Thinking-NonThinking Fusion SFT (Qwen3 / DeepSeek-V3.1+) ↗
After training a reasoning RL policy, conducts a fusion SFT stage that mixes long-CoT trajectories (reasoning traces triggered by /think or system flags) with fast, direct non-thinking responses (reasoning traces…
Critique-and-Repair SFT / Reflection-Tuning
Critique-and-Repair SFT / Reflection-Tuning (Self-Critique SFT / CRITIC) ↗
Rather than training only on clean (x, y^) gold trajectories, trains the model on multi-step trajectories (x, y_draft, c_critique, y_repaired^) containing realistic model failures,…
Tool-Integrated Reasoning
Tool-Integrated Reasoning (TIR) SFT (ToRA / NuminaMath-TIR) ↗
Supervises interleaved natural-language deduction and executable code/tool blocks: x → r_1 → a_1 (Python/SymPy/Bash) → obs_1 → r_2 → … → y,…
Formal Proof-State & Tactic SFT
Formal Proof-State & Tactic SFT (LeanDojo / DeepSeek-Prover / UlamAI Prover) ↗
Extracts (proof_state, premise_context) -> tactic pairs (or whole-proof generation with natural-language informal sketch comments) from formal theorem provers (Lean 4, Isabelle/HOL, Coq) and fine-tunes the LLM to emit formally…
Dynamic Fine-Tuning
Dynamic Fine-Tuning (DFT) / Probability-Stabilized SFT ↗
Reweight supervised token losses with detached target probabilities, then measure the effect on held-out generalization.
Proximal Policy Optimization
Proximal Policy Optimization (PPO) for RLHF ↗
Optimizes the LLM policy π_θ against a token-level reward r_t = -β log(π_θ(y_t| x,y_<t)/π_ref(y_t| x,y_<t)) + I[t=|y|]R_φ(x,y) using Generalized Advantage…
PPO-ptx
PPO-ptx (PPO with Pretraining Mix Gradients) ↗
Augments the standard PPO RLHF objective with an auxiliary next-token log-likelihood loss on the original pretraining distribution D_pretrain to prevent the "alignment tax" (regressions on public NLP…
REINFORCE with Moving-Average / Learned Baseline for LLMs
REINFORCE with Moving-Average / Learned Baseline for LLMs ↗
Treats the entire generated sequence y as a single bandit action (since intermediate environment transitions in text generation are deterministic), optimizing
RLOO
RLOO (REINFORCE Leave-One-Out) ↗
Samples K independent completions y^(1), …, y^(K) ~ π_θ(·| x) per prompt and uses the average reward of the other K-1 completions as an unbiased, low-variance prompt-specific baseline…
ReMax
ReMax (Reward Maximization with Greedy Baseline) ↗
Train from the reward difference between a sampled response and a greedily decoded response, using the greedy response as a baseline without a separate value network.
AWR / AWAC
AWR / AWAC (Advantage-Weighted Regression / Actor-Critic) ↗
Formulates policy improvement as weighted supervised learning where off-policy or replay-buffer trajectories (x,y) are weighted exponentially by their estimated advantage
PPG
PPG (Phasic Policy Gradient) / Decoupled Policy-Value RLHF ↗
Separates policy updates (actor phase) and value-function distillation/training (auxiliary phase) into distinct alternating phases so value estimation can train for more epochs without corrupting shared representations or…
P3O
P3O (Pairwise Proximal Policy Optimization) ↗
Notices that standard PPO is sensitive to additive shifts in the reward model R(x,y) + c(x) unless the value critic is perfectly calibrated.
APA
APA (Advantage-Induced Policy Alignment) ↗
Replaces PPO's multiplicative ratio clipping with a squared-error loss between the log-policy ratio log π_θ(y_t| s_t)π_ref(y_t| s_t) and the scaled token advantage (A_t/β),…
NLPO
NLPO (Natural Language Policy Optimization) ↗
Extends PPO for language generation with a parameterized top-p masking policy that dynamically masks out low-probability vocabulary tokens during RL exploration, preventing grammatical collapse in large action spaces (|V| >…
IMPALA V-Trace / Asynchronous Actor-Learner LLM RL
IMPALA V-Trace / Asynchronous Actor-Learner LLM RL ↗
Decouples rollout actors (running high-throughput vLLM/SGLang inference engines) from gradient learners (running FSDP/Megatron), correcting for policy staleness μ ≠ π_θ using truncated importance sampling weights…
Soft Q-Learning
Soft Q-Learning (SQL) & Path Consistency Learning (PCL) for LLMs ↗
Exploits the equivalence between maximum-entropy RL and soft Bellman consistency along any sub-trajectory (s_t, a_t:t+k, s_t+k), allowing unified off-policy and on-policy token-level updates without separate actor-critic…
RLVR
RLVR (Reinforcement Learning with Verifiable Rewards) ↗
Replaces neural preference reward models R_φ(x,y) with deterministic, programmatic verification functions v(x,y) ∈ 0, 1 (or graded rubric scores v(x,y) ∈ [0, 1]) such as symbolic math equivalence checkers…
Pure Zero-SFT RLVR
Pure Zero-SFT RLVR (DeepSeek-R1-Zero Paradigm) ↗
Applies RLVR (via GRPO with accuracy + format rewards) directly to a raw pretrained base checkpoint π_base with zero prior SFT, allowing the model to discover long-CoT reasoning, self-verification, and…
GRPO
GRPO (Group Relative Policy Optimization) ↗
Samples a group of G outputs y_1, …, y_G from the old policy π_θ_old(·| x) for each prompt x, computes their rewards r_1, …, r_G, and estimates the advantage of token t in…
Dr.GRPO
Dr.GRPO (Group Relative Policy Optimization Done Right) ↗
Remove sequence-length and reward-standard-deviation normalization from GRPO to address the length and question-difficulty biases studied in the paper.
DAPO
DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) ↗
Introduces four synergistic modifications to GRPO for large-scale long-CoT RL: 1.
VAPO
VAPO (Value-Augmented Proximal Policy Optimization for Long-CoT) ↗
Demonstrates that value-based PPO can outperform critic-free GRPO/DAPO on long-CoT reasoning once value-function pathologies are fixed via: (1) Value-Pretraining (warming up the critic from the reward signal before policy…
GSPO
GSPO (Group Sequence Policy Optimization) ↗
Identifies a fundamental flaw in token-level importance ratios ρ_i,t = π_θ(y_i,t| x,y_i,<t)π_θ_old(y_i,t| x,y_i,<t): while the basic policy gradient identity holds at the…
CISPO
CISPO (Clipped Importance Sampling Policy Optimization) ↗
Observes that PPO/GRPO/DAPO zero out the gradient of any token whose importance ratio ρ_i,t(θ) exceeds 1+ε.
REINFORCE++ & REINFORCE++-Baseline
REINFORCE++ & REINFORCE++-Baseline ↗
Combines the simplicity of critic-free REINFORCE with key PPO stability optimizations: (1) token-level KL penalty added to rewards, (2) PPO-style ratio clipping, and (3) global batch advantage normalization (normalizing…
SAO / V-SAO
SAO / V-SAO (Single-Rollout Asynchronous Optimization) ↗
Designed specifically for long-horizon asynchronous agentic RL where group-wise sampling (G=8 or 16 rollouts per prompt) creates severe tail-latency stragglers and off-policy staleness.
VinePPO
VinePPO (Monte Carlo Tree Rollout Credit Assignment) ↗
Replaces the inaccurate neural value network V_ψ(s_t) in PPO with unbiased Monte Carlo rollouts branched directly from intermediate reasoning steps s_t (utilizing fast vLLM prefix caching to sample K fast…
PRIME
PRIME (Process Reinforcement through Implicit Rewards) ↗
Trains an implicit Process Reward Model π_ψ online alongside the policy π_θ using only outcome labels r ∈ 0,1, where token-level implicit process rewards are defined automatically via the log-ratio…
GPG
GPG (Group Policy Gradient) ↗
Minimalist RLVR method that strips away surrogate ratio clipping, reference-model KL divergence, and critic networks altogether, directly optimizing the group-normalized REINFORCE objective with dynamic thresholding of…
TOPR
TOPR (Tapered Off-Policy REINFORCE) ↗
Handles off-policy lag in asynchronous RL by combining asymmetric tapered importance-sampling weights on positive-advantage trajectories with unclipped/bounded updates on negative-advantage trajectories, accelerating learning…
AGPO
AGPO (Adaptive Group Policy Optimization) ↗
Modifies GRPO to handle two boundary cases: (1) when all G responses in a group are uniformly correct or uniformly wrong (std=0), assigns a small non-zero signed advantage (+1 or -1) rather than 0 so the model still…
CPPO
CPPO (Completion Pruning Policy Optimization) ↗
Accelerates GRPO training by up to 4–8× by dynamically pruning low-contribution completions inside each group G before the expensive backward pass (keeping only completions with large absolute relative advantages |A_i|).
LCPO
LCPO (Length-Controlled Policy Optimization / L1) ↗
Trains reasoning models via GRPO to obey explicit user-specified token budgets n_target passed in the prompt
Step-GRPO / S-GRPO
Step-GRPO / S-GRPO (Step-Wise & Early-Exit GRPO) ↗
Augments GRPO's outcome reward with dense step-wise reasoning accuracy and formatting rewards (or serial early-exit rollouts truncated at intermediate reasoning checkpoints), rewarding the model for reaching correct…
Hybrid SFT-RL / Prefix-RFT
Hybrid SFT-RL / Prefix-RFT (UFT / LUFFY / Prefix-Guided RLVR) ↗
When a hard problem has 0% pass rate under current policy sampling (pass@G = 0), pure RLVR receives zero positive signal.
DPO
DPO (Direct Preference Optimization) ↗
Derives the closed-form optimal policy π^(y| x) = (1/Z(x))π_ref(y| x)exp((1/β)R(x,y)) of the KL-regularized RLHF objective, solving for R(x,y) = β log…
IPO
IPO (Identity Preference Optimization / ΨPO) ↗
Shows that under deterministic or near-deterministic preferences P(y_w y_l) → 1, the Bradley-Terry transformation in DPO drives log (π_θ(y_w)/π_θ(y_l)) → ∞ regardless of the KL…
KTO
KTO (Kahneman-Tversky Optimization) ↗
Eliminates the requirement for paired (y_w, y_l) preferences on the same prompt x.
ORPO
ORPO (Odds Ratio Preference Optimization) ↗
Combine supervised learning and preference optimization in one objective, using response odds to separate preferred and rejected answers without a reference model.
SimPO
SimPO (Simple Preference Optimization) ↗
Notices that at inference time, decoding is governed by the policy's own length-normalized log-likelihood (1/|y|)log π_θ(y| x), whereas DPO optimizes the unnormalized log-ratio against π_ref.
CPO
CPO (Contrastive Preference Optimization) ↗
Approximates the ideal reference policy as a uniform prior to drop π_ref memory overhead, combining a reference-free DPO contrastive loss with an explicit SFT negative log-likelihood regularizer on y_w:…
R-DPO
R-DPO (Regularized / Length-Desensitized DPO) ↗
Explicitly penalizes sequence-length exploitation in DPO by adding an analytical length margin α(|y_w| - |y_l|) inside the Bradley-Terry sigmoid argument
cDPO
cDPO (Conservative DPO) ↗
Accounts for a known label-noise rate ε ∈ (0, 0.5) in human/AI preference annotations by applying label smoothing directly to the DPO cross-entropy objective
rDPO
rDPO (Robust DPO under Random Preference Flips) ↗
Correct the DPO loss for a specified rate of random preference-label flips, under the paper’s symmetric-noise assumption.
sDPO
sDPO (Stepwise / Reference-Curriculum DPO) ↗
Splits offline preference datasets into sequential difficulty or quality tranches and updates the reference model π_ref ← π_θ^(k-1) at each tranche boundary so the reference policy stays close…
Step-DPO & MCTS-DPO
Step-DPO & MCTS-DPO (Process-Level Stepwise DPO) ↗
Instead of contrasting two complete responses (y_w, y_l) that share many valid early reasoning steps, identifies the exact first erroneous reasoning step s_k^l vs.
TDPO
TDPO (Token-Level Direct Preference Optimization) ↗
Incorporates forward token-level sequential KL divergence constraints ∑_t=1^|y| D_SeqKL(x, y_<t; π_ref|π_θ) directly into the DPO objective (TDPO1 and TDPO2), regulating…
f-DPO
f-DPO (Generalized f-Divergence DPO) ↗
Generalizes DPO from reverse KL divergence to any convex f-divergence (forward KL, Jensen-Shannon, α-divergence) by replacing log π_θπ_ref with the derivative of the convex conjugate…
β-DPO
β-DPO (Dynamic β Calibration for DPO) ↗
Dynamically adapts the KL penalty parameter β_i per preference pair based on the empirical reward discrepancy Δ R_i relative to batch statistics, increasing β on noisy low-gap pairs and decreasing β on…
DPO-Positive and chosen-response likelihood regularization
DPO-Positive and chosen-response likelihood regularization ↗
Identifies a critical failure mode of DPO on reasoning and math datasets where y_w and y_l differ by only a few tokens (small edit distance): standard DPO reduces log π_θ(y_w| x) along with log…
NCA & InfoNCA
NCA & InfoNCA (Noise Contrastive Alignment) ↗
Formulates LLM alignment from scalar reward annotations r(x, y_k)_k=1^K as a Noise Contrastive Estimation (NCE) problem, training the policy log-ratio β log π_θ(y_k| x){π_ref(y_k|…
BCO
BCO (Binary Classifier Optimization) ↗
Trains a binary logistic classifier separating thumbs-up completions y^+ from thumbs-down completions y^- using density ratio matching with a dynamically shifted reward baseline δ, providing theoretical grounding…
AOT
AOT (Alignment via Optimal Transport) ↗
Enforces first-order stochastic dominance of the implicit reward distribution of positive samples over negative samples using 1D Wasserstein optimal transport sorting across mini-batches, supporting both paired and unpaired…
APO
APO (Anchored Preference Optimization: APO-zero & APO-down) ↗
Controls whether the chosen likelihood log π_θ(y_w) and rejected likelihood log π_θ(y_l) increase or decrease during offline alignment depending on whether the model is already stronger or weaker than the…
WPO
WPO (Weighted Preference Optimization for Off-Policy-to-On-Policy Simulation) ↗
Reweights off-policy preference pairs (x, y_w, y_l) by their sequence probability under the current policy w(x,y) = exp((1/|y|)∑_t log π_θ(y_t| x, y_<t)), making off-policy public…
RainbowPO & MallowsPO
RainbowPO & MallowsPO (Unified Multi-Component Preference Optimization) ↗
Integrates seven orthogonal DPO improvements—length normalization, margin calibration, reference-policy mixing (α π_θ + (1-α)π_ref), WPO reweighting, and contextual scaling (Mallows…
RRHF
RRHF (Rank Responses to Align Language Models with Human Feedback) ↗
Samples K candidate responses y_1, …, y_K with external scores r_1, …, r_K and optimizes a pairwise ranking margin hinge loss over length-normalized conditional log-probabilities p_i =…
PRO
PRO (Preference Ranking Optimization) ↗
Extends pairwise Bradley-Terry preference optimization to a full listwise Plackett-Luce permutation ranking over K ordered candidates y_1 y_2 … y_K
SLiC-HF
SLiC-HF (Sequence Likelihood Calibration with Human Feedback) ↗
Directly calibrates the sequence log-likelihood log π_θ(y| x) using a large-margin rank hinge loss max(0, δ - log π_θ(y_w| x) + log π_θ(y_l| x)) regularized by cross-entropy on the…
LiPO & LiPO-λ
LiPO & LiPO-λ (Listwise Preference Optimization via Learning-to-Rank) ↗
Adapts LambdaRank from classic Information Retrieval to LLM alignment: given a list of K responses with real-valued reward labels ψ = (s_1, …, s_K), weights each pairwise DPO log-ratio gradient by the change in…
Multi-Preference / Plackett-Luce DPO
Multi-Preference / Plackett-Luce DPO (mDPO / K-Pair DPO) ↗
Samples K responses per prompt and constructs either the exact Plackett-Luce listwise likelihood or all 2 thresholded preference pairs where R(y_i) - R(y_j) > δ_min, maximizing data efficiency from…
DNO
DNO (Direct Nash Optimization) ↗
Batched on-policy iterative algorithm that samples K outputs from the current policy π_θ_t, compares them against each other and the gold/teacher output using a general pairwise win-rate oracle P(y_i…
MODPO / Multi-Objective Margin Preference Optimization
MODPO / Multi-Objective Margin Preference Optimization ↗
Aligns a single LLM simultaneously to M competing preference dimensions (e.g., helpfulness, safety, conciseness, citation faithfulness) by incorporating scalarized reward margins ∑_m ≠ k w_m r_m(x, y) from M-1…
Cringe Loss & Contrastive Unlikelihood Training
Cringe Loss & Contrastive Unlikelihood Training ↗
Token-level negative-feedback objective that penalizes toxic, repetitive, or invalid tokens y_t^- by contrasting their logit directly against a top-k plausible positive token sampled from the model's own distribution at…
Iterative DPO / Online DPO
Iterative DPO / Online DPO ↗
Closes the off-policy gap of vanilla DPO by alternating across T rounds: in round t, sample K responses y_1,…,y_K ~ π_θ_t(·| x), score them with a reward model or verifier to form fresh…
XPO
XPO (Exploratory Preference Optimization) ↗
Augments Iterative Online DPO with an explicit theoretical optimism bonus α ∑_y ~ π_ref log π_θ(y| x) that encourages the policy to actively explore out-of-support responses where the…
SELM
SELM (Self-Exploring Language Models) ↗
Bi-level online preference optimization objective that adds an optimism term to the implicit reward during online DPOpair selection, actively seeking out high-potential unseen responses rather than greedily exploiting the…
SPIN
SPIN (Self-Play Fine-Tuning) ↗
Allows an SFT model to improve without any additional human preference data or external reward model via a two-player minimax game: at iteration t, the opponent (old policy π_θ_t) generates synthetic responses…
SPPO
SPPO (Self-Play Preference Optimization) ↗
Formulates alignment with general, potentially intransitive preferences P(y y' | x) as a constant-sum two-player game, seeking the von Neumann winner (Nash equilibrium) via multiplicative weights / mirror…
Nash-MD
Nash-MD (Nash Learning from Human Feedback via Mirror Descent) ↗
Replaces the reward-maximization paradigm of RLHF with finding the Nash equilibrium of a KL-regularized preference model P(y y' | x) against a geometric mixture opponent policy π_t^mix(y| x)…
SPO
SPO (Minimax Self-Play Policy Optimization) ↗
Directly feeds the empirical multi-sample win-rate payoff (y_i y_-i| x) - (1/2) into an on-policy PPO/REINFORCE loop without fitting a parametric reward model, converging to the minimax-optimal…
INPO
INPO (Iterative Nash Policy Optimization) ↗
Approximates the online Nash-MD / Online Mirror Descent update using an offline-compatible DPO-style direct loss over preference pairs sampled from the current policy π_θ_t, avoiding on-policy token-generation…
OAIF
OAIF (Online AI Feedback) ↗
Samples two candidate responses (y_1, y_2) ~ π_θ(·| x) online at every training step, queries an LLM-as-a-Judge in real time to label which response is preferred, and immediately applies an online DPO/IPO…
Self-Rewarding Language Models
Self-Rewarding Language Models ↗
Unifies the actor policy and the LLM-as-a-Judge reward model into a single model π_θ: in Iteration t, π_θ_t generates new candidate prompts, samples K candidate responses per prompt, grades its own…
Meta-Rewarding Language Models
Meta-Rewarding Language Models ↗
Extends Self-Rewarding LMs by adding a third Meta-Judge role to the same model: for each response, the model generates multiple candidate judge critiques/scores, and the Meta-Judge evaluates which critique is most accurate…
Online Mirror Descent
Online Mirror Descent (OMD) Policy Optimization & Long2Short RL (Kimi k1.5) ↗
Formulates long-CoT RLVR as Online Mirror Descent with analytical partition function log Z(x) ≈ log (1/G)∑_i=1^G exp(r_i / τ), optimizing the squared score-matching loss between the log-policy ratio…
Best-of-N Rejection Sampling Fine-Tuning
Best-of-N Rejection Sampling Fine-Tuning (RS-SFT / RAFT) ↗
Samples N candidate completions y_1, …, y_N ~ π_θ_t(·| x) at temperature T ∈ [0.7, 1.0], evaluates all N candidates with a reward model or executable verifier, selects the highest-scoring…
RSO
RSO (Statistical Rejection Sampling Optimization) ↗
Notices that selecting the simple max_1≤ i≤ N R(x, y_i) skews the empirical distribution away from the theoretical KL-regularized optimal policy π^(y| x) ∝ π_ref(y|…
STaR
STaR (Self-Taught Reasoner) ↗
Iterative reasoning bootstrap loop for datasets that only have questions and final answers (x_i, a_i^) without ground-truth chain-of-thought rationales: 1.
Quiet-STaR
Quiet-STaR (Token-Level Internal Thought Bootstrapping) ↗
Generalizes STaR from curated QA datasets to arbitrary unstructured text: at every token position t in pretraining/post-training text, the model generates parallel internal "thought" continuations…
V-STaR
V-STaR (Training Verifiers for Self-Taught Reasoners) ↗
Use both successful and unsuccessful reasoning attempts to train a verifier, then use it to rank candidate solutions at inference time.
ReST
ReST (Reinforced Self-Training: Grow & Improve) ↗
Decouples offline data generation (Grow step: sample a large dataset of trajectories from the current policy π_θ_g) from iterative offline policy optimization (Improve step: filter the grown dataset with an…
ReST^EM
ReST^EM (Expectation-Maximization Reinforced Self-Training) ↗
Formulates binary-reward self-training as formal Expectation-Maximization on latent correct trajectories: - E-step: Sample K completions per problem from π_θ_t and filter by binary verifier r(x,y) ∈ {0,…
ReST-MCTS*
ReST-MCTS* (Process-Reward-Guided Tree Search Self-Training) ↗
Replaces independent flat sampling in the E-step of ReST^EM with Process-Reward-guided Monte Carlo Tree Search (MCTS^), using inferred step-level values from tree backup to simultaneously train both…
SCoRe
SCoRe (Training Language Models to Self-Correct via Multi-Turn RL) ↗
Proves that offline SFT or naive single-turn RL on self-correction tasks fails due to distribution shift and mode collapse (the model either outputs the final answer on Turn 1 and learns nothing about Turn 2 correction, or…
RFT
RFT (Rejection Sampling Fine-Tuning with Reasoning Path Diversity) ↗
Samples K=100 reasoning paths per math problem, verifies the final answer, clusters the correct reasoning paths by distinct intermediate equation/code signatures (discarding near-duplicate algebra paths), and trains on the…
Bradley-Terry Outcome Reward Modeling
Bradley-Terry Outcome Reward Modeling (ORM) ↗
Replaces the final unembedding layer of a pretrained/SFT LLM with a scalar linear head R_φ(x, y) ∈ R evaluated on the final EOS token, trained via pairwise binary cross-entropy on human/AI preference…
Process Reward Modeling
Process Reward Modeling (PRM / PRM800K) ↗
Instead of assigning a single scalar score at the end of a trajectory, assigns a correctness probability p_k = P(step s_k is valid and on-track | x, s_1:k) ∈ [0, 1] at the end of every reasoning step…
Math-Shepherd
Math-Shepherd (Automated Monte Carlo Process Supervision) ↗
Eliminates expensive human step-by-step annotation (like PRM800K) by defining the process label of intermediate step s_k via automated Monte Carlo rollouts from a completer policy π: - Hard Estimation (HE):…
OmegaPRM & Consensus-Filtered PRM
OmegaPRM & Consensus-Filtered PRM (Qwen2.5-Math-PRM) ↗
Scales automated process supervision efficiently using a Divide-and-Conquer Monte Carlo Tree Search (OmegaPRM) that performs binary search over a failing trajectory to locate the exact first error step in O(log T)…
PAV
PAV (Process Advantage Verifiers) ↗
Proves that a step-level reward should measure progress (the change in success probability induced by step s_k) rather than absolute state value
GenRM
GenRM (Generative Reward Models & CoT Verifiers) ↗
Replaces the discriminative scalar classification head R_φ(x,y) with next-token generation: the reward model first generates an explicit natural-language verification chain-of-thought z_critique comparing or…
CLoud
CLoud (Critique-out-Loud) & Self-Taught Evaluators ↗
- CLoud: Trains a single model where the LLM base generates an explicit natural-language critique c ~ π_φ(·| x,y) first, and a lightweight scalar reward head placed on top of the generated critique tokens…
DeepSeek-GRM & SPCT
DeepSeek-GRM & SPCT (Self-Principled Critique Tuning) ↗
Pointwise Generative Reward Model trained via Self-Principled Critique Tuning (SPCT): the model first dynamically generates evaluation principles (rubric criteria and weights tailored to the specific prompt x), then…
Constitutional AI
Constitutional AI (SL-CAI & RL-CAI) and RLAIF ↗
Replaces human harmlessness annotations with a formal written Constitution (a list of normative, ethical, and epistemic principles): 1.
Deliberative Alignment
Deliberative Alignment (OpenAI o1 / o3 / GPT-6 Safety Reasoning) ↗
Rather than hoping a model implicitly infers safety boundaries from millions of binary refusal/compliance examples, Deliberative Alignment explicitly teaches the model the exact text of the safety & behavior policies during…
Rule-Based Rewards
Rule-Based Rewards (RBR) & Instruction Hierarchy Training ↗
- Rule-Based Rewards (RBR): Decomposes desired safety/behavioral style into fine-grained propositional rules (e.g., "Does not include preachy lecturing", "Clearly states inability to fulfill unsafe request", *"Provides…
SteerLM & HelpSteer Multi-Attribute Conditioned Post-Training
SteerLM & HelpSteer Multi-Attribute Conditioned Post-Training ↗
Replaces single-scalar reward modeling with a 5-dimensional regression Reward Model (HelpSteer / HelpSteer2 / HelpSteer3) predicting explicit human-interpretable attributes (Helpfulness, Correctness, Coherence, Complexity,…
WARM
WARM (Weight Averaged Reward Models) & Reward Ensembles ↗
Mitigates Goodhart's Law / reward overoptimization during RLHF without paying the N× inference cost of running an ensemble of N separate reward models.
Sequence-Level Knowledge Distillation
Sequence-Level Knowledge Distillation (SeqKD / Reasoning Trace Distillation) ↗
Samples verified reasoning trajectories y^ ~ π_teacher(·| x) from a frontier reasoning model (e.g., DeepSeek-R1 671B or Qwen3-235B) after rejection sampling against verifiers, and trains a smaller…
On-Policy & Generalized Knowledge Distillation
On-Policy & Generalized Knowledge Distillation (GKD / Strong-to-Weak Logit Distillation) ↗
Standard offline distillation trains the student on teacher-generated sequences y ~ π_teacher, creating a train-test exposure mismatch when the student samples from its own distribution y ~ π_θ at…
MiniLLM
MiniLLM (Reverse-KL Knowledge Distillation) ↗
Replaces forward KL divergence D_KL(π_teacher|π_θ) (which forces a small-capacity student to spread probability mass across all modes of the teacher, producing hallucinations in…
DistiLLM
DistiLLM (Skew KL & Adaptive Off-Policy Distillation) ↗
Optimizes a parameterized Skew KL divergence D_KL(π_teacher | α π_teacher + (1-α)π_θ) (which has bounded, smooth gradients unlike raw reverse KL) paired with an…
Weak-to-Strong Generalization
Weak-to-Strong Generalization (Superalignment Supervision) ↗
Studies how a weak supervisor model π_weak can supervise a much stronger pretrained student π_strong without the student merely imitating the weak supervisor's errors, using an auxiliary confidence…
Model Soups & Multi-Specialist Checkpoint Averaging
Model Soups & Multi-Specialist Checkpoint Averaging ↗
Trains multiple SFT, DPO, or RL checkpoints θ_1, …, θ_M with different hyperparameter seeds, data mixes, or domain specializations (e.g., a math/code specialist, a long-context specialist, a general chat…
Task Arithmetic, TIES-Merging & DARE
Task Arithmetic, TIES-Merging & DARE (Drop And REscale) ↗
Operates on task vectors τ_m = θ_m - θ_base: - Task Arithmetic: θ_new = θ_base + ∑_m λ_m τ_m.
SLERP
SLERP (Spherical Linear Interpolation) ↗
Blend model weights along a spherical path between normalized parameter vectors, with the interpolation amount controlling the mixture.
WARP
WARP (Weight Averaged Rewarded Policies) ↗
Three-level weight-space RLHF algorithm developed by Google DeepMind to achieve Pareto-optimal reward vs.
ExPO
ExPO (Extrapolated Policy Optimization) ↗
Observes that an partially aligned DPO/RLHF checkpoint θ_aligned lies along a beneficial optimization trajectory from the SFT checkpoint θ_SFT, and applies zero-cost linear weight…
Multi-Token Prediction
Multi-Token Prediction (MTP) & Speculative Self-Distillation Post-Training (Medusa / EAGLE / DeepSeek-V3 MTP) ↗
Freezes or co-trains the main LLM backbone during post-training while distilling K lightweight speculative draft heads (or DeepSeek-V3's sequential Multi-Token Prediction modules) using on-policy hidden states and top-k…
LoRA
LoRA (Low-Rank Adaptation) & rsLoRA (Rank-Stabilized LoRA) ↗
Freezes pretrained weight matrices W_0 ∈ R^d_out× d_in and injects trainable low-rank matrices B ∈ R^d_out× r, A ∈ R^r × d_in (r…
QLoRA
QLoRA (4-Bit NormalFloat Quantized LoRA) & LoftQ ↗
Quantizes the frozen base LLM to 4-bit NormalFloat (NF4) (information-theoretically optimal quantile quantization for normal weight distributions) with Double Quantization of quantization scales and Paged Optimizers,…
DoRA
DoRA (Weight-Decomposed Low-Rank Adaptation) ↗
Decomposes pretrained weights into magnitude m ∈ R^1× d_in and unit directional matrix (V/|V|_c), applying low-rank updates exclusively to the directional component while training the…
PiSSA
PiSSA (Principal Singular Values Adaptation) & GaLore (Gradient Low-Rank Projection) ↗
- PiSSA: Instead of initializing A ~ N(0,σ^2), B=0 like LoRA, performs SVD on W_0 = U S V^ᵀ, initializes the trainable adapter (B, A) with the top-r principal singular values and vectors (which…
LISA
LISA (Layerwise Importance Sampled AdamW) & Spectrum ↗
- LISA: Always keeps the token embedding and LM head trainable while randomly unfreezing only γ (e.g., 2 to 4) middle transformer blocks per K iterations, achieving full-parameter fine-tuning convergence at LoRA memory…
ReFT / LoReFT
ReFT / LoReFT (Representation Fine-Tuning) & CAA (Contrastive Activation Addition) ↗
freezes 100% of model weights and intervenes directly on hidden residual stream representations h ∈ R^d at a subset of layers: - LoReFT: Learns a low-rank projection intervention Φ(h) = h + R^ᵀ (W h + b -…
Muon & MuonClip Post-Training Optimization
Muon & MuonClip Post-Training Optimization (Newton-Schulz Orthogonalization + QK-Clip) ↗
Replaces AdamW for 2D hidden matrix parameters with Muon (Momentum Orthogonalized by Newton-Schulz iterations, which equalizes singular value update scales so rare directions learn rapidly).
Multi-Turn Stateful Agentic RL
Multi-Turn Stateful Agentic RL (SWE-RL / WebRL / OSWorld-RL / ToolRL / Harbor Gyms) ↗
Optimizes LLM policies over multi-turn Partially Observable Markov Decision Processes (POMDPs) τ = (s_0, a_0, o_1, a_1, …, o_T, a_T) where actions a_t execute inside stateful external sandboxes (Docker Linux…
Interleaved Search & Retrieval-Augmented RL
Interleaved Search & Retrieval-Augmented RL (Search-R1 / R1-Searcher / DeepResearch RL) ↗
Trains an LLM via GRPO/PPO to autonomously decide when and what to query against a live search engine or vector corpus during multi-hop reasoning by emitting <search>query</search> tokens, receiving retrieved passages…
Self-Play Task-Proposing & Zero-Data Environment RL
Self-Play Task-Proposing & Zero-Data Environment RL (Absolute Zero Reasoner AZR / TTRL) ↗
- Absolute Zero Reasoner (AZR): Eliminates human-curated prompt datasets entirely by training a single model in two roles coupled via a Python code executor: a Proposer (rewarded via RL for generating verifiable…
MoE Routing Replay & Expert-Stabilized RL
MoE Routing Replay & Expert-Stabilized RL (DeepSeek-V3/R1 / MiniMax-M1 / Kimi K2) ↗
In sparse Mixture-of-Experts (MoE) models (e.g., 256 experts with top-8 routing), small weight updates or numerical differences between the inference rollout engine (vLLM FP8) and the training engine (Megatron BF16) cause…
Context Distillation & System-Prompt / Scratchpad Internalization
Context Distillation & System-Prompt / Scratchpad Internalization (GAtt / Context Distillation) ↗
Transfers behaviors that initially require a long in-context system prompt, persona specification, constitutional rulebook, or few-shot scratchpad c into a zero-prompt student π_θ(y| x) by minimizing token-level…
Reading path: start with supervised fine-tuning, follow the feedback signal into preference learning or RLVR, then use the evaluation article to design a matched experiment.